desispec API

desispec

Tools for DESI spectroscopic processing.

desispec.averagefluxcalibration

desispec.badcolumn

Utility functions to treat bad CCD columns

desispec.badcolumn.add_badcolumn_mask(frame, xyset, badcolumns_table, threshold_value=0.005, threshold_specfrac=0.6)[source]

Modifies the input frame spectral mask and fiber mask (frame.mask and frame.fibermap[“FIBERSTATUS”].mask) by adding (bitwise OR) the BADCOLUMN bits for values/fibers affected by bad CCD columns.

Parameters:
  • frame – desispec.frame.Frame instance

  • xyset – desispec.xytraceset.XYTraceSet instance (valid for the frame)

  • badcolumns_table – astropy.table.Table with columns “COLUMN” (CCD column index) and “ELEC_PER_SEC”, value in CCD column, in electron / sec

  • threshold_value – threshold for masking spectral pixels, in electron / sec

  • threshold_specfrac – fraction of specmask.BADCOLUMN masked pixels to trigger a fibermask bitmask fibermask.BADCOLUMN.

Returns:

nothing, the input frame is modified

desispec.badcolumn.compute_badcolumn_fibermask(frame_mask, camera_arm, threshold_specfrac=0.6)[source]

fills a mask for fibers affected by a bad CCD column.

Parameters:
  • frame_mask – 2D integer numpy array (nfibers x nwavelength), with bitmask defined in desispec.maskbits.specmask

  • threshold_specfrac – fraction of specmask.BADCOLUMN masked pixels to trigger a fibermask bitmask fibermask.BADAMP… .

  • camera_arm – ‘B’,’R’ or ‘Z’

Returns:

fiber mask 1D numpy array of size nfibers (bitwise OR of the orginal mask with the newly set bits)

desispec.badcolumn.compute_badcolumn_specmask(frame, xyset, badcolumns_table, threshold_value=0.005)[source]

fills a mask for spectral values affected by a bad CCD column.

Parameters:
  • frame – desispec.frame.Frame instance

  • xyset – desispec.xytraceset.XYTraceSet instance (valid for the frame)

  • badcolumns_table – astropy.table.Table with columns “COLUMN” (CCD column index) and “ELEC_PER_SEC”, value in CCD column, in electron / sec

  • threshold_value – threshold for masking spectral pixels, in electron / sec

Returns:

mask numpy array of same shape as frame.flux, or None if nothing is masked

desispec.badcolumn.flux_bias_function(delta_x)[source]

Multiply the excess counts in a bad column by this function to determine the bias on the extracted flux of a spectral trace situated at a distance dx (in pixels) from the bad CCD column. This has been derived empirically on the r and z cameras (with inverse variance cross-dispersion profile weighted extractions). https://github.com/desihub/desispec/pull/1371#issuecomment-904961936

Parameters:

delta_x – float or numpy array

Returns:

flux bias, same dimension as input delta_x

desispec.bootcalib

Utility functions to perform a quick calibration of DESI data

TODO: 1. Expand to r, i cameras 2. QA plots 3. Test with CR data

desispec.bootcalib.bootcalib(deg, flatimage, arcimage)[source]
Parameters:
  • deg – Legendre polynomial degree to use to fit

  • flatimage – desispec.image.Image object of flatfield

  • arcimage – desispec.image.Image object of arc

Mostly inherited from desispec/bin/desi_bootcalib directly as needed

Returns:

xfit, fdicts, gauss, all_wave_soln

TODO: document what those return objects are

desispec.bootcalib.extract_sngfibers_gaussianpsf(img, img_ivar, xtrc, sigma, box_radius=2, verbose=True)[source]

Extract spectrum for fibers one-by-one using a Gaussian PSF

Parameters:
  • img (ndarray) – Image

  • img_ivar (ndarray) – Image inverse variance

  • xtrc (ndarray) – fiber trace

  • sigma (float) – Gaussian sigma for PSF

  • box_radius (int, optional) – Radius for extraction (+/-)

Returns:

spec – Extracted spectrum

Return type:

ndarray

desispec.bootcalib.fiber_gauss_new(flat, xtrc, xerr, box_radius=2, max_iter=5, debug=False, verbose=False)[source]

Find the PSF sigma for each fiber This serves as an initial guess to what follows

Parameters:
  • flat (ndarray of fiber flat image) –

  • xtrc (ndarray of fiber traces) –

  • xerr (ndarray of error in fiber traces) –

  • box_radius (int, optinal) – Radius of boxcar extraction in pixels

  • max_iter (int, optional) – Maximum number of iterations for rejection

Returns:

list of Gaussian sigma

Return type:

gauss

desispec.bootcalib.fiber_gauss_old(flat, xtrc, xerr, box_radius=2, max_iter=5, debug=False, verbose=False)[source]

Find the PSF sigma for each fiber This serves as an initial guess to what follows

Parameters:
  • flat (ndarray of fiber flat image) –

  • xtrc (ndarray of fiber traces) –

  • xerr (ndarray of error in fiber traces) –

  • box_radius (int, optinal) – Radius of boxcar extraction in pixels

  • max_iter (int, optional) – Maximum number of iterations for rejection

Returns:

list of Gaussian sigma

Return type:

gauss

desispec.bootcalib.find_arc_lines(spec, rms_thresh=7.0, nwidth=5)[source]

Find and centroid arc lines in an input spectrum

Parameters:
  • spec (ndarray) – Arc line spectrum

  • rms_thresh (float) – RMS threshold scale

  • nwidth (int) – Line width to test over

desispec.bootcalib.find_fiber_peaks(flat, ypos=None, nwidth=5, debug=False, thresh=None)[source]

Find the peaks of the fiber flat spectra Preforms book-keeping error checking

Parameters:
  • flat – ndarray of fiber flat image

  • ypos – int [optional] Row for finding peaks Default is half-way up the image

  • nwidth – int [optional] Width of peak (end-to-end)

  • debug – bool, optional

Returns:

xpk, ypos, cut

list of xpk (nearest pixel) at ypos ndarray of cut through the image

desispec.bootcalib.fit_traces(xset, xerr, func='legendre', order=6, sigrej=20.0, RMS_TOLER=0.03, verbose=False)[source]

Fit the traces Default is 6th order Legendre polynomials

Parameters:
  • xset (ndarray) – traces

  • xerr (ndarray) – Error in the trace values (999.=Bad)

  • RMS_TOLER (float, optional [0.02]) – Tolerance on size of RMS in fit

Returns:

  • xnew, fits

  • xnew (ndarray) – New fit values (without error)

  • fits (list) – List of the fit dicts

desispec.bootcalib.fix_ycoeff_outliers(xcoeff, ycoeff, deg=5, tolerance=2)[source]

Fix outliers in coefficients for wavelength solution, assuming a continuous function of CCD coordinates

Parameters:
  • xcoeff[nfiber – 2D array of Legendre coefficients for X(wavelength)

  • ncoeff] – 2D array of Legendre coefficients for X(wavelength)

  • ycoeff[nfiber – 2D array of Legendre coefficients for Y(wavelength)

  • ncoeff] – 2D array of Legendre coefficients for Y(wavelength)

Options:

deg : integer degree of polynomial to fit tolerance : replace fibers with difference of wavelength solution larger than this number of pixels after interpolation

Returns:

new_ycoeff[nfiber, ncoeff] with outliers replaced by interpolations

For each coefficient, fit a polynomial vs. fiber number with one pass of sigma clipping. Remaining outliers are than replaced with the interpolated fit value.

desispec.bootcalib.id_arc_lines_using_triplets(id_dict, w, dwdy_prior, d2wdy2_prior=1.5e-05, toler=0.2, ntrack=50, nmax=40)[source]

Match (as best possible), a set of the input list of expected arc lines to the detected list

Parameters:
  • id_dict (dictionnary with Pixel locations of detected arc lines in "pixpk" and fluxes in "flux") –

  • w (ndarray) – array of expected arc lines to be detected and identified

  • dwdy (float) – Average dispersion in the spectrum

  • d2wdy2_prior (float) – Prior on second derivative

  • toler (float, optional) – Tolerance for matching (20%)

  • ntrack (max. number of solutions to be tracked) –

Returns:

id_dict – dict of identified lines

Return type:

dict

desispec.bootcalib.load_arcline_list(camera, vacuum=True, lamps=None)[source]

Loads arc line list from NIST files Parses and rejects

Taken from PYPIT

Parameters:
  • lines (list) – List of ions to load

  • vacuum (bool, optional) – Use vacuum wavelengths

  • lamps (optional numpy array of ions, ex np.array(["HgI","CdI","ArI","NeI"])) –

Returns:

alist – Table of arc lines

Return type:

Table

desispec.bootcalib.load_gdarc_lines(camera, llist, vacuum=True, lamps=None, good_lines_filename=None)[source]

Loads a select set of arc lines for initial calibrating

Parameters:
  • camera (str) – Camera (‘b’, ‘g’, ‘r’)

  • llist (table of lines to use, with columns Ion, wave) –

  • vacuum (bool, optional) – Use vacuum wavelengths

  • lamps (optional numpy array of ions, ex np.array(["HgI","CdI","ArI","NeI"])) –

Returns:

  • dlamb (float) – Dispersion for input camera

  • wmark (float) – wavelength to key off of [???]

  • gd_lines (ndarray) – Array of lines expected to be recorded and good for ID

  • line_guess (int or None) – Guess at the line index corresponding to wmark (default is to guess the 1/2 way point)

desispec.bootcalib.load_parse_dict()[source]

Dicts for parsing Arc line lists from NIST

Rejected lines are in the rejected_lines.yaml file

desispec.bootcalib.parse_nist(ion, vacuum=True)[source]

Parse a NIST ASCII table.

Note that the long —- should have been commented out and also the few lines at the start.

Taken from PYPIT

Parameters:
  • ion (str) – Name of ion.

  • vacuum (bool, optional) – Use vacuum wavelengths.

Returns:

A Table obtained from the data file with some columns added or renamed.

Return type:

astropy.table.Table

desispec.bootcalib.parse_nist_tbl(tbl, parse_dict)[source]

Parses a NIST table using various criteria

Parameters:
  • tbl (Table) – Read previously from NIST ASCII file

  • parse_dict (dict) – Dict of parsing criteria. Read from load_parse_dict

Returns:

tbl – Rows meeting the criteria

Return type:

Table

desispec.bootcalib.qa_arc_spec(all_spec, all_soln, pp, figsz=None)[source]

Generate QA plots of the arc spectra with IDs

Parameters:
  • all_spec – Arc 1D fiber spectra

  • all_soln – Wavelength solutions

  • pp – PDF file pointer

  • figsz – figure size, optional

desispec.bootcalib.qa_fiber_Dx(xfit, fdicts, pp=None, figsz=None)[source]

Show the spread in the trace per fiber

Used to diagnose the traces

Parameters:
  • xfit – traces

  • fdicts – dict of the traces

  • pp – PDF file pointer

  • figsz – figure size, optional

desispec.bootcalib.qa_fiber_arcrms(all_soln, pp, figsz=None)[source]

Show the RMS of the wavelength solutions vs. fiber

Parameters:
  • all_soln – Wavelength solutions

  • pp – PDF file pointer

  • figsz – figure size, optional

desispec.bootcalib.qa_fiber_dlamb(all_spec, all_soln, pp, figsz=None)[source]

Show the Dlamb of the wavelength solutions vs. fiber

Parameters:
  • all_soln – Wavelength solutions

  • pp – PDF file pointer

  • figsz – figure size, optional

desispec.bootcalib.qa_fiber_gauss(gauss, pp=None, figsz=None)[source]

Show the Gaussian (sigma) fits to each fiber

Parameters:
  • gauss – Gaussian of each fiber

  • pp – PDF file pointer

  • figsz – figure size, optional

desispec.bootcalib.qa_fiber_peaks(xpk, cut, pp=None, figsz=None, nper=100)[source]

Generate a QA plot for the fiber peaks

Parameters:
  • xpk – x positions on the CCD of the fiber peaks at a ypos

  • cut – Spatial cut through the detector

  • pp – PDF file pointer

  • figsz – figure size, optional

  • nper – number of fibers per row in the plot, optional

desispec.bootcalib.qa_fiber_trace(flat, xtrc, outfil=None, Nfiber=25, isclmin=0.5)[source]

Generate a QA plot for the fiber traces

Parameters:
  • flat (ndarray) – image

  • xtrc (ndarray) – Trace array

  • isclmin (float, optional [0.5]) – Fraction of 90 percentile flux to scale image by

  • outfil (str, optional) – Output file

  • normalize (bool, optional) – Normalize the flat? If not, use zscale for output

desispec.bootcalib.reject_lines(tbl, rej_dict, rej_tol=0.1)[source]

Rejects lines from a NIST table

Taken from PYPIT

Parameters:
  • tbl (Table) – Read previously from NIST ASCII file

  • rej_dict (dict) – Dict of rejected lines

  • rej_tol (float, optional) – Tolerance for matching a line to reject to linelist (Angstroms)

Returns:

tbl – Rows not rejected

Return type:

Table

desispec.bootcalib.script_bootcalib(arc_idx, flat_idx, cameras=None, channels=None, nproc=10)[source]

Runs desi_bootcalib on a series of preproc files

Returns:

script_bootcalib([0,1,2,3,4,5,6,7,8,9], [10,11,12,13,14])

desispec.bootcalib.trace_crude_init(image, xinit0, ypass, invvar=None, radius=2.0, maxshift0=0.5, maxshift=0.15, maxerr=0.2)[source]

Python port of trace_crude_idl.pro from IDLUTILS

Modified for initial guess

Parameters:
  • image (2D ndarray) – Image for tracing

  • xinit (ndarray) – Initial guesses for trace peak at ypass

  • ypass (int) – Row for initial guesses

Returns:

  • xset (Trace for each fiber)

  • xerr (Estimated error in that trace)

desispec.bootcalib.trace_fweight(fimage, xinit, ycen=None, invvar=None, radius=2.0, debug=False)[source]

Python port of trace_fweight.pro from IDLUTILS

Parameters:
  • fimage (2D ndarray) – Image for tracing

  • xinit (ndarray) – Initial guesses for x-trace

  • invvar (ndarray, optional) – Inverse variance array for the image

  • radius (float, optional) – Radius for centroiding; default to 3.0

desispec.bootcalib.write_psf(outfile, xfit, fdicts, gauss, wv_solns, legendre_deg=5, without_arc=False, XCOEFF=None, fiberflat_header=None, arc_header=None, fix_ycoeff=True)[source]

Write the output to a Base PSF format

Parameters:
  • outfile (str) – Output file

  • xfit (ndarray) – Traces

  • gauss (list) – List of gaussian sigmas

  • fdicts (list) – List of trace fits

  • wv_solns (list) – List of wavelength calibrations

  • ncoeff (int) – Number of Legendre coefficients in fits

desispec.calibfinder

Reading and selecting calibration data from $DESI_SPECTRO_CALIB using content of image headers

desispec.calibfinder._load_smsp()[source]

Loads $DESI_SPECTRO_CALIB/spec/smsp.txt into global _sp2sm and _sm2sp dicts

desispec.calibfinder.badfibers(headers, keys=['BROKENFIBERS', 'BADCOLUMNFIBERS', 'LOWTRANSMISSIONFIBERS'], yaml_file=None)[source]

find list of bad fibers from $DESI_SPECTRO_CALIB using the keywords found in the headers

Parameters:

headers – list of fits headers, or list of dictionnaries

Optional:

keys: list of keywords, among [“BROKENFIBERS”,”BADCOLUMNFIBERS”,”LOWTRANSMISSIONFIBERS”]. Default is all of them. yaml_file: path to a specific yaml file. By default, the code will automatically find the yaml file from the environment variable DESI_SPECTRO_CALIB and the CAMERA keyword in the headers

Returns List of bad fibers as a 1D array of intergers

desispec.calibfinder.ccdregionmask(headers)[source]

Looks for regions of CCD to mask for a given NIGHT EXPID CAMERA and returns a list of dictionnaries. NIGHT EXPID CAMERA are retrieved from the input image headers and compared to corresponding columns in the cvs table $DESI_SPECTRO_CALIB/ccd/ccd-region-mask.csv

Parameters:

headers – list of fits headers, or list of dictionnaries

Returns list of dictionnaries with keys XMIN, XMAX, YMIN,YMAX

desispec.calibfinder.findcalibfile(headers, key, yaml_file=None)[source]

read and select calibration data file from $DESI_SPECTRO_CALIB using the keywords found in the headers

Parameters:
  • headers – list of fits headers, or list of dictionnaries

  • key – type of calib file, e.g. ‘PSF’ or ‘FIBERFLAT’

Optional:

yaml_file: path to a specific yaml file. By default, the code will automatically find the yaml file from the environment variable DESI_SPECTRO_CALIB and the CAMERA keyword in the headers

Returns path to calibration file

desispec.calibfinder.parse_date_obs(value)[source]

converts DATE-OBS keywork to int with for instance DATE-OBS=2016-12-21T18:06:21.268371-05:00

desispec.calibfinder.sm2sp(sm, night=None)[source]

Converts spectrograph sm hardware number to sp logical number

Parameters:

sm – spectrograph sm number 1-10 or str sm[1-10]

Returns “spP” if input is str “smM”, or int P if input is int M

Note: uses $DESI_SPECTRO_CALIB/spec/smsp.txt

TODO: add support for different mappings based on night

desispec.calibfinder.sp2sm(sp)[source]

Converts spectrograph sp logical number to sm hardware number

Parameters:

sp – spectrograph int 0-9 or str sp[0-9]

Returns “smM” if input is str “spP”, or int M if input is int P

Note: uses $DESI_SPECTRO_CALIB/spec/smsp.txt

TODO: add support for different mappings based on night

desispec.ccdcalib

desispec.ccdcalib._find_zeros(night, cameras, nzeros=25, nskip=2)[source]

Find all OBSTYPE=ZERO exposures on a given night

Parameters:
  • night (int) – YEARMMDD night to search

  • cameras (list of str) – list of cameras to process

Options:

nzeros (int): number of zeros desired from valid all-cam observations to not worry about partials nskip (int): number of initial zeros to skip

Returns:

dictionary of expids (values) for each camera (keys)

for calibration zeros

noncalib_expdict (dict): dictionary of expids (values) for each camera (keys)

for non calibration zeros.

Return type:

calib_expdict (dict)

Uses production exposure tables to veto known bad ZEROs, but it will also find any ZEROs on disk for that night, regardless of whether they are in the exposures table or not.

desispec.ccdcalib.compare_bias(rawfile, biasfile1, biasfile2, ny=8, nx=40)[source]

Compare rawfile image to bias images in biasfile1 and biasfile2

Parameters:
  • rawfile – full filepath to desi*.fits.fz raw data file

  • biasfile1 – filepath to bias model made from OBSTYPE=ZERO exposures

  • biasfile2 – filepath to bias model made from OBSTYPE=ZERO exposures

Options:

ny (even int): number of patches in y (row) direction nx (even int): number of patches in x (col) direction

Returns tuple (mdiff1[ny,nx], mdiff2[ny,nx]) median(image-bias) in patches

The rawfile camera is derived from the biasfile CAMERA header keyword.

median(raw-bias) is calculated in ny*nx patches using only the DATASEC portion of the images. Since the DESI CCD bias features tend to vary faster with row than column, default patches are (4k/8 x 4k/40) = 500x100.

desispec.ccdcalib.compare_dark(preprocfile1, preprocfile2, ny=8, nx=40)[source]

Compare preprocessed dark images based on different dark models

Parameters:
  • preprocfile1 – filepath to bias model made from OBSTYPE=ZERO exposures

  • preprocfile2 – filepath to bias model made from OBSTYPE=ZERO exposures

Options:

ny (even int): number of patches in y (row) direction nx (even int): number of patches in x (col) direction

Returns tuple (mdiff1[ny,nx], mdiff2[ny,nx]) median(image-bias) in patches

median(raw-bias) is calculated in ny*nx patches using only the DATASEC portion of the images. Since the DESI CCD bias features tend to vary faster with row than column, default patches are (4k/8 x 4k/40) = 500x100.

desispec.ccdcalib.compute_bias_file(rawfiles, outfile, camera, explistfile=None, extraheader=None)[source]

Compute a bias file from input ZERO rawfiles

Parameters:
  • rawfiles – list of input raw file names

  • outfile (str) – output filename

  • camera (str) – camera, e.g. b0, r1, z9

Options:

explistfile: filename with text list of NIGHT EXPID to use extraheader: dict-like key/value header keywords to add

Notes: explistfile is only used if rawfiles=None; it should have one NIGHT EXPID entry per line.

desispec.ccdcalib.compute_dark_file(rawfiles, outfile, camera, bias=None, nocosmic=False, scale=False, exptime=None)[source]

Compute classic dark model from input dark images

Parameters:
  • rawfiles (list of str) – list of input raw data files (desi-*.fits.fz)

  • outfile (str) – output file with dark model to write

  • camera (str) – camera to process, e.g. b0, r1, z9

Options:

bias (str or list): bias file to use, or list of bias files nocosmic (bool): use medians instead of cosmic identification scale (bool): apply scale correction for EM0 teststand data exptime (float): write EXPTIME header keyword; all inputs must match

Note: if bias is None, no bias correction is applied. If it is a single file, then use that bias for all darks. If it is a list, it must have len(rawfiles) and gives the per-file bias to use.

Note: this computes a classic dark model without any non-linear terms. see bin/compute_dark_nonlinear for current DESI dark model.

TODO: separate algorithm from I/O

desispec.ccdcalib.compute_nightly_bias(night, cameras, outdir=None, nzeros=25, minzeros=15, nskip=2, anyzeros=False, comm=None)[source]

Create nightly biases for cameras on night

Parameters:
  • night (int) – YEARMMDD night to process

  • cameras (list of str) – list of cameras to process

Options:

outdir (str): write files to this output directory nzeros (int): number of OBSTYPE=ZERO exposures to use minzeros (int): minimum number of OBSTYPE=ZERO exposures required nskip (int): number of initial zeros to skip anyzeros (bool): allow any ZEROs, not just those taken for CCD calib seq comm: MPI communicator for parallelism

Returns:

number of cameras that failed across all ranks

Return type:

nfail (int)

Writes biasnight*.fits files in outdir or $DESI_SPECTRO_REDUX/$SPECPROD/calibnight/night/

Note: compute_bias_file requires ~12 GB memory per camera, so limit the size of the MPI communicator depending upon the memory available.

desispec.ccdcalib.fit_const_plus_dark(exp_arr, image_arr)[source]

fit const + dark*t model given images and exptimes

Parameters:
  • exp_arr – list of exposure times

  • image_arr – list of average dark images

returns: (const, dark) tuple of images

NOTE: the image_arr should not be divided by the exposure time

desispec.ccdcalib.fit_dark(exp_arr, image_arr)[source]

fit dark*t model given images and exptimes

Parameters:
  • exp_arr – list of exposure times

  • image_arr – list of average dark images

returns: dark_image

NOTE: the image_arr should not be divided by the exposure time

desispec.ccdcalib.make_dark_scripts(outdir, days=None, nights=None, cameras=None, linexptime=None, nskip_zeros=None, tempdir=None, nosubmit=False, first_expid=None, night_for_name=None, use_exptable=True, queue='realtime', copy_outputs_to_split_dirs=False, prepared_exptable=None, system_name=None, min_vccdsec=None)[source]

Generate batch script to run desi_compute_dark_nonlinear

Parameters:
  • outdir (str) – output directory

  • nights (days or) – days or nights to include

Options:

cameras (list of str): cameras to include, e.g. b0, r1, z9 linexptime (float): exptime after which dark current is linear nskip_zeros (int): number of ZEROs at beginning of day/night to skip tempdir (str): tempfile working directory nosubmit (bool): generate scripts but don’t submit them to batch queue first_expid (int): ignore expids prior to this use_exptable (bool): use shortened copy of joined exposure tables instead of spectable (need to have right $SPECPROD set) queue (str): which batch queue to use for submission copy_outputs_to_split_dirs (bool): whether to copy outputs to bias_frames/dark_frames subdirs prepared_exptable (exptable): if a table is submitted here, no further spectra will be searched and this will be used instead system_name (str): the system for which batch files should be created, defaults to guessing current system min_vccdsec(str): minimum time a ccd needs to be turned on to be allowed in the model

Args/Options are passed to the desi_compute_dark_nonlinear script

desispec.ccdcalib.make_regular_darks(outdir=None, lastnight=None, cameras=None, window=30, linexptime=None, nskip_zeros=None, tempdir=None, nosubmit=False, first_expid=None, night_for_name=None, use_exptable=True, queue='realtime', copy_outputs_to_split_dirs=None, transmit_obslist=True, system_name=None, no_obslist=False, min_vccdsec=None)[source]

Generate batch script to run desi_compute_dark_nonlinear

Options:

outdir (str): output directory lastnight (int): last night to take into account (inclusive), defaults to tonight

window (int): length of time window to take into account cameras (list of str): cameras to include, e.g. b0, r1, z9 linexptime (float): exptime after which dark current is linear nskip_zeros (int): number of ZEROs at beginning of day/night to skip tempdir (str): tempfile working directory nosubmit (bool): generate scripts but don’t submit them to batch queue first_expid (int): ignore expids prior to this use_exptable (bool): use shortened copy of joined exposure tables instead of spectable (need to have right $SPECPROD set) queue (str): which batch queue to use for submission transmit_obslist(bool): if True will give use the obslist from here downstream system_name(str): allows to overwrite the system for which slurm scripts are created, will default to guessing the current system no_obslist(str): just use exactly the specified night-range, but assume we do not have exposure tables for this (useful when there is no exposure_table yet) min_vccdsec(float): minimum time a ccd needs to be turned on to be allowed in the model (default: 21600=6h)

Args/Options are passed to the desi_compute_dark_nonlinear script

desispec.ccdcalib.model_y1d(image, smooth=0)[source]

Model image as a sigma-clipped mean 1D function of row

Parameters:

image – 2D array to model

Options:

smooth (int): if >0, Savitzky-Golay filter curve by this window

Returns 1D model of image with len = image.shape[0]

desispec.ccdcalib.select_zero_expids(calib_exps, noncalib_exps, night, cam, nzeros=25, minzeros=15, nskip=2, anyzeros=False)[source]

Select which ZERO exposure IDs to use for nightly bias

Parameters:
  • calib_exps (array-like) – expids of calibration zeros

  • noncalib_exps (array-like) – expids of non calibration zeros

  • night (int) – YEARMMDD night to process

  • cam (str) – camera to process

Options:

nzeros (int): number of zeros desired from valid all-cam observations to not worry about partials minzeros (int): minimum number of zeros to sufficiently compute a nightly bias nskip (int): number of initial zeros to skip

Returns:

the list of expids selected

Return type:

expids (array)

desispec.coaddition

Coadd spectra.

desispec.coaddition.calc_mean_std_ra_dec(ras, decs)[source]

Calculate mean/std of ras, decs accounting for RA wraparound and cos(dec)

Parameters:
  • ras (array) – input RA values in degrees

  • decs (array) – input declination values in degrees

Returns: mean_ra, std_ra, mean_dec, std_dec

where the means are in degrees and the standard deviations are in arcsec, including cos(dec) correction.

For efficiency, this does not try to handle dec= +/-90 poles correctly, nor arbitrarily large spreads of angles. i.e. this is ok for a mean of fiber positions scattered about a single target, but not for e.g. a science analysis of the central location of a cluster of galaxies.

desispec.coaddition.coadd(spectra, cosmics_nsig=None, onetile=False)[source]

Coadd spectra for each target and each camera, modifying input spectra obj.

Parameters:

spectra – desispec.spectra.Spectra object

Options:

cosmics_nsig: float, nsigma clipping threshold for cosmics rays (default 4) onetile: bool, if True, inputs are from a single tile

Notes: if onetile is True, additional tile-specific columns

like LOCATION and FIBER are included the FIBERMAP; otherwise these are only in the EXP_FIBERMAP since for the same target they could be different on different tiles.

desispec.coaddition.coadd_cameras(spectra, cosmics_nsig=0.0, onetile=False)[source]

Return coadd across both exposures and cameras

Parameters:

spectra – desispec.spectra.Spectra object

Options:

cosmics_nsig: float, nsigma clipping threshold for cosmics rays onetile: bool, if True, inputs are from a single tile

If onetile is True, additional tile-specific columns like LOCATION and FIBER are included the FIBERMAP; otherwise these are only in the EXP_FIBERMAP since for the same target they could be different on different tiles.

Note: unlike coadd, this does not modify the input spectra object

desispec.coaddition.coadd_fibermap(fibermap, onetile=False)[source]

Coadds fibermap

Parameters:

fibermap (Table or ndarray) – fibermap of individual exposures

Options:

onetile (bool): this is a coadd of a single tile, not across tiles

Returns: (coadded_fibermap, exp_fibermap) Tables

coadded_fibermap contains the coadded_fibermap for the columns that can be coadded, while exp_fibermap is the subset of columns of the original fibermap that can’t be meaningfully coadded because they are per-exposure quantities like FIBER_X.

If onetile is True, the coadded_fibermap includes additional columns like MEAN_FIBER_X that are meaningful if coadding a single tile, but not if coadding across tiles.

desispec.coaddition.decorrelate_divide_and_conquer(Cinv, Cinvf, wavebin, flux, ivar, rdata)[source]

Decorrelate an inverse covariance using the matrix square root.

Implements the decorrelation part of the spectroperfectionism algorithm described in Bolton & Schlegel 2009 (BS) http://arxiv.org/abs/0911.2689.

with the divide and conquer approach, i.e. per diagonal block of the matrix, with an overlapping ‘skin’ from one block to another.

Parameters:
  • Cinv – Square 2D array: input inverse covariance matrix

  • Cinvf – 1D array: input

  • wavebin – minimal size of wavelength bin in A, used to define the core and skin size

  • flux – 1D array: output flux (has to be allocated)

  • ivar – 1D array: output flux inverse variance (has to be allocated)

  • rdata – 2D array: output resolution matrix per diagonal (has to be allocated)

desispec.coaddition.fast_resample_spectra(spectra, wave)[source]

Fast resampling of spectra file. The output resolution = Id. The neighboring flux bins are correlated.

Parameters:
  • spectra – desispec.spectra.Spectra object

  • wave – 1D numy array with new wavelenght grid

Returns:

desispec.spectra.Spectra object, resolution data=Id

desispec.coaddition.get_resampling_matrix(global_grid, local_grid, sparse=False)[source]

Build the rectangular matrix that linearly resamples from the global grid to a local grid.

The local grid range must be contained within the global grid range.

Parameters:
  • global_grid (numpy.ndarray) – Sorted array of n global grid wavelengths.

  • local_grid (numpy.ndarray) – Sorted array of m local grid wavelengths.

Returns:

Array of (m,n) matrix elements that perform the linear resampling.

Return type:

numpy.ndarray

desispec.coaddition.resample_spectra_lin_or_log(spectra, linear_step=0, log10_step=0, fast=False, wave_min=None, wave_max=None, nproc=1)[source]

Resampling of spectra file.

Parameters:
  • spectra – desispec.spectra.Spectra object

  • linear_step – if not null the ouput wavelenght grid will be linear with this step

  • log10_step – if not null the ouput wavelenght grid will be logarthmic with this step

Options:

fast: simple resampling. fast but at the price of correlated output flux bins and no information on resolution wave_min: if set, use this min wavelength wave_max: if set, use this max wavelength

Returns:

desispec.spectra.Spectra object

desispec.coaddition.spectroperf_resample_spectra(spectra, wave, nproc=1)[source]

Resampling of spectra file using the spectrophotometic approach

Parameters:
  • spectra – desispec.spectra.Spectra object

  • wave – 1D numy array with new wavelenght grid

Returns:

desispec.spectra.Spectra object

desispec.coaddition.use_for_coadd(fiberstatus, band)[source]

Determine which exposures should be used for a per-camera coadd

Parameters:
  • fiberstatus (array) – FIBERSTATUS bitmasks, one per exposure

  • band (str) – camera band, ‘b’, ‘r’, or ‘z’

Returns: boolean array of whether exposure should be used in coadd or not

This is factored into a separate function because it is used in coadd, coadd_cameras, and coadd_fibermap

desispec.correct_cte

Methods to fit CTE effects and remove them from images.

desispec.correct_cte.add_cte(img, cte_transfer_func=None, **cteparam)[source]

Add CTE to a region affected by CTE.

Assumes all of img is affected.

Parameters:
  • img (np.ndarray) – input image

  • cte_transfer_func (callable) – function describing how many electrons trap soaks up and emits

  • **cteparam (dictionary) – additional arguments to cte_transfer_func

Returns:

  • np.ndarray

  • img with additional CTE applied

desispec.correct_cte.apply_multiple_cte_effects(amp, locations, ctefuns)[source]

Apply a series of traps to an amp.

This function assumes that it is only being given a single amp that has some traps on it. The readout direction is always assumed to be toward smaller numbers on the second axis.

Parameters:
  • amp (np.ndarray) – image of amplifier

  • locations (np.ndarray [int]) – locations of traps

  • ctefuns (callables) – functions that can be called to apply CTE to an amp

Return type:

amplifier after applying traps

desispec.correct_cte.chi_simplified_regnault(param, cleantraces=None, ctetraces=None, uncertainties=None)[source]

Chi loss function for a Regnault transfer function.

desispec.correct_cte.correct_amp(image, ctefun, niter=1, apply_only=False)[source]

Correct an amp for CTE.

In the limit that CTE is perturbative, one can correct for CTE as follows: for i in range(niter): I’ = CTE(I) I = I - (I’ - I) This function implements that approach.

However, for large traps, this approach correlates the noise and leads to poor performance; we do not use it for DESI.

Parameters:
  • image (np.ndarray) – CTE-affected amplifier

  • ctefun (callable) – function that applies CTE to an amplifier

  • niter (int) – number of iterations to run

  • apply_only (bool) – Do not correct, just apply the CTE function to the image.

Return type:

CTE-corrected image

desispec.correct_cte.correct_image(image, **kw)[source]

Correct an image for CTE.

This function wraps correct_amp, looping over amplifiers and reversing readout directions as appropriate. The approach taken here correlates the noise and is not actually used for DESI.

Parameters:
  • image (desispec.io.image.Image) – the image to correct

  • cteparam (dict or None) – dictionary of form {‘z1C’: {‘543:2057’: (115.0, 0.21)}} that provides the CTE parameters for the trap on a particular image.

  • **kw (dict) – additional arguments passed to correct_amp

Returns:

  • outimage (desispec.io.image.Image)

  • CTE-corrected image

desispec.correct_cte.correct_image_via_model(image, niter=10, cteparam=None)[source]

Correct for CTE via an image model.

The idea here is that you can roughly extract spectra from a CTE-affected image just by extracting as usual. You can then create a noise-free image from that extraction in the context of a PSF and traces. You can then apply CTE to that noise-free image. The difference between the CTE-affected image and the original image is a noise-free version of what CTE is doing to your data, which you can then subtract.

You can then re-extract from the corrected image, and repeat, improving the CTE correction.

As pseudocode, this corresponds to: for i in range(niter): M = get_model(I) M’ = CTE(M) I = I - (M’ - M) This function implements that approach.

Parameters:
  • image (Image) – input image

  • niter (int) – number of iterations to run

  • cteparam (dict or None) – dictionary of form {‘z1C’: {‘543:2057’: (115.0, 0.21)}} that provides the CTE parameters for the trap on a particular image.

Returns:

outimage – image after correction for CTE

Return type:

Image

desispec.correct_cte.fit_cte(images)[source]

Fits CTE models to a list of images.

This fits the parameters of the Regnault transfer function model to an image with a trap. It works by comparing an unaffected amplifier to an amplifier with a trap along the boundary between the two amplifiers, solving for the trap parameters that make the unaffected image + the CTE effect a good match to the CTE-affected image.

It will not work if both amplifiers have CTE, or if there are multiple CTE effects to solve for.

It assumes that amp A is across from amp C and amp B is across from amp D.

It would likely fail badly for a two-amp mode image.

Parameters:

images (list) – A list of images. Usually these are all flats with different exposure lengths on the same device.

Returns:

  • dict[dict[tuple(tuple, float)]]

  • The top-level dictionary has amplifier names as keys.

  • The next-level dictionary has a string identifying the specific trap as

  • keys.

  • The tuple contains the parameters of the CTE function as its first

  • element, and the chi^2 per degree of freedom of the fit as its second

  • element.

desispec.correct_cte.fit_cte_night(night, camera)[source]

Fit the CTE parameters for a particular night.

Parameters:
  • night (int) – the night YEARMMDD integer

  • camera (str) – the camera, e.g., z1

Return type:

Fit results; see fit_cte for details.

desispec.correct_cte.get_amps_and_cte(image, cteparam=None)[source]

Get amp and CTE information from image metadata.

Parameters:
  • image (object) – Must have ‘meta’ attribute.

  • cteparam (dict or None) – dictionary of form {‘z1C’: {‘543:2057’: (115.0, 0.21)}} that provides the CTE parameters for the trap on a particular image.

Returns:

  • amp_regions, cte_regions

  • amp_regions (dict) – dictionary of slices defining each amp

  • cte_regions (dict) – dictionary. The keys are the names of the amplifier. Each entry is a list containing one entry for each trap. on that amplifier. Each item in the list is dictionary containing the start location, stop location, and a callable giving the CTE function for a trap.

desispec.correct_cte.get_cte_images(night, camera)[source]

Get the images needed for a CTE fit for a particular night.

This function looks up the appropriate exposure tables to find the CTE-detection image and the previous image, which is usually a normal flat field image.

Parameters:
  • night (int) – the night YEARMMDD integer

  • camera (str) – the camera, e.g., z1

Return type:

Fit results; see fit_cte for details.

desispec.correct_cte.get_image_model(preproc, psf=None)[source]

Compute model for an image using aperture extraction.

This computes a simple model for an image based on an aperture extraction.

Parameters:

preproc (Image) – Image to model

Returns:

  • np.ndarray

  • Model image

desispec.correct_cte.get_rowbyrow_image_model(preproc, fibermap=None, spectral_smoothing_sigma_length=31, nspec=500, psf=None)[source]

Compute row-by-row image model.

This model uses a simultaneous PSF fit in each row to get better performance than get_image_model at the expense of reduced speed. The extracted fluxes are then combined with the PSF to produce a 2D model image.

Parameters:
  • preproc (Image) – image to model

  • fibermap (astropy.Table) – fibermap to use with image

  • spectral_smoothing_sigma_length (int) – amount to smooth source spectra in model

  • nspec (int) – number of spectra to extract and model

  • psf (specter.psf.gausshermite.GaussHermitePSF) – PSF to use

Returns:

  • np.ndarray

  • Model image.

desispec.correct_cte.simplified_regnault(pixval, in_trap, amplitude, fracleak)[source]

CTE transfer function of Regnault+.

The model is transfer = in_trap * fracleak - pixval * (1 - in_trap / amplitude) with slight elaborations to prevent more electrons than exist from entering or leaving the trap.

Parameters:
  • pixval (float) – value of uncorrupted image

  • in_trap (float) – number of electrons presently in trap

  • amplitude (float) – amplitude of trap

  • fracleak (float) – fraction of electrons that leak out of trap each transfer

Returns:

  • int

  • number of electrons that leak out of the trap into the pixel

desispec.cosmics

Utility functions to find cosmic rays

desispec.cosmics._reject_cosmic_rays_ala_sdss_single(pix, ivar, selection, psf_gradients, nsig, cfudge, c2fudge)[source]

Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)

This routine is a single call, similar to IDL routine reject_cr_single Called by reject_cosmic_rays_ala_sdss with an iteration

Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image

A faster version of this routine using numba in implemented in _reject_cosmic_rays_ala_sdss_single_numba

Parameters:
  • pix – input desispec.Image.pix (counts in pixels, 2D image)

  • ivar – inverse variance of pix

  • selection – array of booleans

  • psf_gradients – 1D array of size 4, for 4 axes: horizontal,vertical and 2 diagonals

  • nsig – number of sigma above background required

  • cfudge – number of sigma inconsistent with PSF required

  • c2fudge – fudge factor applied to PSF

desispec.cosmics._reject_cosmic_rays_ala_sdss_single_numba(pix, ivar, selection, psf_gradients, nsig, cfudge, c2fudge)

Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)

This routine is a single call, similar to IDL routine reject_cr_single Called by reject_cosmic_rays_ala_sdss with an iteration

Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image

This routine is much faster than _reject_cosmic_rays_ala_sdss_single (if you have numba installed, otherwise it is catastrophically slower)

Parameters:
  • pix – input desispec.Image.pix (counts in pixels, 2D image)

  • ivar – inverse variance of pix

  • selection – array of booleans

  • psf_gradients – 1D array of size 4, for 4 axes: horizontal,vertical and 2 diagonals

  • nsig – number of sigma above background required

  • cfudge – number of sigma inconsistent with PSF required

  • c2fudge – fudge factor applied to PSF

desispec.cosmics.reject_cosmic_rays(img, nsig=5.0, cfudge=3.0, c2fudge=0.9, niter=100, dilate=True)[source]

Cosmic ray rejection Input is a pre-processed image : desispec.Image The image mask is modified

Parameters:

img – input desispec.Image

desispec.cosmics.reject_cosmic_rays_1d(frame, nsig=3, psferr=0.05)[source]

Use resolution matrix in frame to detect spikes in the spectra that are narrower than the PSF, and mask them

desispec.cosmics.reject_cosmic_rays_ala_sdss(img, nsig=6.0, cfudge=3.0, c2fudge=0.5, niter=6, dilate=True)[source]

Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)

This routine is calling several times reject_cosmic_rays_ala_sdss_single, similar to IDL routine reject_cr. There is an optionnal dilatation of the mask by one pixel, as done in sdssproc.pro for SDSS

Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image

Parameters:
  • img – input desispec.Image

  • nsig – number of sigma above background required

  • cfudge – number of sigma inconsistent with PSF required

  • c2fudge – fudge factor applied to PSF

  • niter – number of iterations on neighboring pixels of rejected pixels

  • dilate – force +1 pixel dilation of rejection mask

desispec.darktrail

Utility functions to correct dark trails found for some LBNL CCD amplifiers due to an imperfect CDS (correlated double sampling) suppression of the reset level. The cause of the effect is unknown (as this is written), and we have not been able to correcting it with adjustment of the readout electronic parameters.

The function here are used to do a software correction of the effect.

desispec.darktrail.correct_dark_trail(image, xyslice, left, width, amplitude)[source]

remove the dark trails from a preprocessed image with a processing of the form:

image[j,i] += sum_{k<i} image[j,k] * a * |i-k|/width * exp( -|i-k|/width)
Parameters:
  • image – desispec.Image class instance

  • xyslice – tuple of python slices (yy,xx) where yy is indexing CCD rows (firt index in 2D numpy array) and xx is the other

  • left – if true , the trails are to the left of the image spots, as is the case for amplifiers B and D.

  • width – width parameter in unit of pixels

  • amplitude – amplitude of the dark trail

desispec.database

Tools for loading DESI data into databases.

At the present time, this covers:

  1. Raw metadata database.

  2. Redshift results database (a.k.a “SpectraDB”).

It does not include:

  1. Pipeline processing status database.

  2. Imaging and targeting databases.

desispec.database.duplicates

Find rows in a file that contain the same value in a certain column. The file in question could potentially be very large.

desispec.database.duplicates.find_duplicate_rows(data, column)[source]

Find rows in data for which column has the same value.

Parameters:
Returns:

A mapping of unique values of column to row numbers in data.

Return type:

dict

desispec.database.duplicates.get_options(*args)[source]

Parse command-line options.

Parameters:

args (iterable) – If arguments are passed, use them instead of sys.argv.

Returns:

The parsed options.

Return type:

argparse.Namespace

desispec.database.duplicates.main()[source]

Entry point for command-line script.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desispec.database.metadata

Code for interacting with the file metadatabase.

class desispec.database.metadata.Brick(*args: Any, **kwargs: Any)[source]

Representation of a region of the sky.

class desispec.database.metadata.BrickStatus(*args: Any, **kwargs: Any)[source]

Representation of the status of a particular Brick.

class desispec.database.metadata.ExposureFlavor(*args: Any, **kwargs: Any)[source]

List of exposure flavors. Used to constrain the possible values.

class desispec.database.metadata.Frame(*args: Any, **kwargs: Any)[source]

Representation of a particular pointing, exposure, spectrograph and band (a.k.a. ‘channel’ or ‘arm’).

class desispec.database.metadata.FrameStatus(*args: Any, **kwargs: Any)[source]

Representation of the status of a particular Frame.

class desispec.database.metadata.Night(*args: Any, **kwargs: Any)[source]

List of observation nights. Used to constrain the possible values.

class desispec.database.metadata.Status(*args: Any, **kwargs: Any)[source]

List of possible processing statuses.

class desispec.database.metadata.Tile(*args: Any, **kwargs: Any)[source]

Representation of a particular pointing of the telescope.

_coarse_overlapping_bricks(session)[source]

Get the bricks that may overlap a tile.

Parameters:

session (sqlalchemy.orm.session.Session) – Database connection.

Returns:

A list of Brick objects.

Return type:

list

_constants()[source]

Define mathematical constants associated with a tile.

property area

Area of the tile in steradians.

brick_offset(brick)[source]

Offset a brick in the same way as a tile.

Parameters:

brick (Brick) – A brick.

Returns:

A tuple containing the shifted ra1 and ra2.

Return type:

tuple()

property circum_square

Defines a square-like region on the sphere which circumscribes the tile.

property cos_radius

Cosine of the radius, precomputed for speed.

offset(shift=10.0)[source]

Provide an offset to move RA away from wrap-around.

Parameters:

shift (float, optional) – Amount to offset in degrees.

Returns:

An amount to offset in degrees.

Return type:

float

overlapping_bricks(session, map_petals=False)[source]

Perform a geometric calculation to find bricks that overlap a tile.

Parameters:
  • session (sqlalchemy.orm.session.Session) – Database connection.

  • map_petals (bool, optional) – If True a map of petal number to a list of overlapping bricks is returned.

Returns:

If map_petals is False, a list of Polygon objects. Otherwise, a dict mapping petal number to the Brick objects that overlap that petal.

Return type:

list

petals(Npetals=10)[source]

Convert a tile into a set of Wedge objects.

Parameters:

Npetals (int, optional) – Number of petals (default 10).

Returns:

A list of Wedge objects.

Return type:

list

property radius

Radius of tile in degrees.

simulate_frame(session, band, spectrograph, flavor='science', exptime=1000.0)[source]

Simulate a DESI frame given a Tile object.

Parameters:
  • session (sqlalchemy.orm.session.Session) – Database connection.

  • band (str) – ‘b’, ‘r’, ‘z’

  • spectrograph (int) – Spectrograph number [0-9].

  • flavor (str, optional) – Exposure flavor (default ‘science’).

  • exptime (float, optional) – Exposure time in seconds (default 1000).

Returns:

A tuple containing a Frame object ready for loading, and a list of bricks that overlap.

Return type:

tuple

desispec.database.metadata.get_all_tiles(session, obs_pass=0, limit=0)[source]

Get all tiles from the database.

Parameters:
  • session (sqlalchemy.orm.session.Session) – Database connection.

  • obs_pass (int, optional) – Select only tiles from this pass.

  • limit (int, optional) – Limit the number of tiles returned

Returns:

A list of Tiles.

Return type:

list

desispec.database.metadata.load_data(session, datapath)[source]

Load a night or multiple nights into the frame table.

Parameters:
  • session (sqlalchemy.orm.session.Session) – Database connection.

  • datapath (str) – Name of a data directory.

Returns:

A list of the exposure numbers found.

Return type:

list

desispec.database.metadata.load_simulated_data(session, obs_pass=0)[source]

Load simulated frame and brick data.

Parameters:
  • session (sqlalchemy.orm.session.Session) – Database connection.

  • obs_pass (int, optional) – If set, only simulate one pass.

desispec.database.metadata.main()[source]

Entry point for command-line script.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desispec.database.redshift

Code for loading a spectroscopic production database. This includes both targeting and redshift data.

Notes

  • EDR development:

    • Load as much data as possible from patched fiberassign files.

    • If necessary, create intermediate files for ingestion by running the lsdr9-photometry code.

    • In the tractorphot files, TARGETID really is unique. However, some TARGETIDs are not present because they do not match an object to within 1 arcsec. For these, fill in what can be filled in from target files or fiberassign files.

    • For every duplicate potential targetid, use only the most recent desitarget version containing that targetid, rather than the default associated with the fiberassign file.

    • Read redshifts from zall-*.fits files to get the *_primary columns, or run the primary generation code on the data (find_primary_spectra()).

  • Future devlopment:

    • Plan for how to support fuji+guadalupe combined analysis. May need to look into cross-schema views, or daughter tables that inherit from both schemas.

    • Anticipating loading afterburners and VACs into the database.

    • Load redshifts from all redrock files in tiles/cumulative, rather than from the ztile-*-cumulative.fits summary file.

class desispec.database.redshift.Exposure(*args: Any, **kwargs: Any)[source]

Representation of the EXPOSURES HDU in the exposures file.

Notes

The column program is filled in via faflavor2program().

class desispec.database.redshift.Fiberassign(*args: Any, **kwargs: Any)[source]

Representation of the FIBERASSIGN table in a fiberassign file.

Notes

  • Targets are assigned to a location. A location happens to correspond to a fiber, but this correspondence could change over time, and therefore should not be assumed to be a rigid 1:1 mapping.

  • PLATE_RA, PLATE_DEC are sometimes missing. These can be copies of TARGET_RA, TARGET_DEC, but in principle they could be different if chromatic offsets in targeting positions were ever implemented.

class desispec.database.redshift.Frame(*args: Any, **kwargs: Any)[source]

Representation of the FRAMES HDU in the exposures file.

Notes

The column frameid is a combination of expid and the camera name:

frameid = 100*expid + cameraid(camera)

where cameraid() is desispec.database.util.cameraid().

class desispec.database.redshift.Photometry(*args: Any, **kwargs: Any)[source]

Contains only photometric quantities associated with a TARGETID.

This table is deliberately designed so that TARGETID can serve as a primary key. Any quantities created or modified by desitarget are defined in the Target class.

However we avoid the use of the term “tractor” for this table, because not every target will have tractor photometry,

Notes

The various LC (light curve) columns, which are vector-valued, are not yet implemented.

class desispec.database.redshift.Potential(*args: Any, **kwargs: Any)[source]

Representation of the POTENTIAL_ASSIGNMENTS table in a fiberassign file.

class desispec.database.redshift.SchemaMixin

Mixin class to allow schema name to be changed at runtime. Also automatically sets the table name.

class desispec.database.redshift.Target(*args: Any, **kwargs: Any)[source]

Representation of the pure-desitarget quantities in the TARGETPHOT table in the targetphot files.

class desispec.database.redshift.Tile(*args: Any, **kwargs: Any)[source]

Representation of the tiles file.

Notes

Most of the data that are currently in the tiles file are derivable from the exposures table with much greater precision:

CREATE VIEW f5.tile AS SELECT tileid,
    -- SURVEY, FAPRGRM, FAFLAVOR?
    COUNT(*) AS nexp, SUM(exptime) AS exptime,
    MIN(tilera) AS tilera, MIN(tiledec) AS tiledec,
    SUM(efftime_etc) AS efftime_etc, SUM(efftime_spec) AS efftime_spec,
    SUM(efftime_gfa) AS efftime_gfa, MIN(goaltime) AS goaltime,
    -- OBSSTATUS?
    SUM(lrg_efftime_dark) AS lrg_efftime_dark,
    SUM(elg_efftime_dark) AS elg_efftime_dark,
    SUM(bgs_efftime_bright) AS bgs_efftime_bright,
    SUM(lya_efftime_dark) AS lya_efftime_dark,
    -- GOALTYPE?
    MIN(mintfrac) AS mintfrac, MAX(night) AS lastnight
FROM f5.exposure GROUP BY tileid;

However because of some unresolved discrepancies, we’ll just load the full tiles file for now.

class desispec.database.redshift.Zpix(*args: Any, **kwargs: Any)[source]

Representation of the ZCATALOG table in zpix files.

class desispec.database.redshift.Ztile(*args: Any, **kwargs: Any)[source]

Representation of the ZCATALOG table in ztile files.

desispec.database.redshift._add_ls_id(data)[source]

Add LS_ID to targetphot data.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

Updated data table.

Return type:

astropy.table.Table

desispec.database.redshift._deduplicate_targetid(data)[source]

Find targetphot rows that are not already loaded into the Photometry table and resolve any duplicate TARGETID.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

An array of rows that are safe to load.

Return type:

numpy.array

desispec.database.redshift._frameid(data)[source]

Update the frameid column.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

Updated data table.

Return type:

astropy.table.Table

desispec.database.redshift._remove_loaded_targetid(data)[source]

Remove rows with TARGETID already loaded into the database.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

An array of rows that are safe to load.

Return type:

numpy.array

desispec.database.redshift._remove_loaded_unique_id(data)[source]

Remove rows with UNIQUE ID already loaded into the database.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

An array of rows that are safe to load.

Return type:

numpy.array

desispec.database.redshift._survey_program(data)[source]

Add SURVEY, PROGRAM, SPGRP columns to zpix and ztile tables.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

Updated data table.

Return type:

astropy.table.Table

Raises:

KeyError – If a necessary header could not be found.

desispec.database.redshift._target_unique_id(data)[source]

Add composite ID column for later conversion.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

Updated data table.

Return type:

astropy.table.Table

desispec.database.redshift._tileid(data)[source]

Update the tileid column. Also check for the presence of PLATE_RA, PLATE_DEC.

Parameters:

data (astropy.table.Table) – The initial data read from the file.

Returns:

Updated data table.

Return type:

astropy.table.Table

desispec.database.redshift.get_options(*args)[source]

Parse command-line options.

Parameters:

args (iterable) – If arguments are passed, use them instead of sys.argv.

Returns:

The parsed options.

Return type:

argparse.Namespace

desispec.database.redshift.load_file(filepaths, tcls, hdu=1, preload=None, expand=None, insert=None, convert=None, index=None, rowfilter=None, q3c=None, chunksize=50000, maxrows=0)[source]

Load data file into the database, assuming that column names map to database column names with no surprises.

Parameters:
  • filepaths (str or list) – Full path to the data file or set of data files.

  • tcls (sqlalchemy.ext.declarative.api.DeclarativeMeta) – The table to load, represented by its class.

  • hdu (int or str, optional) – Read a data table from this HDU (default 1).

  • preload (callable, optional) – A function that takes a Table as an argument. Use this for more complicated manipulation of the data before loading, for example a function that depends on multiple columns. The return value should be the updated Table.

  • expand (dict, optional) – If set, map FITS column names to one or more alternative column names.

  • insert (dict, optional) – If set, insert one or more columns, before an existing column. The existing column will be copied into the new column(s).

  • convert (dict, optional) – If set, convert the data for a named (database) column using the supplied function.

  • index (str, optional) – If set, add a column that just counts the number of rows.

  • rowfilter (callable, optional) – If set, apply this filter to the rows to be loaded. The function should return bool, with True meaning a good row.

  • q3c (str, optional) – If set, create q3c index on the table, using the RA column named q3c.

  • chunksize (int, optional) – If set, load database chunksize rows at a time (default 50000).

  • maxrows (int, optional) – If set, stop loading after maxrows are loaded. Alteratively, set maxrows to zero (0) to load all rows.

Returns:

The grand total of rows loaded.

Return type:

int

desispec.database.redshift.load_redrock(datapath=None, hdu='REDSHIFTS', q3c=False)[source]

Load redrock files into the zcat table.

This function is deprecated since there should now be a single redshift catalog file.

Parameters:
  • datapath (str) – Full path to the directory containing redrock files.

  • hdu (int or str, optional) – Read a data table from this HDU (default ‘REDSHIFTS’).

  • q3c (bool, optional) – If set, create q3c index on the table.

desispec.database.redshift.main()[source]

Entry point for command-line script.

Returns:

An integer suitable for passing to sys.exit().

Return type:

int

desispec.database.redshift.q3c_index(table, ra='ra')[source]

Create a q3c index on a table.

Parameters:
  • table (str) – Name of the table to index.

  • ra (str, optional) – If the RA, Dec columns are called something besides “ra” and “dec”, set its name. For example, ra='target_ra'.

desispec.database.redshift.setup_db(options=None, **kwargs)[source]

Initialize the database connection.

Parameters:
  • options (argpare.Namespace) – Parsed command-line options.

  • kwargs (keywords) – If present, use these instead of options. This is more user-friendly than setting up a Namespace object in, e.g. a Jupyter Notebook.

Returns:

True if the configured database is a PostgreSQL database.

Return type:

bool

desispec.database.redshift.update_truth(filepath, hdu=2, chunksize=50000, skip=('SLOPES', 'EMLINES'))[source]

Add data from columns in other HDUs of the Truth table.

Parameters:
  • filepath (str) – Full path to the data file.

  • hdu (int or str, optional) – Read a data table from this HDU (default 2).

  • chunksize (int, optional) – If set, update database chunksize rows at a time (default 50000).

  • skip (tuple(), optional) – Do not load columns with these names (default, ('SLOPES', 'EMLINES'))

desispec.database.util

Classes and functions for use by all database code.

desispec.database.util.cameraid(camera)[source]

Converts camera (e.g. ‘b0’) to an integer in a simple but ultimately arbitrary way.

Parameters:

camera (str) – Camera name.

Returns:

An arbitrary integer, though in the range [0, 29].

Return type:

int

desispec.database.util.convert_dateobs(timestamp, tzinfo=None)[source]

Convert a string timestamp into a datetime.datetime object.

Parameters:
  • timestamp (str) – Timestamp in string format.

  • tzinfo (datetime.tzinfo, optional) – If set, add time zone to the timestamp.

Returns:

The converted timestamp.

Return type:

datetime.datetime

desispec.database.util.decode_surveyid(surveyid)[source]

Converts surveyid to its corresponding name.

Parameters:

surveyid (int) – Survey number

Returns:

The name of the corresponding survey.

Return type:

str

desispec.database.util.decode_targetphotid(targetphotid)[source]

Convert id into its components.

Parameters:

targetphotid (int) – The 128-bit id.

Returns:

A tuple of targetid, tileid and survey.

Return type:

tuple

desispec.database.util.fiberassignid(targetid, tileid, location)[source]

Convert inputs into an arbitrary large integer.

Parameters:
  • targetid (int) – Standard TARGETID.

  • tileid (int) – Standard TILEID.

  • location (int) – Location on the tile.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

int

desispec.database.util.frameid(expid, camera)[source]

Converts the pair expid, camera into an arbitrary integer suitable for use as a primary key.

desispec.database.util.parse_pgpass(hostname='nerscdb03.nersc.gov', username='desidev_admin')[source]

Read a ~/.pgpass file.

Parameters:
  • hostname (str, optional) – Database hostname.

  • username (str, optional) – Database username.

Returns:

A string suitable for creating a SQLAlchemy database engine, or None if no matching data was found.

Return type:

str

desispec.database.util.programid(program)[source]

Converts program (e.g. ‘bright’) to an integer in a simple but ultimately arbitrary way.

Parameters:

program (str) – Program name.

Returns:

An arbitrary, small integer.

Return type:

int

desispec.database.util.spgrpid(spgrp)[source]

Converts spgrp (e.g. ‘cumulative’) to an integer in a simple but ultimately arbitrary way.

Parameters:

spgrp (str) – SPGRP name.

Returns:

An arbitrary, small integer.

Return type:

int

desispec.database.util.surveyid(survey)[source]

Converts survey (e.g. ‘main’) to an integer in a simple but ultimately arbitrary way.

Parameters:

survey (str) – Survey name.

Returns:

An arbitrary, small integer.

Return type:

int

desispec.database.util.targetphotid(targetid, tileid, survey)[source]

Convert inputs into an arbitrary large integer.

Parameters:
  • targetid (int) – Standard TARGETID.

  • tileid (int) – Standard TILEID.

  • survey (str) –

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

int

desispec.database.util.zpixid(targetid, survey, program)[source]

Convert inputs into an arbitrary large integer.

Parameters:
  • targetid (int) – Standard TARGETID.

  • survey (str) – Survey name.

  • program (str) – Program name.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

int

desispec.database.util.ztileid(targetid, spgrp, spgrpval, tileid)[source]

Convert inputs into an arbitrary large integer.

Parameters:
  • targetid (int) – Standard TARGETID.

  • spgrp (str) – Tile grouping.

  • spgrpval (str) – Id with in spgrp.

  • tileid (int) – Standard TILEID.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

int

desispec.desi_bias_dark_1d_model

Usage:

python3 desi_bias_dark_1d_model.py

desispec.desi_create_bias_dark

Usage Manual

  • Pipeline for gnerating new bias+dark files

  • Input: night array to use

Example

python3 desi_create_bias_dark.py -n 20200729 20200730 -r 0060633 0060634 0060635 0060636 0060637 0060638 0060639 0060640 0060641 0060642

desispec.desi_dashboard

class desispec.desi_dashboard.DESI_DASHBOARD[source]

Code to generate the statistic of desi_pipe production status Usage: python3 desi_dashboard.py –prod realtime10 –prod_dir /global/cscratch1/sd/zhangkai/desi/ –output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_dashboard/ –output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_dashboard/

desispec.desi_proc_time_distribution

class desispec.desi_proc_time_distribution.DESI_PROC_TIME_DISTRIBUTION[source]

Code to generate a webpage for monitoring of desi_dailyproc production status

Normal Mode:

desi_proc_time_distribution -n all --n_night 3  --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_proc_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_proc_dashboard/
python3 desi_proc_time_distribution.py -n all --n_night 3  --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_proc_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_proc_dashboard/

Cron job script:

*/30 * * * * /global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/bin/python3 /global/project/projectdirs/desi/users/zhangkai/desi/code/desispec/py/desispec/workflow/proc_dashboard_funcs.py -n all --n_night 30 --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_proc_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_proc_dashboard/ >/global/project/projectdirs/desi/users/zhangkai/desi_proc_dashboard.log 2>/global/project/projectdirs/desi/users/zhangkai/desi_proc_dashboard.err &
_add_html_table(table, night)[source]

Add a collapsible and extendable table to the html file for one specific night

Parameters:
  • table – the table generated by ‘calculate_one_night’

  • night – like 20200131

Returns:

The string to be added to the html file

_add_js_script1()[source]

Return the javascript script to be added to the html file

_add_js_script2(n_modal)[source]

Return the another javascript script to be added to the html file

_init_parser(parser)[source]

Initialize the parser to read input

_initialize_page()[source]

Initialize the html file for showing the statistics, giving all the headers and CSS setups.

calculate_one_night(night)[source]

For a given night, return the file counts and other other information for each exposure taken on that night

Parameters:

night

Returns:

A dictionary containing the statistics with expid as key name:

FLAVOR: FLAVOR of this exposure
OBSTYPE: OBSTYPE of this exposure
EXPTIME: Exposure time
SPECTROGRAPHS: a list of spectrographs used
n_spectrographs: number of spectrographs
n_psf: number of PSF files
n_ff:  number of fiberflat files
n_frame: number of frame files
n_sframe: number of sframe files
n_cframe: number of cframe files
n_sky: number of sky files

check_running()[source]

Check if the desi_dailyproc process is running

find_newexp(night, fileglob, known_exposures)[source]

Check the path given for new exposures

what_night_is_it()[source]

Return the current night

desispec.efftime

Effective exposure time formulae.

desispec.efftime.compute_efftime(table, kterm=0.114, ebv_r_coeff=2.165, fiber_diameter_arcsec=1.52)[source]

Computes the effective exposure times using transparency and fiber acceptance from the GFAs offline analysis, and the sky magnitudes from the spectroscopy. Uses the formulae described in https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed for the dark, bright and backup programs.

Parameters:

table – astropy.table.Table with columns - EXPTIME exposure time in seconds - SKY_MAG_R_SPEC AB magnitude per arcsec2 of sky, in decam-r filter - EBV E(B-V) from SFD map - TRANSPARENCY_GFA transparency (number between 0 and 1, ~1 for photometric nights irrespectively of AIRMASS) - AIRMASS (airmass >=1) - FIBERFAC_GFA PMGSTARS forced photometry amount of light in 1.52 asec diameter aperture normalized to nominal, assuming a point source profile - FIBERFAC_ELG_GFA PMGSTARS forced photometry amount of light in 1.52 asec diameter aperture normalized to nominal, assuming an ELG-like profile (r_half = 0.45 asec exponential) - FIBERFAC_BGS_GFA PMGSTARS forced photometry amount of light in 1.52 asec diameter aperture normalized to nominal, assuming a BGS-like profile (r_half = 1.5 asec de Vaucouleurs) - FIBER_FRACFLUX_GFA fraction of light in fiber for point source - FIBER_FRACFLUX_ELG_GFA fraction of light in fiber for typical ELG source - FIBER_FRACFLUX_BGS_GFA fraction of light in fiber for typical BGS source

desispec.emlinefit

Routines for desi_emlinefit_afterburner.

desispec.emlinefit.emlines_gaussfit(emname, zspec, waves, fluxes, ivars, rf_fit_hw=40, min_rf_fit_hw=20, rf_cont_w=200, p0_sigma=2.5, p0_flux=10, p0_share=0.58, min_sigma=1e-05, max_sigma=10.0, min_flux=-1000000000.0, max_flux=1000000000.0, min_share=0.1, max_share=1, log=None)[source]

Fits the [OII] doublet line profile with 2 related Gaussians.

Parameters:
  • emname – “OII” or “OIII” or “HALPHA”, “HBETA”, “HGAMMA”, “HDELTA” (string)

  • zspec – redshift (float)

  • waves – wavelength in Angstroms (numpy array)

  • fluxes – flux observed in the broad band (in erg/s/cm2/A) (numpy array)

  • ivars – inverse variance on the flux (numpy array)

  • rf_fit_hw (optional, defaults to 40) – rest-frame wavelength width (in A) used for fitting on each side of the line (float)

  • min_rf_fit_hw (optional, defaults to 20) – minimum requested rest-frame width (in A) on each side of the line to consider the fitting (float)

  • rf_cont_w (optional, defaults to 200) – rest-frame wavelength extent (in A) to fit the continuum (float)

  • p0_sigma (optional, defaults to 2.5) – initial guess on the line width in A (float)

  • p0_flux (optional, defaults to 0.1) – initial guess on the line flux in 1e-17 * erg/cm2/s (float)

  • p0_share (optional, defaults to 0.58) – initial guess on the share between the two [OII] lines (float)

  • min_sigma (optional, defaults to 1e-5) – minimum allowed value for the line width in A (float)

  • max_sigma (optional, defaults to 10.) – maximum allowed value for the line width in A (float)

  • min_flux (optional, defaults to 1e-5) – minimum allowed value for the flux in 1e-17 * erg/cm2/s (float)

  • max_flux (optional, defaults to 1e9) – maximum allowed value for the flux in 1e-17 * erg/cm2/s (float)

  • min_share (optional, defaults to 1e-1) – minimum allowed value for the share (float)

  • max_share (optional, defaults to 1) – maximum allowed value for the share (float)

  • log (optional, defaults to get_logger()) – Logger object

Returns:

a dictionary with various quantities, noticely “FLUX” and “FLUX_IVAR” (dictionary of floats) list of all keys:

CHI2, NDOF: *reduced* chi2 and nb of degrees of freedom
CONT, CONT_IVAR: continuum in 1e-17 * erg/cm2/s/A
FLUX, FLUX_IVAR: flux in 1e-17 * erg/cm2/s
SIGMA, SIGMA_IVAR: line width in A (observed frame)
SHARE, SHARE_IVAR: f1/(f0+f1) for OII and OIII doublets
EW, EW_IVAR: rest-frame equivalent width (in A)
waves: wavelength values (in A) used for the fitting (numpy array of floats)
fluxes: flux values (in 1e-17 * erg/cm2/s/A) used for the fitting (numpy array of floats)
ivars: ivar values used for the fitting (numpy array of floats)
models: model flux values (in 1e-17 * erg/cm2/s/A) from the fit (numpy array of floats)

succeed: did the fit succeed? (boolean)

Return type:

emdict

Note

  • Adapted/simplified from elgredshiftflag from J. Comparat (used for eBOSS/ELG): https://svn.sdss.org/repo/eboss/elgredshiftflag/

  • Returns np.nan in emdict (and NDOF=-99) if not enough pixels to fit or if fit fails.

  • Default settings designed for ELGs (e.g. max_sigma); need to re-assess if run on other targets.

  • For “OII”, let the doublet line ratio free during the fit.

  • For “OIII”, fits the 4960 and 5007 lines with a fixed line ratio.

  • For the Balmer lines, SHARE is not fitted and set to np.nan.

desispec.emlinefit.get_emlines(zspecs, waves, fluxes, ivars, emnames=['OII', 'HDELTA', 'HGAMMA', 'HBETA', 'OIII', 'HALPHA'], rf_fit_hw=40, min_rf_fit_hw=20, rf_cont_w=200, log=None)[source]

Get Gaussian-fitted emission line fluxes.

Parameters:
  • zspecs – redshifts (numpy array of shape (Nspec))

  • waves – wavelengths (numpy array of shape (Nwave))

  • fluxes – fluxes (numpy array of shape (Nspec, Nwave))

  • ivars – inverse variances (numpy array of shape (Nspec, Nwave))

  • emnames (optional, defaults to ["OII", "HDELTA", "HGAMMA", "HBETA", "OIII", "HALPHA"]) – list of lines to fit (list of string)

  • rf_fit_hw (optional, defaults to 40) – rest-frame wavelength width (in A) used for fitting on each side of the line (float)

  • min_rf_fit_hw (optional, defaults to 20) – minimum requested rest-frame width (in A) on each side of the line to consider the fitting (float)

  • rf_cont_w (optional, defaults to 200) – rest-frame wavelength extent (in A) to fit the continuum (float)

  • log (optional, defaults to get_logger()) – Logger object

Returns:

a dictionary with a subdictionary for each emname in emnames, with:,

CHI2, NDOF: reduced chi2 and nb of degrees of freedom CONT, CONT_IVAR: continuum in 1e-17 * erg/cm2/s/A FLUX, FLUX_IVAR: flux in 1e-17 * erg/cm2/s SIGMA, SIGMA_IVAR: line width in A (observed frame) SHARE, SHARE_IVAR: f1/(f0+f1) for OII and OIII doublets EW, EW_IVAR: rest-frame equivalent width in A waves, fluxes, ivars, models: data used for fitting + fitted model

Return type:

emdict

desispec.emlinefit.get_rf_em_waves(emname)[source]

Returns the rest-frame, vacuum, wavelengths.

Parameters:
  • emname – “OII”, “HDELTA”, “HGAMMA”, “HBETA”, “OIII”, “HALPHA” (string)

  • from – “OII”, “HDELTA”, “HGAMMA”, “HBETA”, “OIII”, “HALPHA” (string)

Returns:

rest-frame wavelength(s) (np array)

Return type:

rf_em_waves

Notes

For OII and OIII returns a two-elements array; one-element array otherwise. https://github.com/desihub/fastspecfit/blob/60393296e0cc466858f70a5d021d02693eff9375/py/fastspecfit/data/emlines.ecsv

desispec.exposure_qa

Utility functions to compute an exposure QA scores.

desispec.exposure_qa.compute_exposure_qa(night, expid, specprod_dir=None)[source]

Computes the exposure_qa

Parameters:
  • night – int, YYYYMMDD

  • expid – int, exposure id

  • specprod_dir – str, optional, specify the production directory. default is $DESI_SPECTRO_REDUX/$SPECPROD

Returns:

two tables (astropy.table.Table), fiberqa (with one row per target and at least a TARGETID column) and petalqa (with one row per petal and at least a PETAL_LOC column)

desispec.exposure_qa.get_qa_params()[source]

Returns a dictionnary with the content of data/qa/qa-params.yaml

desispec.fiberbitmasking

Functions to properly take FIBERSTATUS into account in the variances for data reduction

desispec.fiberbitmasking.get_all_nonamp_fiberbitmask_val()[source]

Return a mask for all fatally bad FIBERSTATUS bits except BADAMPB/R/Z

Note

Does not include STUCKPOSITIONER or RESTRICTED, which could still be on a valid sky location, or even a target for RESTRICTED. Also does not include POORPOSITION which is bad for stdstars but not necessarily fatal for otherwise processing a normal fiber.

desispec.fiberbitmasking.get_fiberbitmask_comparison_value(kind, band)[source]

Takes a string argument and returns a 32-bit integer representing the logical OR of all relevant fibermask bits for that given reduction step

Parameters:
  • kind – str : string designating which combination of bits to use based on the operation. Possible values are “all”, “sky” (or “skysub”), “flat”, “flux” (or “fluxcalib”), “star” (or “stdstars”)

  • band – str : BADAMP band bits to set. Values include ‘b’, ‘r’, ‘z’, or combinations thereof such as ‘brz’

Returns:

32 bit bitmask corresponding to the fiberbitmask of the desired kind

in the desired cameras (bands).

Return type:

bitmask

desispec.fiberbitmasking.get_fiberbitmasked_frame(frame, bitmask=None, ivar_framemask=True)[source]

Wrapper script of get_fiberbitmasked_frame_arrays that will return a modified version of the frame instead of just the flux and ivar

Note

The input “frame” variable itself is modified and returned, not a copy.

desispec.fiberbitmasking.get_fiberbitmasked_frame_arrays(frame, bitmask=None, ivar_framemask=True, return_mask=False)[source]

Function that takes a frame object and a bitmask and returns ivar (and optionally mask) array(s) that have fibers with offending bits in fibermap[‘FIBERSTATUS’] set to 0 in ivar and optionally flips a bit in mask.

Parameters:
  • frame – frame object

  • bitmask – int32 or list/array of int32’s derived from desispec.maskbits.fibermask OR string indicating a keyword for get_fiberbitmask_comparison_value()

  • ivar_framemask – bool (default=True), tells code whether to multiply the output variance by (frame.mask==0)

  • return_mask – bool, (default=False). Returns the frame.mask with the logic of FIBERSTATUS applied.

Returns:

frame.ivar where the fibers with FIBERSTATUS & bitmask > 0

set to zero ivar

mask: (optional) frame.mask logically OR’ed with BADFIBER bit in cases with

a bad FIBERSTATUS

Return type:

ivar

Examples

Example bitmask list:

bitmask = [fmsk.BROKENFIBER,fmsk.UNASSIGNED,fmsk.BADFIBER,                        fmsk.BADTRACE,fmsk.MANYBADCOL, fmsk.MANYREJECTED]
bitmask = get_fiberbitmask_comparison_value(kind='fluxcalib', band='brz')
bitmask = 'fluxcalib'
bitmask = 4128780

desispec.fibercrosstalk

Utility functions to correct for the fibercrosstalk

desispec.fibercrosstalk.compute_contamination(frame, dfiber, kernel, params, xyset, fiberflat=None, fractional_error=0.1)[source]

Computes the contamination of a frame from a given fiber offset

Parameters:
  • frame – a desispec.frame.Frame object

  • dfiber – int, fiber offset (-2,-1,1,2)

  • kernel – 1D numpy array, convolution kernel

  • params – nested dictionnary, parameters of the crosstalk model

  • xyset

    desispec.xytraceset.XYTraceSet object with trace coordinates to shift the spectra

    fiberflat : desispec.fiberflat.FiberFlat object, optional if the frame has already been fiber flatfielded fractionnal_error : float, optional consider this systematic relative error on the correction

Returns:

the contamination of the frame, 2D numpy array of same shape as frame.flux, and its variance

Return type:

contamination , contamination_var

desispec.fibercrosstalk.compute_crosstalk_kernels(max_fiber_offset=2, fiber_separation_in_pixels=7.3, asymptotic_power_law_index=2.5)[source]

Computes the fiber crosstalk convolution kernels assuming a power law PSF tail

Parameters:
  • max_fiber_offset – positive int, maximum fiber offset, 2 by default

  • optional – positive int, maximum fiber offset, 2 by default

  • fiber_separation_in_pixels – float, distance between neighboring fiber traces in the CCD in pixels, default=7.3

  • optional – float, distance between neighboring fiber traces in the CCD in pixels, default=7.3

  • asymptotic_power_law_index – float, power law index of PSF tail

  • optional – float, power law index of PSF tail

Returns:

A dictionnary of kernels, with key the positive fiber offset 1,2,…. Each entry is an 1D array.

desispec.fibercrosstalk.correct_fiber_crosstalk(frame, fiberflat=None, xyset=None)[source]

Apply a fiber cross talk correction. Modifies frame.flux and frame.ivar.

Parameters:
  • frame – desispec.frame.Frame object

  • fiberflat – desispec.fiberflat.FiberFlat object

  • optional – desispec.fiberflat.FiberFlat object

  • xyset – desispec.xytraceset.XYTraceSet object with trace coordinates to shift the spectra (automatically found with calibration finder otherwise)

  • optional – desispec.xytraceset.XYTraceSet object with trace coordinates to shift the spectra (automatically found with calibration finder otherwise)

desispec.fibercrosstalk.eval_crosstalk(camera, wave, fibers, dfiber, params, apply_scale=True, nfiber_per_bundle=25)[source]

Computes the crosstalk as a function of wavelength from a fiber offset dfiber (positive and negative) for an input set of fibers

Parameters:
  • camera – str, camera identifier (b8,r7,z3, …)

  • wave – 1D array, wavelength

  • fibers – list or 1D array of int, list of contaminated fibers

  • dfiber – int, positive or negative fiber offset, contaminating fibers = contaminated fibers + dfiber

  • params – nested dictionnary, parameters of the crosstalk model

  • apply_scale – boolean, optional apply or not the scale factor if found in the list of parameters

  • nfiber_per_bundle – int, optional number of fibers per bundle, only the fibers in the same bundle are considered

Returns:

2D array of crosstalk fraction (between 0 and 1) of shape ( len(fibers),len(wave) )

desispec.fibercrosstalk.read_crosstalk_parameters()[source]

Reads the crosstalk parameters in desispec/data/fiber-crosstalk.yaml

Returns:

nested dictionary with parameters per camera

desispec.fiberflat

Utility functions to compute a fiber flat correction and apply it We try to keep all the (fits) io separated.

desispec.fiberflat.apply_fiberflat(frame, fiberflat)[source]
Apply fiberflat to frame. Modifies frame.flux and frame.ivar.

Checks whether an heliocentric correction has been applied to the frame wavelength in which case also apply it to the flat field array.

Parameters:
  • framedesispec.Frame object

  • fiberflatdesispec.FiberFlat object

The frame is divided by the fiberflat, except where the fiberflat=0.

frame.mask gets bit specmask.BADFIBERFLAT set where
  • fiberflat.fiberflat == 0

  • fiberflat.ivar == 0

  • fiberflat.mask != 0

desispec.fiberflat.autocalib_fiberflat(fiberflats)[source]

Combine fiberflats of all spectrographs from different lamps to maximize uniformity :param fiberflats: list of desispec.FiberFlat object

returns a dictionary of desispec.FiberFlat objects , one per spectrograph

desispec.fiberflat.average_fiberflat(fiberflats)[source]

Average several fiberflats :param fiberflats: list of desispec.FiberFlat object

returns a desispec.FiberFlat object

desispec.fiberflat.compute_fiberflat(frame, nsig_clipping=10.0, accuracy=0.0005, minval=0.1, maxval=10.0, max_iterations=15, smoothing_res=5.0, max_bad=100, max_rej_it=5, min_sn=0, diag_epsilon=0.001)[source]

Compute fiber flat by deriving an average spectrum and dividing all fiber data by this average. Input data are expected to be on the same wavelength grid, with uncorrelated noise. They however do not have exactly the same resolution.

Parameters:
  • frame (desispec.Frame) – input Frame object with attributes wave, flux, ivar, resolution_data

  • nsig_clipping – [optional] sigma clipping value for outlier rejection

  • accuracy – [optional] accuracy of fiberflat (end test for the iterative loop)

  • minval – [optional] mask pixels with flux < minval * median fiberflat.

  • maxval – [optional] mask pixels with flux > maxval * median fiberflat.

  • max_iterations – [optional] maximum number of iterations

  • smoothing_res – [optional] spacing between spline fit nodes for smoothing the fiberflat

  • max_bad – [optional] mask entire fiber if more than max_bad-1 initially unmasked pixels are masked during the iterations

  • max_rej_it – [optional] reject at most the max_rej_it worst pixels in each iteration

  • min_sn – [optional] mask portions with signal to noise less than min_sn

  • diag_epsilon – [optional] size of the regularization term in the deconvolution

Returns:

desispec.FiberFlat object with attributes

wave, fiberflat, ivar, mask, meanspec

Notes: - we first iteratively :

  • compute a deconvolved mean spectrum

  • compute a fiber flat using the resolution convolved mean spectrum for each fiber

  • smooth the fiber flat along wavelength

  • clip outliers

  • then we compute a fiberflat at the native fiber resolution (not smoothed)

  • the routine returns the fiberflat, its inverse variance , mask, and the deconvolved mean spectrum

  • the fiberflat is the ratio data/mean , so this flat should be divided to the data

NOTE THAT THIS CODE HAS NOT BEEN TESTED WITH ACTUAL FIBER TRANSMISSION VARIATIONS, OUTLIER PIXELS, DEAD COLUMNS …

desispec.fiberflat.qa_fiberflat(param, frame, fiberflat)[source]

Calculate QA on FiberFlat object

Parameters:
  • param – dict of QA parameters

  • frame – Frame

  • fiberflat – FiberFlat

Returns:

dict of QA outputs

Need to record simple Python objects for yaml (str, float, int)

Return type:

qadict

desispec.fiberflat_vs_humidity

Utility functions to compute a fiber flat corrected for variations with humidity in the shack

desispec.fiberflat_vs_humidity._fit_flat(wavelength, flux, ivar, fibers, mean_fiberflat_vs_humidity, humidity_array)[source]

Finds best fit interpolation of fiberflat templates that matches an input flux frame Works only if wavelength array intersects the range [4000,4600]A, i.e. the blue cameras

Parameters:
  • wavelength – 1D numpy array (n_wavelength) in Angstrom

  • flux – 2D numpy array (n_fibers,n_wavelength) unit does not matter

  • ivar – 2D numpy array (n_fibers,n_wavelength) inverse variance of flux

  • fibers – list or 1D number arrays of fibers to use among range(n_fibers)

  • mean_fiberflat_vs_humidity – 3D numpy array (n_humidity,n_fibers,n_wavelength)

  • humidity_array – 1D numpy array (n_humidity)

Returns best_fit_flat best_fit_humidity (2D numpy array (n_fibers,n_wavelength) and float)

desispec.fiberflat_vs_humidity._interpolated_fiberflat_vs_humidity(fiberflat_vs_humidity, humidity_indices, humidity_index)[source]

Interpolates between fiberflat templates indexed by humidity.

Parameters:
  • fiberflat_vs_humidity – 3D numpy array (n_humidity,n_fibers,n_wavelength)

  • humidity_indices – 1D numpy array (n_humidity)

  • humidity_index – float, humidity index value (same unit as humidity_indices)

Returns 2D numpy array (n_fibers,n_wavelength)

desispec.fiberflat_vs_humidity.compute_humidity_corrected_fiberflat(calib_fiberflat, mean_fiberflat_vs_humidity, humidity_array, current_humidity, frame)[source]

Apply a humidity-dependent correction to an input fiber flat Returns frame_fiberflat = calib_fiberflat / flat_vs_humidity_model(calib) * flat_vs_humidity_model(frame)

Parameters:
  • calib_fiberflat – desispec.FiberFlat object

  • mean_fiberflat_vs_humidity – 3D numpy array (n_humidity,n_fibers,n_wavelength)

  • humidity_array – 1D numpy array (n_humidity)

  • current_humidity – float (same unit as humidity_array)

  • frame – desispec.Frame object

Returns modified desispec.FiberFlat object

desispec.fiberfluxcorr

Routines to compute fiber flux corrections based on the fiber location, the exposure seeing, and the target morphology.

desispec.fiberfluxcorr.flat_to_psf_flux_correction(fibermap, exposure_seeing_fwhm=1.1)[source]

Multiplicative factor to apply to the flat-fielded spectroscopic flux of a fiber to calibrate the spectrum of a point source, given the current exposure seeing

Parameters:
  • fibermap – fibermap of frame, astropy.table.Table

  • exposure_seeing_fwhm – seeing FWHM in arcsec

Returns: 1D numpy array with correction factor to apply to fiber fielded fluxes, valid for point sources.

desispec.fiberfluxcorr.psf_to_fiber_flux_correction(fibermap, exposure_seeing_fwhm=1.1)[source]

Multiplicative factor to apply to the psf flux of a fiber to obtain the fiber flux, given the current exposure seeing. The fiber flux is the flux one would collect for this object in a fiber of 1.5 arcsec diameter, for a 1 arcsec seeing, FWHM (same definition as for the Legacy Surveys).

Parameters:
  • fibermap – fibermap of frame, astropy.table.Table

  • exposure_seeing_fwhm – seeing FWHM in arcsec

Returns: 1D numpy array with correction factor to apply to fiber fielded fluxes, valid for any sources.

desispec.fluxcalibration

Flux calibration routines.

desispec.fluxcalibration.ZP_from_calib(exptime, wave, calib)[source]

Calculate the ZP in AB magnitudes given the calibration and the wavelength arrays :param exptime: float; exposure time in seconds :param wave: 1D array (A) :param calib: 1D array (converts erg/s/A to photons/s/A)

Returns:

1D array of ZP values in AB magnitudes

Return type:

ZP_AB

desispec.fluxcalibration._compute_coef(coord, node_coords)[source]

Function used by interpolate_on_parameter_grid2

Parameters:
  • coord – 1D array of coordinates of size n_axis

  • node_coords – 2D array of coordinates of nodes, shape = (n_nodes,n_axis)

Returns:

1D array of linear coefficients for each node, size = n_nodes

Return type:

coef

desispec.fluxcalibration._func(arg)[source]

Used for multiprocessing.Pool

desispec.fluxcalibration._func2(arg)[source]

Used for multiprocessing.Pool

desispec.fluxcalibration._smooth_template(template_id, camera_index, template_flux)[source]

Used for multiprocessing.Pool

desispec.fluxcalibration.applySmoothingFilter(flux, width=200)[source]

Return a smoothed version of the input flux array using a median filter

Parameters:
  • flux – 1D array of flux

  • width – size of the median filter box

Returns:

median filtered flux of same size as input

Return type:

smooth_flux

desispec.fluxcalibration.apply_flux_calibration(frame, fluxcalib)[source]

Applies flux calibration to input flux and ivar

Parameters:
  • frame – Spectra object with attributes wave, flux, ivar, resolution_data

  • fluxcalib – FluxCalib object with wave, calib, …

Modifies frame.flux and frame.ivar

desispec.fluxcalibration.compute_flux_calibration(frame, input_model_wave, input_model_flux, input_model_fibers, nsig_clipping=10.0, deg=2, debug=False, highest_throughput_nstars=0, exposure_seeing_fwhm=1.1, stdcheck=True, nsig_flux_scale=3)[source]

Compute average frame throughput based on data frame.(wave,flux,ivar,resolution_data) and spectro-photometrically calibrated stellar models (model_wave,model_flux). Wave and model_wave are not necessarily on the same grid

Parameters:
  • frame – Frame object with attributes wave, flux, ivar, resolution_data

  • input_model_wave – 1D[nwave] array of model wavelengths

  • input_model_flux – 2D[nstd, nwave] array of model fluxes

  • input_model_fibers – 1D[nstd] array of model fibers

  • nsig_clipping – (optional) sigma clipping level

  • exposure_seeing_fwhm – (optional) seeing FWHM in arcsec of the exposure

  • stdcheck – check if the model stars are actually standards according to the fibermap and only rely on those

  • nsig_flux_scale – n sigma cutoff on the flux scale among standard stars

Returns:

desispec.FluxCalib object calibration: mean calibration (without resolution)

Notes

  • we first resample the model on the input flux wave grid

  • then convolve it to the data resolution (the input wave grid is supposed finer than the spectral resolution)

  • then iteratively - fit the mean throughput (deconvolved, this is needed because of sharp atmospheric absorption lines) - compute a scale factor to fibers (to correct for small mis-alignement for instance) - perform outlier rejection

There is one subtelty with the relation between calibration and resolution.
  • The input frame flux is on average flux^frame_fiber = R_fiber*C*flux^true where C is the true calibration (or throughput) which is a function of wavelength. This is the system we solve.

  • But we want to return a calibration vector per fiber C_fiber defined by flux^cframe_fiber = flux^frame_fiber/C_fiber, such that flux^cframe can be compared with a convolved model of the truth, flux^cframe_fiber = R_fiber*flux^true, i.e. (R_fiber*C*flux^true)/C_fiber = R_fiber*true_flux, giving C_fiber = (R_fiber*C*flux^true)/(R_fiber*flux^true)

  • There is no solution for this for all possible input specta. The solution for a flat spectrum is returned, which is very close to C_fiber = R_fiber*C (but not exactly).

desispec.fluxcalibration.interpolate_on_parameter_grid(data_wave, data_flux, data_ivar, template_flux, teff, logg, feh, template_chi2)[source]
3D Interpolation routine among templates based on a grid of parameters teff, logg, feh.

The tricky part is to define a cube on the parameter grid populated with templates, and it is not always possible. The routine never extrapolates, so that we stay in the range of input parameters.

Parameters:
  • data_wave – 1D[nwave] array of wavelength (concatenated list of input wavelength of different cameras and exposures)

  • data_flux – 1D[nwave] array of normalized flux = (input flux)/median_filter(input flux) (concatenated list)

  • data_ivar – 1D[nwave] array of inverse variance of normalized flux

  • template_flux – 2D[ntemplates,nwave] array of normalized flux of templates (after resample, convolution and division by median_filter)

  • teff – 1D[ntemplates]

  • logg – 1D[ntemplates]

  • feh – 1D[ntemplates]

  • template_chi2 – 1D[ntemplatess] array of precomputed chi2 = sum(data_ivar*(data_flux-template_flux)**2)

Returns:

best fit coefficient of linear combination of templates chi2 : chi2 of the linear combination

Return type:

coefficients

desispec.fluxcalibration.isStdStar(fibermap, bright=None)[source]

Determines if target(s) are standard stars

Parameters:

fibermap – table including DESI_TARGET or SV1_DESI_TARGET bit mask(s)

Optional:

bright: if True, only bright time standards; if False, only darktime, otherwise both

Returns bool or array of bool

desispec.fluxcalibration.match_templates(wave, flux, ivar, resolution_data, stdwave, stdflux, teff, logg, feh, ncpu=1, z_max=0.005, z_res=2e-05, template_error=0, comm=None)[source]

For each input spectrum, identify which standard star template is the closest match, factoring out broadband throughput/calibration differences.

Parameters:
  • wave – A dictionary of 1D array of vacuum wavelengths [Angstroms]. Example below.

  • flux – A dictionary of 1D observed flux for the star

  • ivar – A dictionary 1D inverse variance of flux

  • resolution_data – resolution corresponding to the star’s fiber

  • stdwave – 1D standard star template wavelengths [Angstroms]

  • stdflux – 2D[nstd, nwave] template flux

  • teff – 1D[nstd] effective model temperature

  • logg – 1D[nstd] model surface gravity

  • feh – 1D[nstd] model metallicity

  • ncpu – number of cpu for multiprocessing

  • comm – MPI communicator; if given, ncpu will be ignored and only rank 0 will return results that are not None

Returns:

numpy.array of linear coefficient of standard stars redshift : redshift of standard star chipdf : reduced chi2

Return type:

coef

Notes

  • wave and stdwave can be on different grids that don’t necessarily overlap

  • wave does not have to be uniform or monotonic. Multiple cameras can be supported by concatenating their wave and flux arrays

desispec.fluxcalibration.normalize_templates(stdwave, stdflux, mag, band, photsys)[source]

Returns spectra normalized to input magnitudes.

Parameters:
  • stdwave – 1D array of standard star wavelengths [Angstroms]

  • stdflux – 1D observed flux

  • mag – float desired magnitude

  • band – G,R,Z,W1 or W2

  • photsys – N or S (for Legacy Survey North or South)

Returns:

same as input normflux : normalized flux array

Return type:

stdwave

Only SDSS_r band is assumed to be used for normalization for now.

desispec.fluxcalibration.qa_fluxcalib(param, frame, fluxcalib)[source]
Parameters:
  • param – dict of QA parameters

  • frame – Frame

  • fluxcalib – FluxCalib

Returns:

dict of QA outputs

Need to record simple Python objects for yaml (str, float, int)

Return type:

qadict

desispec.fluxcalibration.redshift_fit(wave, flux, ivar, resolution_data, stdwave, stdflux, z_max=0.005, z_res=5e-05, template_error=0.0)[source]

Redshift fit of a single template

Parameters:
  • wave – A dictionary of 1D array of vacuum wavelengths [Angstroms]. Example below.

  • flux – A dictionary of 1D observed flux for the star

  • ivar – A dictionary 1D inverse variance of flux

  • resolution_data – resolution corresponding to the star’s fiber

  • stdwave – 1D standard star template wavelengths [Angstroms]

  • stdflux – 1D[nwave] template flux

  • z_max – float, maximum blueshift and redshift in scan, has to be positive

  • z_res – float, step of of redshift scan between [-z_max,+z_max]

  • template_error – float, assumed template flux relative error

Returns:

redshift of standard star

Return type:

redshift

Notes

  • wave and stdwave can be on different grids that don’t necessarily overlap

  • wave does not have to be uniform or monotonic. Multiple cameras can be supported by concatenating their wave and flux arrays

desispec.fluxcalibration.resample_template(data_wave_per_camera, resolution_obj_per_camera, template_wave, template_flux, template_id)[source]

Resample a spectral template on the data wavelength grid. Then convolve the spectra by the resolution for each camera. Also returns the result of applySmoothingFilter. This routine is used internally in a call to multiprocessing.Pool.

Parameters:
  • data_wave_per_camera – A dictionary of 1D array of vacuum wavelengths [Angstroms], one entry per camera and exposure.

  • resolution_obj_per_camera – A dictionary of Resolution objects corresponding for the fiber, one entry per camera and exposure.

  • template_wave – 1D array, input spectral template wavelength [Angstroms] (arbitrary spacing).

  • template_flux – 1D array, input spectral template flux density.

  • template_id – int, template identification index, used to ensure matching of input/output after a multiprocessing run.

Returns:

int, template identification index, same as input. output_wave : A dictionary of 1D array of vacuum wavelengths output_flux : A dictionary of 1D array of output template flux output_norm : A dictionary of 1D array of output template smoothed flux

Return type:

template_id

desispec.focus

Utility functions for spectrographs focus.

desispec.focus.piston_and_tilt_to_gauge_offsets(camera_spectro, focus_plane_coefficients)[source]

Computes gauge offsets to apply for a given best focus plane as a function of xccd and yccd

Parameters:
  • camera_spectro – str, camera identifier starting with b, r or z for NIR

  • focus_plane_coefficients – array with 3 values (c0,c1,c2), defining the best focus plane offsets: c0+c1*(xccd/2048-1)+c2*(yccd/2048-1) in microns

Returns:

dictionnary of offsets to apply, in mm, for the “TOP”,”LEFT” and “RIGHT” gauge

desispec.focus.test_gauge_offsets()[source]

Test function

desispec.frame

Lightweight wrapper class for spectra, to be returned by io.read_frame

desispec.gpu

Utility functions for working with GPUs

class desispec.gpu.NoGPU[source]

Context manager to temporarily disable GPU usage, e.g.

with desispec.gpu.NoGPU()

blat()

desispec.gpu.free_gpu_memory()[source]

Release all cupy GPU memory; ok to call even if no GPUs

desispec.gpu.is_gpu_available()[source]

Return whether cupy and numba.cuda are installed and a GPU is available to use, and $DESI_NO_GPU is not set

desispec.gpu.redistribute_gpu_ranks(comm, method='round-robin')[source]

Redistribute which MPI ranks are assigned to which GPUs

Parameters:

comm – MPI communicator, or None

Options:

method: ‘round-robin’ (default) or ‘contiguous’

Returns:

device_id assigned (-1 if no GPUs)

‘round-robin’ assigns cyclically, e.g. 8 ranks on 4 GPUs would be assigned [0,1,2,3,0,1,2,3].

‘continuous’ assigns contiguous ranks to the same GPU, e.g. [0,0,1,1,2,2,3,3,4,4].

CAUTION: If the MPI communicator spans multiple nodes, this assumes that all nodes have the same number of GPUs, the same number of ranks per node, and that the MPI ranks are themselves contiguously assigned to nodes (which is often the case, but not required in general).

If comm is None, assign the process to GPU 0 (if present).

Note: this also calls free_gpu_memory to release memory on each GPU before assigning ranks to other GPUs.

desispec.heliocentric

heliocentric correction routine

desispec.heliocentric.barycentric_velocity_corr_kms(ra, dec, mjd)[source]

Barycentric velocity correction routine.

See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final barycentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the heliocentric frame.

Parameters:
  • system (dec - Declination [degrees] in ICRS) –

  • system

  • DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –

Returns:

vcorr - Velocity correction term, in km/s, to add to measured

radial velocity to convert it to the heliocentric frame.

desispec.heliocentric.barycentric_velocity_multiplicative_corr(ra, dec, mjd)[source]

Barycentric velocity correction routine.

See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final barycentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the barycentric frame.

Parameters:
  • system (dec - Declination [degrees] in ICRS) –

  • system

  • DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –

Returns:

(1+vcorr/c) - multiplicative term to correct the wavelength

desispec.heliocentric.heliocentric_velocity_corr_kms(ra, dec, mjd)[source]

Heliocentric velocity correction routine.

See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final heliocentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the heliocentric frame.

Parameters:
  • system (dec - Declination [degrees] in ICRS) –

  • system

  • DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –

Returns:

vcorr - Velocity correction term, in km/s, to add to measured

radial velocity to convert it to the heliocentric frame.

desispec.heliocentric.heliocentric_velocity_multiplicative_corr(ra, dec, mjd)[source]

Heliocentric velocity correction routine.

See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final heliocentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the heliocentric frame.

Parameters:
  • system (dec - Declination [degrees] in ICRS) –

  • system

  • DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –

Returns:

(1+vcorr/c) - multiplicative term to correct the wavelength

desispec.heliocentric.main()[source]

Entry-point for command-line scripts.

Comparison test with IDL routine:

pro helio
dec=20
epoch=2000
longitude=-111.59989
latitude=31.96403
altitude=2097
for j=0,2 do begin
mjd=58600+100*j
jd=mjd+2400000.5
for i=0,20 do begin
    ra=(360.*i)/20
    vkms = heliocentric(ra, dec, epoch, jd=jd, longitude=longitude, latitude=latitude, altitude=altitude)
    print,"ra,dec,mjd,vkms=",ra,dec,mjd,vkms
endfor
endfor
end


ra,dec,mjd,vkms=      0.00000      20       58600      -12.407597
ra,dec,mjd,vkms=      18.0000      20       58600      -5.1777692
ra,dec,mjd,vkms=      36.0000      20       58600       2.8805023
ra,dec,mjd,vkms=      54.0000      20       58600       10.978417
ra,dec,mjd,vkms=      72.0000      20       58600       18.323295
ra,dec,mjd,vkms=      90.0000      20       58600       24.196169
ra,dec,mjd,vkms=      108.000      20       58600       28.022160
ra,dec,mjd,vkms=      126.000      20       58600       29.426754
ra,dec,mjd,vkms=      144.000      20       58600       28.272459
ra,dec,mjd,vkms=      162.000      20       58600       24.672266
ra,dec,mjd,vkms=      180.000      20       58600       18.978587
ra,dec,mjd,vkms=      198.000      20       58600       11.748759
ra,dec,mjd,vkms=      216.000      20       58600       3.6904873
ra,dec,mjd,vkms=      234.000      20       58600      -4.4074277
ra,dec,mjd,vkms=      252.000      20       58600      -11.752306
ra,dec,mjd,vkms=      270.000      20       58600      -17.625179
ra,dec,mjd,vkms=      288.000      20       58600      -21.451170
ra,dec,mjd,vkms=      306.000      20       58600      -22.855764
ra,dec,mjd,vkms=      324.000      20       58600      -21.701469
ra,dec,mjd,vkms=      342.000      20       58600      -18.101277
ra,dec,mjd,vkms=      360.000      20       58600      -12.407597
ra,dec,mjd,vkms=      0.00000      20       58700      -23.152438
ra,dec,mjd,vkms=      18.0000      20       58700      -27.333872
ra,dec,mjd,vkms=      36.0000      20       58700      -29.104026
ra,dec,mjd,vkms=      54.0000      20       58700      -28.289624
ra,dec,mjd,vkms=      72.0000      20       58700      -24.970386
ra,dec,mjd,vkms=      90.0000      20       58700      -19.471222
ra,dec,mjd,vkms=      108.000      20       58700      -12.330428
ra,dec,mjd,vkms=      126.000      20       58700      -4.2469952
ra,dec,mjd,vkms=      144.000      20       58700       3.9878138
ra,dec,mjd,vkms=      162.000      20       58700       11.567919
ra,dec,mjd,vkms=      180.000      20       58700       17.751326
ra,dec,mjd,vkms=      198.000      20       58700       21.932760
ra,dec,mjd,vkms=      216.000      20       58700       23.702914
ra,dec,mjd,vkms=      234.000      20       58700       22.888512
ra,dec,mjd,vkms=      252.000      20       58700       19.569274
ra,dec,mjd,vkms=      270.000      20       58700       14.070110
ra,dec,mjd,vkms=      288.000      20       58700       6.9293160
ra,dec,mjd,vkms=      306.000      20       58700      -1.1541168
ra,dec,mjd,vkms=      324.000      20       58700      -9.3889258
ra,dec,mjd,vkms=      342.000      20       58700      -16.969031
ra,dec,mjd,vkms=      360.000      20       58700      -23.152438
ra,dec,mjd,vkms=      0.00000      20       58800       19.006564
ra,dec,mjd,vkms=      18.0000      20       58800       12.826196
ra,dec,mjd,vkms=      36.0000      20       58800       5.1371364
ra,dec,mjd,vkms=      54.0000      20       58800      -3.3079572
ra,dec,mjd,vkms=      72.0000      20       58800      -11.682420
ra,dec,mjd,vkms=      90.0000      20       58800      -19.166501
ra,dec,mjd,vkms=      108.000      20       58800      -25.027606
ra,dec,mjd,vkms=      126.000      20       58800      -28.692009
ra,dec,mjd,vkms=      144.000      20       58800      -29.801014
ra,dec,mjd,vkms=      162.000      20       58800      -28.246063
ra,dec,mjd,vkms=      180.000      20       58800      -24.179365
ra,dec,mjd,vkms=      198.000      20       58800      -17.998997
ra,dec,mjd,vkms=      216.000      20       58800      -10.309937
ra,dec,mjd,vkms=      234.000      20       58800      -1.8648438
ra,dec,mjd,vkms=      252.000      20       58800       6.5096188
ra,dec,mjd,vkms=      270.000      20       58800       13.993700
ra,dec,mjd,vkms=      288.000      20       58800       19.854805
ra,dec,mjd,vkms=      306.000      20       58800       23.519208
ra,dec,mjd,vkms=      324.000      20       58800       24.628213
ra,dec,mjd,vkms=      342.000      20       58800       23.073262
ra,dec,mjd,vkms=      360.000      20       58800       19.006564

desispec.image

Lightweight wrapper class for preprocessed image data.

desispec.image_model

Function to provide rapidly a model of a preprocessed image in order to have a pixel variance estimation that is almost not correlated with pixel value.

desispec.image_model.compute_image_model(image, xytraceset, fiberflat=None, fibermap=None, with_spectral_smoothing=True, with_sky_model=True, spectral_smoothing_sigma_length=10.0, spectral_smoothing_nsig=4.0, psf=None, fit_x_shift=True)[source]

Returns a model of the input image, using a fast extraction, a processing of spectra with a common sky model and a smoothing, followed by a reprojection on the CCD image.

Parameters:
  • image – a preprocessed image in the form of a desispec.image.Image object

  • xytraceset – a desispec.xytraceset.XYTraceSet object with trace coordinates

  • fiberflat – a desispec.fiberflat.FiberFlat object

  • optional – a desispec.fiberflat.FiberFlat object

  • with_spectral_smoothing – try and smooth the spectra to reduce noise (and eventualy reduce variance correlation)

  • optional – try and smooth the spectra to reduce noise (and eventualy reduce variance correlation)

  • with_sky_model – use a sky model as part of the spectral modeling to reduce the noise (requires a fiberflat)

  • optional – use a sky model as part of the spectral modeling to reduce the noise (requires a fiberflat)

  • spectral_smoothing_sigma_length – sigma of Gaussian smoothing along wavelength in A

  • optional – sigma of Gaussian smoothing along wavelength in A

  • spectral_smoothing_nsig – number of sigma rejection threshold to fall back to the original extracted spectrum instead of the smooth one

  • optional – number of sigma rejection threshold to fall back to the original extracted spectrum instead of the smooth one

  • psf (slow, by default=None, in which case a Gaussian profile is used) –

  • projection (optional specter.psf.GaussHermitePSF object to be used for the 1D) –

  • fit_x_shift – fit for an offset of the spectral traces.

  • optional – fit for an offset of the spectral traces.

Returns:

a 2D np.array of same shape as image.pix

desispec.image_model.numba_proj(image, x, sigma, flux)

Add a spectrum to a model of the pixel values in a CCD image assuming a Gaussian cross-dispersion profile.

Parameters:
  • image – 2D numpy array , this array will be modified

  • x – 1D numpy array of size image.shape[0], coordinate of center of cross-dispersion profile of spectral trace, in pixel units, for each CCD row

  • sigma – 1D numpy array of size image.shape[0], sigma of cross-dispersion profile of spectral trace, in pixel units, for each CCD row

  • flux – 1D numpy array of size image.shape[0], quantity to project (for design usage: electrons per pixel) for each CCD row

desispec.interpolation

Utility functions for interpolation of spectra over different wavelength grids.

desispec.interpolation._unweighted_resample(output_x, input_x, input_flux_density, extrapolate=False)[source]

Returns a flux conserving resampling of an input flux density. The total integrated flux is conserved.

Parameters:
  • output_x – SORTED vector, not necessarily linearly spaced

  • input_x – SORTED vector, not necessarily linearly spaced

  • input_flux_density – input flux density dflux/dx sampled at x

both must represent the same quantity with the same unit input_flux_density = dflux/dx sampled at input_x

Options:
extrapolate: extrapolate using edge values of input array, default is False,

in which case values outside of input array are set to zero

Returns:

returns output_flux

This interpolation conserves flux such that, on average, output_flux_density = input_flux_density

The input flux density outside of the range defined by the edges of the first and last bins is considered null. The bin size of bin ‘i’ is given by (x[i+1]-x[i-1])/2 except for the first and last bin where it is (x[1]-x[0]) and (x[-1]-x[-2]) so flux density is zero for x<x[0]-(x[1]-x[0])/2 and x>x[-1]+(x[-1]-x[-2])/2

The input is interpreted as the nodes positions and node values of a piece-wise linear function:

y(x) = sum_i y_i * f_i(x)
with::
f_i(x) = (x_{i-1}<x<=x_{i})*(x-x_{i-1})/(x_{i}-x_{i-1})
  • (x_{i}<x<=x_{i+1})*(x-x_{i+1})/(x_{i}-x_{i+1})

the output value is the average flux density in a bin flux_out(j) = int_{x>(x_{j-1}+x_j)/2}^{x<(x_j+x_{j+1})/2} y(x) dx / 0.5*(x_{j+1}+x_{j-1})

desispec.interpolation.resample_flux(xout, x, flux, ivar=None, extrapolate=False)[source]

Returns a flux conserving resampling of an input flux density. The total integrated flux is conserved.

Parameters:
  • xout (-) – output SORTED vector, not necessarily linearly spaced

  • x (-) – input SORTED vector, not necessarily linearly spaced

  • flux (-) – input flux density dflux/dx sampled at x

both x and xout must represent the same quantity with the same unit

Options:
  • ivar: weights for flux; default is unweighted resampling

  • extrapolate: extrapolate using edge values of input array, default is False, in which case values outside of input array are set to zero.

Setting both ivar and extrapolate raises a ValueError because one cannot assign an ivar outside of the input data range.

Returns:

if ivar is None, returns outflux if ivar is not None, returns outflux, outivar

This interpolation conserves flux such that, on average, output_flux_density = input_flux_density

The input flux density outside of the range defined by the edges of the first and last bins is considered null. The bin size of bin ‘i’ is given by (x[i+1]-x[i-1])/2 except for the first and last bin where it is (x[1]-x[0]) and (x[-1]-x[-2]) so flux density is zero for x<x[0]-(x[1]-x[0])/2 and x>x[-1]-(x[-1]-x[-2])/2

The input is interpreted as the nodes positions and node values of a piece-wise linear function:

y(x) = sum_i y_i * f_i(x)

with:

f_i(x) =    (x_{i-1}<x<=x_{i})*(x-x_{i-1})/(x_{i}-x_{i-1})
          + (x_{i}<x<=x_{i+1})*(x-x_{i+1})/(x_{i}-x_{i+1})

the output value is the average flux density in a bin:

flux_out(j) = int_{x>(x_{j-1}+x_j)/2}^{x<(x_j+x_{j+1})/2} y(x) dx /  0.5*(x_{j+1}+x_{j-1})

desispec.io

Tools for data and metadata I/O.

desispec.io.download

Download files from DESI repository.

desispec.io.download._auth(machine='data.desi.lbl.gov')[source]

Get authentication credentials.

desispec.io.download._map_download(map_tuple)[source]

Wrapper function to pass to multiprocess.Pool.map().

desispec.io.download.download(filenames, single_thread=False, workers=None, baseurl='https://data.desi.lbl.gov/desi')[source]

Download files from the DESI repository.

This function will try to create any directories that don’t already exist, using the exact paths specified in filenames.

Parameters:
  • filenames – string or list-like object containing filenames.

  • single_thread – (optional) if True, do not use multiprocessing to download files.

  • workers – (optional) integer indicating the number of worker processes to create.

  • baseurl – (optional) string containing the URL of the top-level DESI directory.

Returns:

Full, local path to the file(s) downloaded.

desispec.io.emlinefit

IO routines for desi_emlinefit_afterburner.

desispec.io.emlinefit.get_targetids(d, bitnames, log=None)[source]

Returns the TARGETIDs passing the bitnames for the CMX_TARGET, SV{1,2,3}_DESI_TARGET, or DESI_TARGET mask.

Parameters:
  • d – structured array, typically a FIBERMAP catalog, with TARGETID and the CMX_TARGET, SV{1,2,3}_DESI_TARGET, or DESI_TARGET column.

  • bitnames – comma-separated list of target bitnames to fit from the *DESI_TARGET mask (string)

  • log (optional, defaults to get_logger()) – Logger object

Returns:

list of TARGETIDs passing the {CMX,*DESI}_TARGET mask (np.array)

Return type:

targetids

Note

  • If several bitnames are provided, selects the union of those.

  • Safer if d is already trimmed on unique TARGETIDs (see FIBERMAP format with zbest-*.fits files)

desispec.io.emlinefit.plot_emlines(outpdf, zspecs, emdict, emnames=['OII', 'HDELTA', 'HGAMMA', 'HBETA', 'OIII', 'HALPHA'], targetids=None, objtypes=None, spectypes=None, deltachi2s=None, ylim=(None, None), nrow=10, rowsort_byz=True)[source]

Plot the fitted data from get_emlines().

Parameters:
  • outpdf – output pdf filename (string)

  • zspecs – list of redrock redshifts (numpy array)

  • emdict – a dictionary with various quantities, output by get_lines().

  • emnames (optional, defaults to ["OII", "HDELTA", "HGAMMA", "HBETA", "OIII", "HALPHA"]) – list of plotted lines (list of string)

  • targetids (optional, defaults to None) – list of TARGETIDs (numpy array)

  • objtypes (optional, defaults to None) – list of fibermap OBJTYPEs (numpy array)

  • spectypes (optional, defaults to None) – list of redrock SPECTYPEs (numpy array)

  • deltachi2s (optional, defaults to None) – list of redrock DELTACHI2s (numpy array)

  • ylim (optional, defaults to (None, None)) – ylim for plotting (float doublet)

  • nrow (optional, defaults to 10) – number of rows, i.e. galaxy, per pdf page (int)

  • rowsort_byz (optional, defaults to True) – the spectra are displayed by increasing redshifts (boolean)

Note

  • For emdict:

    • Each emission line has its own subdictionary.

    • Each emission line dictionary should contain these keys:

      • CONT : continuum in 1e-17 * erg/cm2/s/A

      • FLUX, FLUX_IVAR: flux in 1e-17 * erg/cm2/s/A

      • waves, fluxes, ivars, models: data used for fitting + fitted model

  • If some emname is not present in emdict, it will be discarded.

  • Each spectra will be one row of plot, each emission line corresponds to a column.

  • If rowsort_byz=False, then the spectra will be displayed by order of appearance.

desispec.io.emlinefit.read_emlines_inputs(redrock, coadd, mwext_corr=True, rv=3.1, bitnames='ALL', targetids=None, rr_keys='TARGETID,Z,ZWARN,SPECTYPE,DELTACHI2', fm_keys='TARGET_RA,TARGET_DEC,OBJTYPE', log=None)[source]

Read the columns and spectra information (waves, fluxes, ivars) from the redrock and coadd files, which will go as input to get_emlines().

Parameters:
  • redrock – full path to a redrock/zbest file

  • coadd – full path to a coadd file (everest-format)

  • mwext_corr (optional, defaults to True) – correct flux for foreground MW extinction? (boolean)

  • rv (optional, defaults to 3.1) – value of R_V, used if mwext_corr=True (float)

  • bitnames (optional, defaults to "ALL", meaning fitting all fibers) – comma-separated list of target bitnames to fit from the *DESI_TARGET mask (string)

  • targetids (optional, defaults to None) – list of TARGETIDs to restrict to (int, list, or numpy array)

  • rr_keys (optional, defaults to "TARGETID,Z,ZWARN,SPECTYPE,DELTACHI2") – comma-separated list of columns from REDSHIFTS to propagate (string)

  • fm_keys (optional, defaults to "TARGET_RA,TARGET_DEC,OBJTYPE") – comma-separated list of columns from FIBERMAP to propagate (string)

  • log (optional, defaults to get_logger()) – Logger object

Returns:

structured array with the desired redrock columns for the desired TARGETIDs fm: structured array with the desired fibermap columns for the desired TARGETIDs waves: wavelengths (numpy array of shape (Nwave)) fluxes: Galactic-extinction-corrected fluxes (numpy array of shape (Nspec, Nwave)) ivars: inverse variances (numpy array of shape (Nspec, Nwave))

Return type:

rr

Note

  • We add TARGETID and Z to rr_keys if TARGETID not present in rr_keys nor in fm_keys.

  • If keys in rr_keys or fm_keys are not present in the redrock, those will be ignored.

  • If both bitnames and targetids are provided, we take the overlap of the two.

desispec.io.emlinefit.write_emlines(output, emdict, rr=None, fm=None, redrock=None, coadd=None, rf_fit_hw=None, min_rf_fit_hw=None, rf_cont_w=None, rv=None, log=None)[source]

Writes the emission line fitting to a file.

Parameters:
  • output – output fits file (string)

  • emdict – dictionary output by get_emlines()

  • rr (optional, defaults to None) – redrock REDSHIFTS/ZBEST extension

  • fm (optional, defaults to None) – redrock FIBERMAP extension

  • redrock (optional, defaults to None) – used redrock/zbest file

  • coadd (optional, defaults to None) – used coadd file (everest-format)

  • rf_fit_hw (optional, defaults to None) – rest-frame wavelength width (in A) used for fitting on each side of the line (float)

  • min_rf_fit_hw (optional, defaults to None) – minimum requested rest-frame width (in A) on each side of the line to consider the fitting (float)

  • rf_cont_w (optional, defaults to None) – rest-frame wavelength extent (in A) to fit the continuum (float)

  • rv (optional, defaults to None) – value of R_V (float)

  • log (optional, defaults to get_logger()) – Logger object

desispec.io.exposure_tile_qa

I/O routines for exposure and tile qa

desispec.io.exposure_tile_qa.read_exposure_qa(filename)[source]

Reads an exposure-qa fits file.

Parameters:

filename – full path to file

Returnsfiber_qa_table, petal_qa_table

two astropy.table.Table objects

desispec.io.exposure_tile_qa.read_tile_qa(filename)[source]

Reads an tile-qa fits file.

Parameters:

filename – full path to file

Returnsfiber_qa_table, petal_qa_table

two astropy.table.Table objects

desispec.io.exposure_tile_qa.write_exposure_qa(filename, fiber_qa_table, petal_qa_table=None)[source]

Writes an exposure-qa fits file.

Parameters:
  • filename – full path to file

  • fiber_qa_table – astropy.table.Table object with one row per target

  • petal_qa_table – astropy.table.Table object with one row per petal

desispec.io.exposure_tile_qa.write_tile_qa(filename, fiber_qa_table, petal_qa_table=None)[source]

Writes an tile-qa fits file.

Parameters:
  • filename – full path to file

  • fiber_qa_table – astropy.table.Table object with one row per target

  • petal_qa_table – astropy.table.Table object with one row per petal

desispec.io.fiberflat

IO routines for fiberflat.

desispec.io.fiberflat.read_fiberflat(filename)[source]

Read fiberflat from filename

Parameters:

filename (str) – Name of fiberflat file, or (night, expid, camera) tuple

Returns:

FiberFlat object with attributes

fiberflat, ivar, mask, meanspec, wave, header

Notes

fiberflat, ivar, mask are 2D [nspec, nwave] meanspec and wave are 1D [nwave]

desispec.io.fiberflat.write_fiberflat(outfile, fiberflat, header=None, fibermap=None)[source]

Write fiberflat object to outfile

Parameters:
  • outfile – filepath string or (night, expid, camera) tuple

  • fiberflat – FiberFlat object

Optional:

header: dict or fits.Header object to use as HDU 0 header fibermap: table to store as FIBERMAP HDU

Returns:

filepath of file that was written

desispec.io.fiberflat_vs_humidity

desispec.io.fiberflat_vs_humidity.read_fiberflat_vs_humidity(filename)[source]

Read fiberflat vs humidity from filename

Parameters:

filename (str) – path to fiberflat_vs_humidity file

Returns: fiberflat , humidity , wave

fiberflat is 3D [nhumid, nspec, nwave] humidity is 1D [nhumid] (and in percent) wave is 1D [nwave] (and in Angstrom) header (fits header)

desispec.io.fibermap

IO routines for fibermap.

desispec.io.fibermap._set_fibermap_columns()[source]

Prepare survey-specific list of columns.

Returns:

As a convenience, return the full set of survey-specific columns.

Return type:

dict

desispec.io.fibermap.assemble_fibermap(night, expid, badamps=None, badfibers_filename=None, force=False, allow_svn_override=True)[source]

Create a fibermap for a given night and expid.

Parameters:
  • night (int) – YEARMMDD night of sunset.

  • expid (int) – Exposure ID.

  • badamps (str, optional) – Comma separated list of "{camera}{petal}{amp}", i.e. "[brz][0-9][ABCD]". Example: 'b7D,z8A'.

  • badfibers_filename (str, optional) – Filename with table of bad fibers with at least two columns: FIBER and FIBERSTATUS

  • force (bool, optional) – Create fibermap even if missing coordinates/guide files.

  • allow_svn_override (bool, optional) – If True (default), allow fiberassign SVN to override raw data.

Returns:

A representation of a fibermap FITS file.

Return type:

astropy.io.fits.HDUList

desispec.io.fibermap.compare_fiberassign(fa1, fa2, compare_all=False)[source]

Check whether two fiberassign tables agree for cols used by ICS/platemaker

Parameters:
  • fa1 – fiberassign astropy Tables or numpy structured arrays

  • fa2 – fiberassign astropy Tables or numpy structured arrays

Options:

compare_all: if True, compare all columns, not just those used by ops

Returns list of columns with mismatches; empty list if all agree

Note: if both are NaN, it is considered a match

desispec.io.fibermap.empty_fibermap(nspec, specmin=0, survey='main')[source]

Return an empty fibermap Table to be filled in.

Parameters:
  • nspec (int) – Number of fibers (spectra) to include.

  • specmin (int, optional) – Staring spectrum index.

  • survey (string, optional) – Define columns for this survey; default ‘main’.

Returns:

An empty Table.

Return type:

Table

desispec.io.fibermap.fibermap_new2old(fibermap)[source]

Converts new format fibermap into old format fibermap

Parameters:

fibermap – new-format fibermap table (e.g. with FLUX_G column)

Returns:

old format fibermap (e.g. with MAG column)

Note: this is a transitional convenience function to allow us to simulate new format fibermaps while still running code that expects the old format. After all code has been converted to use the new format, this will be removed.

desispec.io.fibermap.find_fiberassign_file(night, expid, tileid=None, nightdir=None)[source]

Walk backwards in exposures to find matching fiberassign file

Parameters:
  • night (int) – YEARMMDD night of observations

  • expid (int) – spectroscopic exposure ID

Options:

tileid (int): tileid to look for nightdir (str): base directory for raw data on that night

Returns first fiberassign file found on or before expid on night.

Raises FileNotFoundError if no fiberassign file is found

desispec.io.fibermap.read_fibermap(fp)[source]

Reads a fibermap file and returns its data as an astropy Table

Parameters:

fp – input file name, or opened fitsio.FITS or astropy.io.fits.HDUList

desispec.io.fibermap.update_survey_keywords(hdr)[source]

Standardize fiberassign header keywords SURVEY, FAPRGRM, FAFLAVOR

Parameters:

hdr (dict-like) – HDU 0 header from a fiberassign file

Returns:

value pairs added

Return type:

dict of key

Note: updates hdr object in-place

desispec.io.fibermap.write_fibermap(outfile, fibermap, header=None, clobber=True, extname='FIBERMAP')[source]

Write fibermap binary table to outfile.

Parameters:
  • outfile (str) – output filename

  • fibermap – astropy Table of fibermap data

  • header – header data to include in same HDU as fibermap

  • clobber (bool, optional) – overwrite outfile if it exists

  • extname (str, optional) – set the extension name.

Returns:

full path to filename of fibermap file written.

Return type:

write_fibermap (str)

desispec.io.filters

desispec.io.filters.load_filter(given_filter)[source]

Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object

Parameters:
  • given_filter – given filter for which the qe is to be loaded. Desi templates/

  • uppercase (files have them in) –

  • SDSS (so it should be in upper case like) –

  • or (DECAM) –

  • here. (WISE. Speclite has lower case so are mapped) –

desispec.io.filters.load_gaia_filter(band, dr=2)[source]

Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object

Parameters:
  • band – filter pass-band in “G”,”BP”,”RP”

  • dr – 2 or 3

desispec.io.filters.load_legacy_survey_filter(band, photsys)[source]

Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object

Parameters:
  • band – filter pass-band in “G”,”R”,”Z”,”W1”,”W2”

  • photsys – “N” or “S” for North (BASS+MzLS) or South (CTIO/DECam)

desispec.io.fluxcalibration

IO routines for flux calibration.

desispec.io.fluxcalibration.read_average_flux_calibration(filename)[source]

Read average flux calibration file; returns an AverageFluxCalib object

desispec.io.fluxcalibration.read_flux_calibration(filename)[source]

Read flux calibration file; returns a FluxCalib object

desispec.io.fluxcalibration.read_stdstar_models(filename)[source]

Read stdstar models from filename.

Parameters:

filename (str) – File containing standard star models.

Returns:

flux[nspec, nwave], wave[nwave], fibers[nspec]

Return type:

read_stdstar_models (tuple)

desispec.io.fluxcalibration.read_stdstar_templates(stellarmodelfile)[source]

Reads an input stellar model file

Parameters:

stellarmodelfile – input filename

Returns (wave, flux, templateid, teff, logg, feh) tuple:

wave : 1D[nwave] array of wavelengths [Angstroms] flux : 2D[nmodel, nwave] array of model fluxes templateid : 1D[nmodel] array of template IDs for each spectrum teff : 1D[nmodel] array of effective temperature for each model logg : 1D[nmodel] array of surface gravity for each model feh : 1D[nmodel] array of metallicity for each model

desispec.io.fluxcalibration.write_average_flux_calibration(outfile, averagefluxcalib)[source]

Writes average flux calibration.

Parameters:
  • outfile – output file name

  • averagefluxcalib – AverageFluxCalib object

Options:

header : dict-like object of key/value pairs to include in header

desispec.io.fluxcalibration.write_flux_calibration(outfile, fluxcalib, header=None)[source]

Writes flux calibration.

Parameters:
  • outfile – output file name

  • fluxcalib – FluxCalib object

Options:

header : dict-like object of key/value pairs to include in header

desispec.io.fluxcalibration.write_stdstar_models(norm_modelfile, normalizedFlux, wave, fibers, data, fibermap, input_frames, header=None)[source]

Writes the normalized flux for the best models.

Parameters:
  • norm_modelfile – output file path

  • normalizedFlux – 2D array of flux[nstdstars, nwave]

  • wave – 1D array of wavelengths[nwave] in Angstroms

  • fibers – 1D array of fiberids for these spectra

  • data – meta data table about which templates best fit

  • fibermap – fibermaps rows for the input standard stars

  • input_frames – Table with NIGHT, EXPID, CAMERA of input frames used

desispec.io.frame

I/O routines for Frame objects

desispec.io.frame.read_frame(filename, nspec=None, skip_resolution=False)[source]

Reads a frame fits file and returns its data.

Parameters:
  • filename – path to a file, or (night, expid, camera) tuple where night = string YEARMMDD expid = integer exposure ID camera = b0, r1, .. z9

  • skip_resolution – bool, option Speed up read time (>5x) by avoiding the Resolution matrix

Returns:

desispec.Frame object with attributes wave, flux, ivar, etc.

desispec.io.frame.read_meta_frame(filename, extname=0)[source]

Load the meta information of a Frame :param filename: path to a file :param extname: int, optional; Extension for grabbing header info

Returns:

dict or astropy.fits.header

Return type:

meta

desispec.io.frame.search_for_framefile(frame_file, specprod_dir=None)[source]

Search for an input frame_file in the desispec redux hierarchy :param frame_file: str :param specprod_dir: str, optional

Returns:

str, full path to frame_file if found else raise error

Return type:

mfile

desispec.io.frame.write_frame(outfile, frame, header=None, fibermap=None, units=None)[source]

Write a frame fits file and returns path to file written.

Parameters:
  • outfile – full path to output file, or tuple (night, expid, channel)

  • frame – desispec.frame.Frame object with wave, flux, ivar…

Optional:

header: astropy.io.fits.Header or dict to override frame.header fibermap: table to store as FIBERMAP HDU

Returns:

full filepath of output file that was written

Note

to create a Frame object to pass into write_frame, frame = Frame(wave, flux, ivar, resolution_data)

desispec.io.image

I/O routines for Image objects

desispec.io.image.read_image(filename)[source]

Returns desispec.image.Image object from input file

desispec.io.image.write_image(outfile, image, meta=None)[source]

Writes image object to outfile

Parameters:
  • outfile – output file string

  • image – desispec.image.Image object (or any object with 2D array attributes image, ivar, mask)

Optional:

meta : dict-like object with metadata key/values (e.g. FITS header)

desispec.io.iotime

Utilities for parsing and plotting I/O timing from logfiles.

desispec.io.iotime._ordered_unique_names(names)[source]

Return unique list of names, ordered by first appearance in list

Doesn’t scale well; intended for inputs <10000 long

desispec.io.iotime.format(readwrite, filename, duration)[source]

Return standardized I/O timing message string for logging

Parameters:
  • readwrite (str) – either “read” or “write”

  • filename (str) – filename that was read or written

  • duration (float) – time in seconds to perform I/O operations

Returns: I/O timing message to log

Note: this function does not call log.info() itself so that the logging source file and line number can be associated with the I/O function itself instead of this utility formatting function.

desispec.io.iotime.hist_iotimes(timing, tmax=10, plottitle=None)[source]

Histogram function timing from Table read with read_iotimes

Parameters:

timing – Table with columns FUNC, DATETIME, IOTIME

Options:

tmax (float): upper bound of histogram (overflows included in last bin)o plottitle (str): plot title

Returns matplotlib Figure

desispec.io.iotime.parse(line)[source]

Parse a line for an iotime message produced by format

Parameters:

line (str) – the line to parse

Returns None if no match, or dict with keys function,duration,readwrite,filename,timestamp

desispec.io.iotime.parse_logfile(logfile)[source]

Read iotime log entries from logfile

Return Table with columns function duration readwrite filename timestamp datetime

desispec.io.iotime.plot_iotimes(timing, plottitle=None, outfile=None)[source]

Plot I/O duration vs. time of I/O operation

Parameters:

timing – Table with columns FUNC, DATETIME, IOTIME

Options:

plottitle (str): Title to include for plot outfile (str): write plot to this file

Returns matplotlib figure; does not call plt.show()

desispec.io.meta

IO metadata functions.

desispec.io.meta.faflavor2program(faflavor)[source]

Map FAFLAVOR keywords to what we wish we had set for FAPRGRM

Parameters:

faflavor (str or array of str) – FAFLAVOR keywords from fiberassign

Returns:

what FAPRGM would be if we had set it (dark, bright, backup, other)

Return type:

faprgm (str or array of str)

Note: this was standardized by sv3 and main, but evolved during sv1 and sv2

desispec.io.meta.find_exposure_night(expid, specprod_dir=None)[source]

Find the night that has the exposure :param expid: int :param specprod_dir: str, optional

Returns:

str

Return type:

night

desispec.io.meta.findfile(filetype, night=None, expid=None, camera=None, tile=None, groupname=None, healpix=None, nside=64, band=None, spectrograph=None, survey=None, faprogram=None, rawdata_dir=None, specprod_dir=None, download=False, outdir=None, qaprod_dir=None, return_exists=False, readonly=False, logfile=False)[source]

Returns location where file should be

Parameters:

filetype – file type, typically the prefix, e.g. “frame” or “psf”

Args depending upon filetype:

night : YEARMMDD string expid : integer exposure id camera : ‘b0’ ‘r1’ .. ‘z9’ tile : integer tile (pointing) number groupname : spectral grouping name (e.g. “healpix”, “cumulative”, “pernight”) healpix : healpix pixel number nside : healpix nside band : one of ‘b’,’r’,’z’ identifying the camera band spectrograph : integer spectrograph number, 0-9 survey : e.g. sv1, sv3, main, special faprogram : fiberassign program, e.g. dark, bright

Options:

rawdata_dir : overrides $DESI_SPECTRO_DATA specprod_dir : overrides $DESI_SPECTRO_REDUX/$SPECPROD/ qaprod_dir : defaults to $DESI_SPECTRO_REDUX/$SPECPROD/QA/ if not provided download : if not found locally, try to fetch remotely outdir : use this directory for output instead of canonical location return_exists: if True, also return whether the file exists readonly: if True, return read-only version of path if possible logfile: if True, returns the pathname of the log instead of the data product itself

Returns filename, or (filename, exists) if return_exists=True

Raises:
  • ValueError – for invalid file types, and other invalid input

  • KeyError – for missing environment variables

Notes

The readonly option uses $DESI_ROOT_READONLY if it is set and exists; otherwise it returns the normal read/write path.

desispec.io.meta.get_desi_root_readonly()[source]

Returns $DESI_ROOT_READONLY if set and path exists, otherwise $DESI_ROOT.

Caches answer upon first call, i.e. setting $DESI_ROOT_READONLY to a different value part way through running will use previously cached value. This prevents it from re-checking a non-existent path N>>1 times.

desispec.io.meta.get_exposures(night, raw=False, rawdata_dir=None, specprod_dir=None)[source]

Get a list of available exposures for the specified night.

Exposures are identified as correctly formatted subdirectory names within the night directory, but no checks for valid contents of these exposure subdirectories are performed.

Parameters:
  • night (str) – Date string for the requested night in the format YYYYMMDD.

  • raw (bool) – Returns raw exposures if set, otherwise returns processed exposures.

  • rawdata_dir (str) – [optional] overrides $DESI_SPECTRO_DATA

  • specprod_dir (str) – Path containing the exposures/ directory to use. If the value is None, then the value of specprod_root() is used instead. Ignored when raw is True.

Returns:

List of integer exposure numbers available for the specified night. The

list will be empty if no the night directory exists but does not contain any exposures.

Return type:

list

Raises:
desispec.io.meta.get_files(filetype, night, expid, specprod_dir=None, qaprod_dir=None, **kwargs)[source]

Get files for a specified exposure.

Uses findfile() to determine the valid file names for the specified type. Any camera identifiers not matching the regular expression [brz][0-9] will be silently ignored.

Parameters:
  • filetype (str) – Type of files to get. Valid choices are ‘frame’, ‘cframe’, ‘psf’, etc.

  • night (str) – Date string for the requested night in the format YYYYMMDD.

  • expid (int) – Exposure number to get files for.

  • specprod_dir (str) – Path containing the exposures/ directory to use. If the value is None, then the value of specprod_root() is used instead. Ignored when raw is True.

Returns:

Dictionary of found file names using camera id strings as keys,

which are guaranteed to match the regular expression [brz][0-9].

Return type:

dict

desispec.io.meta.get_nights(strip_path=True, specprod_dir=None, sub_folder='exposures')[source]

Generate a list of nights in a given folder (default is exposures/) Demands an 8 digit name beginning with 20

Parameters:
  • strip_path – bool, optional; Strip the path to the nights folders

  • rawdata_dir

  • specprod_dir

  • sub_root – str, optional; ‘exposures’, ‘calib2d’

Returns:

list of nights (without or with paths)

Return type:

nights

desispec.io.meta.get_nights_up_to_date(date, specprod_dir=None)[source]

Returns nights with an existing exposure_table in the current specprod

Parameters:
  • date (int) – return all dates up to and including this date

  • specprod_dir (str) – the specprod directory where the exposure_tables reside

Returns:

list of nights up to and including date that

have an exposure table

Return type:

nights (list of ints)

desispec.io.meta.get_pipe_database()[source]

Get the production database location based on the environment.

desispec.io.meta.get_pipe_logdir()[source]

Return the name of the subdirectory containing pipeline logs.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_pipe_nightdir()[source]

Return the name of the subdirectory containing per-night files.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_pipe_pixeldir()[source]

Return the name of the subdirectory containing per-pixel files.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_pipe_rundir(specprod_dir=None)[source]

Return the directory path for pipeline runtime files.

Parameters:

specprod_dir (str) – Optional path to production directory. If None, the this is obtained from specprod_root().

Returns (str):

the directory path for pipeline runtime files.

desispec.io.meta.get_pipe_scriptdir()[source]

Return the name of the subdirectory containing pipeline scripts.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_raw_files(filetype, night, expid, rawdata_dir=None)[source]

Get files for a specified exposure.

Uses findfile() to determine the valid file names for the specified type. Any camera identifiers not matching the regular expression [brz][0-9] will be silently ignored.

Parameters:
  • filetype (str) – Type of files to get. Valid choices are ‘raw’, ‘preproc’, ‘fibermap’.

  • night (str) – Date string for the requested night in the format YYYYMMDD.

  • expid (int) – Exposure number to get files for.

  • rawdata_dir (str) – [optional] overrides $DESI_SPECTRO_DATA

Returns:

Dictionary of found file names using camera id strings as keys,

which are guaranteed to match the regular expression [brz][0-9].

Return type:

dict

desispec.io.meta.get_readonly_filepath(filepath)[source]

Generate optimized path for read-only usage of filepath

Parameters:

filepath (str) – full path to input file

Returns: readonly_filepath using $DESI_ROOT_READONLY

If a readonly filepath can’t be derived, return original filepath

desispec.io.meta.get_reduced_frames(channels=['b', 'r', 'z'], nights=None, ftype='cframe', **kwargs)[source]

Loops through a production to find all reduced frames (default is cframes) One can choose a subset of reduced frames by argument :param channels: list, optional :param nights: list, optional :param ftype: str, optional :param kwargs: passed to get_files()

Returns:

list for frame filenames

Return type:

all_frames

desispec.io.meta.qaprod_root(specprod_dir=None)[source]

Return directory root for spectro production QA, i.e. $DESI_SPECTRO_REDUX/$SPECPROD/QA.

Raises:

KeyError – if these environment variables aren’t set.

desispec.io.meta.rawdata_root()[source]

Returns directory root for raw data, i.e. $DESI_SPECTRO_DATA

Raises:

KeyError – if these environment variables aren’t set.

desispec.io.meta.shorten_filename(filename)[source]

Attempt to shorten filename to fit in FITS header without CONTINUE

Parameters:

filename (str) – input filename

Returns potentially shortened filename

Replaces prefixes from environment variables:
  • $DESI_SPECTRO_CALIB -> SPCALIB

  • $DESI_SPECTRO_REDUX/$SPECPROD -> SPECPROD

desispec.io.meta.specprod_root(specprod=None)[source]

Return directory root for spectro production, i.e. $DESI_SPECTRO_REDUX/$SPECPROD.

Options:

specprod (str): overrides $SPECPROD

Raises:

KeyError – if these environment variables aren’t set.

desispec.io.meta.validate_night(night)[source]

Validates a night string and converts to a date.

Parameters:

night (str) – Date string for the requested night in the format YYYYMMDD.

Returns:

Date object representing this night.

Return type:

datetime.date

Raises:

ValueError – Badly formatted night string.

desispec.io.params

IO routines for parameter values

desispec.io.params.read_params(filename=None, reload=False)[source]

Read parameter data from file

desispec.io.photo

Simple methods for gathering photometric and targeting catalogs.

desispec.io.photo._gather_tractorphot_onebrick(input_cat, legacysurveydir, radius_match, racolumn, deccolumn, datamodel)[source]

Support routine for gather_tractorphot.

desispec.io.photo.gather_targetdirs(tileid, fiberassign_dir=None, verbose=False)[source]

Gather all the targeting directories used to build a given fiberassign catalog, including both primary and secondary targets and ToOs.

Parameters:
  • tileid (int) – tile number

  • fiberassign_dir (str, optional) – directory to fiberassign tables

  • verbose (bool) – print debugging as well as informational messages

Returns:

list of targeting directories and files

Return type:

targetdirs (numpy.ndarray)

desispec.io.photo.gather_targetphot(input_cat, photocache=None, racolumn='TARGET_RA', deccolumn='TARGET_DEC', columns=None, fiberassign_dir=None, verbose=False)[source]

Find and stack the photometric targeting information given a set of targets.

Parameters:
  • input_cat (astropy.table.Table) – input table with the following (required) columns: TARGETID, RACOLUMN, DECCOLUMN, TILEID.

  • photocache (dict, optional) – dictionary cache of targetids for large targeting catalogs.

  • racolumn (str) – name of the RA column in input_cat (defaults to RA_COLUMN)

  • deccolumn (str) – name of the RA column in input_cat (defaults to DEC_COLUMN)

  • columns (str array) – return this subset of columns

  • fiberassign_dir (str, optional) – top-level directory to fiberassign tables

Returns a table of targeting photometry using a consistent data model across primary (DR9) targets, secondary targets, and targets of opportunity. The data model is documented in targetphot_datamodel.

desispec.io.photo.gather_tractorphot(input_cat, racolumn='TARGET_RA', deccolumn='TARGET_DEC', legacysurveydir=None, dr9dir=None, radius_match=1.0, columns=None, verbose=False)[source]

Retrieve the Tractor catalog for all the objects in this catalog (one brick).

Parameters:
  • input_cat (astropy.table.Table) – input table with the following (required) columns: TARGETID, TARGET_RA, TARGET_DEC. Additional optional columns that will ensure proper matching are BRICKNAME, RELEASE, PHOTSYS, BRICKID, and BRICK_OBJID.

  • legacysurveydir (str) – full path to the location of the Tractor catalogs

  • dr9dir (str) – relegated keyword; please use legacysurveydir

  • radius_match (float, arcsec) – matching radius (default, 1 arcsec)

  • columns (str array) – return this subset of columns

Returns a table of Tractor photometry. Matches are identified either using BRICKID and BRICK_OBJID or using positional matching (1 arcsec radius).

desispec.io.photo.targetphot_datamodel(from_file=False)[source]

Initialize the targetphot data model.

Parameters:

from_file (bool, optional) – read the datamodel from a file on-disk.

Returns an astropy.table.Table with a consistent data model across cmx, sv[1-2], and main-survey observations.

desispec.io.photo.tractorphot_datamodel(from_file=False, datarelease='dr9')[source]

Initialize the tractorphot data model for a given Legacy Surveys data release.

Parameters:
  • from_file (bool, optional) – read the datamodel from a file on-disk.

  • datarelease (str, optional) – data release to read; currently only dr9 and dr10 are supported.

Returns an astropy.table.Table which follows the Tractor catalog datamodel for the given data release.

desispec.io.qa

IO routines for QA.

desispec.io.qa.load_qa_brick(filename)[source]

Load an existing QA_Brick or generate one, as needed :param filename: str

Returns: qa_brick: QA_Brick object

desispec.io.qa.load_qa_frame(filename, frame_meta=None, flavor=None)[source]

Load an existing QA_Frame or generate one, as needed

Parameters:
  • filename – str

  • frame_meta – dict like, optional

  • flavor – str, optional Type of QA_Frame

Returns:

QA_Frame object

Return type:

qa_frame

desispec.io.qa.load_qa_multiexp(inroot)[source]

Load QA for a given production

Parameters:

inroot – str base filename without format extension

Returns:

dict

Return type:

odict

desispec.io.qa.qafile_from_framefile(frame_file, qaprod_dir=None, output_dir=None)[source]

Derive the QA filename from an input frame file :param frame_file: str :param output_dir: str, optional Over-ride default output path :param qa_dir: str, optional Over-ride default QA

Returns:

desispec.io.qa.read_qa_brick(filename)[source]

Generate a QA_Brick object from a data file

desispec.io.qa.read_qa_data(filename)[source]

Read data from a QA file

desispec.io.qa.read_qa_frame(filename)[source]

Generate a QA_Frame object from a data file

desispec.io.qa.write_qa_brick(outfile, qabrick)[source]

Write QA for a given exposure

Parameters:
  • outfile – filename

  • qabrick – QA_Brick object _data: dict of QA info

desispec.io.qa.write_qa_exposure(outroot, qaexp, ret_dict=False)[source]

Write QA for a given exposure

Parameters:
  • outroot – str filename without format extension

  • qa_exp – QA_Exposure object

  • ret_dict – bool, optional Return dict only? [for qa_prod, mainly]

Returns:

str or dict

Return type:

outfile or odict

desispec.io.qa.write_qa_frame(outfile, qaframe, verbose=False)[source]

Write QA for a given frame

Parameters:
  • outfile – str filename

  • qa_exp – QA_Frame object, with the following attributes qa_data: dict of QA info

desispec.io.qa.write_qa_multiexp(outroot, mdict, indent=True)[source]

Write QA for a given production

Parameters:
  • outroot – str filename without format extension

  • mdict – dict

Returns:

str

output filename

Return type:

outfile

desispec.io.qa.write_qa_ql(outfile, qaresult)[source]

Write QL output files

Parameters:
  • outfile – str filename to be written (yaml)

  • qaresult – dict QAresults from run_qa()

Returns:

str

Return type:

outfile

desispec.io.raw

I/O for DESI raw data files

See DESI-1229 for format details TODO: move into datamodel after we have verified the format

desispec.io.raw.read_raw(filename, camera, fibermapfile=None, fill_header=None, **kwargs)[source]

Returns preprocessed raw data from camera extension of filename.

Parameters:
  • filename (str) – Input FITS filename with DESI raw data.

  • camera (str) – Camera name (B0, R1, … Z9) or FITS extension name.

  • fibermapfile (str, optional) – Read fibermap from this file; if None create blank fibermap.

  • fill_header (list, optional) – A list of HDU names or numbers. The header cards from these HDUs will be added to the header of the camera HDU read from filename.

Returns:

Image object with member variables pix, ivar, mask, readnoise.

Return type:

desispec.image.Image

Raises:
  • IOError – If camera is not a HDU in filename.

  • KeyError – If EXPTIME is not present in any header in filename, or if both NIGHT and DATE-OBS are missing from input headers.

  • ValueError – If NIGHT in the primary header does not match NIGHT in the camera header, or if fill_header is not a list.

Notes

Other keyword arguments are passed to desispec.preproc.preproc(), e.g. bias, pixflat, mask. See preproc() documentation for details.

desispec.io.raw.write_raw(filename, rawdata, header, camera=None, primary_header=None)[source]

Write raw pixel data to a DESI raw data file

Parameters:
  • filename – file name to write data; if this exists, append a new HDU

  • rawdata – 2D ndarray of raw pixel data including overscans

  • header – dict-like object or fits.Header with keywords CCDSECx, BIASSECx, DATASECx where x=A,B,C,D

Options:

camera : b0, r1 .. z9 - override value in header primary_header : header to write in HDU0 if filename doesn’t yet exist

The primary utility of this function over raw fits calls is to ensure that all necessary keywords are present before writing the file. CCDSECx, BIASSECx, DATASECx where x=A,B,C,D DATE-OBS will generate a non-fatal warning if missing

desispec.io.sky

IO routines for sky.

desispec.io.sky.read_sky(filename)[source]

Read sky model and return SkyModel object with attributes wave, flux, ivar, mask, header.

skymodel.wave is 1D common wavelength grid, the others are 2D[nspec, nwave]

desispec.io.sky.write_sky(outfile, skymodel, header=None)[source]

Write sky model.

Parameters:
  • outfile – filename or (night, expid, camera) tuple

  • skymodel – SkyModel object, with the following attributes

  • wave – 1D wavelength in vacuum Angstroms

  • flux – 2D[nspec, nwave] sky flux

  • ivar – 2D inverse variance of sky flux

  • mask – 2D mask for sky flux

  • stat_ivar – 2D inverse variance of sky flux (statistical only)

  • dwavecoeff – 1D[ncoeff] array of PCA dwavelength coefficients (optional)

  • dlsfcoeff – 1D[ncoeff] array of PCA dlsf coefficients (optional)

  • header – optional fits header data (fits.Header, dict, or list)

desispec.io.skycorr

IO routines for sky.

desispec.io.skycorr.read_skycorr(filename)[source]

Read sky correction and return SkyCorr object with attributes wave, flux, ivar, mask, header.

skymodel.wave is 1D common wavelength grid, the others are 2D[nspec, nwave]

desispec.io.skycorr.read_skycorr_pca(filename)[source]

Read sky correction pca file and return SkyCorrPCA object.

desispec.io.skycorr.write_skycorr(outfile, skycorr)[source]

Write sky model.

Parameters:
  • outfile – filename or (night, expid, camera) tuple

  • skycorr – SkyCorr object

desispec.io.skycorr.write_skycorr_pca(outfile, skycorrpca)[source]

Write sky model.

Parameters:
  • outfile – filename or (night, expid, camera) tuple

  • skycorrpca – SkyCorrPCA object

desispec.io.skygradpca

IO routines for sky gradient pca.

desispec.io.skygradpca.read_skygradpca(filename)[source]

Read sky grad pca file and return SkyGradPCA object.

desispec.io.skygradpca.write_skygradpca(outfile, skygradpca)[source]

Write sky model.

Parameters:
  • outfile – filename or (night, expid, camera) tuple

  • skygradpca – SkyGradPCA object

desispec.io.spectra

I/O routines for working with spectral grouping files.

desispec.io.spectra._read_image(hdus, extname, dtype, rows=None)[source]

Helper function to read extname from fitsio.FITS hdus, filter by rows, convert to native endian, and cast to dtype. Returns image.

desispec.io.spectra.read_frame_as_spectra(filename, night=None, expid=None, band=None, single=False)[source]

Read a FITS file containing a Frame and return a Spectra.

A Frame file is very close to a Spectra object (by design), and only differs by missing the NIGHT and EXPID in the fibermap, as well as containing only one band of data.

Parameters:

infile (str) – path to read

Options:

night (int): the night value to use for all rows of the fibermap. expid (int): the expid value to use for all rows of the fibermap. band (str): the name of this band. single (bool): if True, keep spectra as single precision in memory.

Returns (Spectra):

The object containing the data read from disk.

desispec.io.spectra.read_spectra(infile, single=False, targetids=None, rows=None, skip_hdus=None, select_columns={'EXP_FIBERMAP': None, 'EXTRA_CATALOG': None, 'FIBERMAP': None, 'SCORES': None})[source]

Read Spectra object from FITS file.

This reads data written by the write_spectra function. A new Spectra object is instantiated and returned.

Parameters:
  • infile (str) – path to read

  • single (bool) – if True, keep spectra as single precision in memory.

  • targetids (list) – Optional, list of targetids to read from file, if present.

  • rows (list) – Optional, list of rows to read from file

  • skip_hdus (list) – Optional, list/set/tuple of HDUs to skip

  • select_columns (dict) – Optional, dictionary to select column names to be read. Default, all columns are read.

Returns (Spectra):

The object containing the data read from disk.

skip_hdus options are FIBERMAP, EXP_FIBERMAP, SCORES, EXTRA_CATALOG, MASK, RESOLUTION; where MASK and RESOLUTION mean to skip those for all cameras. Note that WAVE, FLUX, and IVAR are always required.

If a table HDU is not listed in select_columns, all of its columns will be read

User can optionally specify targetids OR rows, but not both

desispec.io.spectra.read_tile_spectra(tileid, night=None, specprod=None, reduxdir=None, coadd=False, single=False, targets=None, fibers=None, redrock=True, group='cumulative')[source]

Read and return combined spectra for a tile/night

Parameters:

tileid (int) – Tile ID

Options:

night (int or str) : YEARMMDD night specprod (str) : overrides $SPECPROD reduxdir (str) : overrides $DESI_SPECTRO_REDUX/$SPECPROD coadd (bool) : if True, read coadds instead of per-exp spectra single (bool) : if True, use float32 instead of double precision targets (array-like) : filter by TARGETID fibers (array-like) : filter by FIBER redrock (bool) : if True, also return row-matched redrock redshift catalog group (str) : reads spectra in group (pernight, cumulative, …)

Returns: spectra or (spectra, redrock)

combined Spectra obj for all matching targets/fibers filter row-matched redrock catalog (if redrock=True)

Raises:

ValueError if no files or matching spectra are found

Note: the returned spectra are not necessarily in the same order as the targets or fibers input filters

desispec.io.spectra.write_spectra(outfile, spec, units=None)[source]

Write Spectra object to FITS file.

This places the metadata into the header of the (empty) primary HDU. The first extension contains the fibermap, and then HDUs are created for the different data arrays for each band.

Floating point data is converted to 32 bits before writing.

Parameters:
  • outfile (str) – path to write

  • spec (Spectra) – the object containing the data

  • units (str) – optional string to use for the BUNIT key of the flux HDUs for each band.

Returns:

The absolute path to the file that was written.

desispec.io.table

Utility functions for reading FITS tables

desispec.io.table.read_table(filename, ext=None)[source]

Reads a FITS table into an astropy Table, avoiding masked columns

Parameters:

filename (str) – full path to input fits file

Options:

ext (str or int): EXTNAME or extension number to read

Context: astropy 5.0 Table.read converts NaN and blank strings to masked values, which is a pain. This function reads the file with fitsio and then converts to a Table.

desispec.io.tpcorrparam

desispec.io.util

Utility functions for desispec IO.

desispec.io.util._dict2ndarray(data, columns=None)[source]

Convert a dictionary of ndarrays into a structured ndarray

Also works if DATA is an AstroPy Table.

Parameters:
  • data – input dictionary, each value is an ndarray

  • columns – optional list of column names

Returns:

structured numpy.ndarray with named columns from input data dictionary

Notes

data[key].shape[0] must be the same for every key every entry in columns must be a key of data

Example

d = dict(x=np.arange(10), y=np.arange(10)/2) nddata = _dict2ndarray(d, columns=[‘x’, ‘y’])

desispec.io.util._supports_memmap(filename)[source]

Returns True if the filesystem containing filename supports opening memory-mapped files in update mode.

desispec.io.util.add_columns(data, colnames, colvals)[source]

Adds extra columns to a data table

Parameters:
  • data – astropy Table or numpy structured array

  • colnames – list of new column names to add

  • colvals – list of column values to add; each element can be scalar or vector

Returns:

new table with extra columns added

Example

fibermap = add_columns(fibermap,

[‘NIGHT’, ‘EXPID’], [20102020, np.arange(len(fibermap))])

Notes

This is similar to numpy.lib.recfunctions.append_fields, but it also accepts astropy Tables as the data input, and accepts scalar values to expand as entries in colvals.

desispec.io.util.addkeys(hdr1, hdr2, skipkeys=None)[source]

Add new header keys from hdr2 to hdr1, skipping skipkeys

Parameters:
  • hdr1 (dict-like) – destination header for keywords

  • hdr2 (dict-like) – source header for keywords

Modifies hdr1 in place

desispec.io.util.backup_filename(filename)[source]

rename filename to next available filename.N

Parameters:

filename (str) – full path to filename

Returns:

New filename.N, or filename if original file didn’t already exist

desispec.io.util.camword_intersection(camwords, full_spectros_only=False)[source]

Return the camword intersection of cameras in a list of camwords

Parameters:

camwords – list of str camwords

Options:

full_spectros_only: if True, only include spectrographs that have all 3 brz cameras

Returns:

intersection of input camwords

Return type:

final_camword (str)

desispec.io.util.camword_to_spectros(camword, full_spectros_only=False)[source]

Takes a camword as input and returns any spectrograph represented within that camword. By default this includes partial spectrographs (with one or two cameras represented). But if full_spectros_only is set to True, only spectrographs with all cameras represented are given.

Parameters:
  • camword

  • cameras. (str. The camword of all) –

  • full_spectros_only – in the camword (the default) or if you only want fully populated spectrographs.

  • existing (bool. Default is False. Flag to specify if you want all spectrographs with any cameras) – in the camword (the default) or if you only want fully populated spectrographs.

Returns:

spectros, list. A list of integer spectrograph numbers represented in the camword input.

desispec.io.util.camword_union(camwords, full_spectros_only=False)[source]

Returns the union of a list of camwords. Optionally can return only those spectros with complete b, r, and z cameras. Note this intentionally does the union before truncating spectrographs, so two partial camwords can lead to an entire spectrograph,

e.g. [a0b1z1, a3r1z2] -> [a013z2] if full_spectros_only=False

[a0b1z1, a3r1z2] -> [a013] if full_spectros_only=True

even through no camword has a complete set of camera 1, a complete set is represented in the union.

Parameters:
  • camwords

  • camwords. (list or array of strings. List of) –

  • full_spectros_only – b, r, and z cameras in the funal union should be returned.

  • with (bool. True if only complete spectrographs) – b, r, and z cameras in the funal union should be returned.

Returns:

final_camword, str. The final union of all input camwords, where

truncation of incomplete spectrographs may or may not be performed based on full_spectros_only.

desispec.io.util.checkgzip(filename)[source]

Check for existence of filename, with or without .gz extension

Parameters:

filename (str) – filename to check for

Returns path of existing file without or without .gz, or raises FileNotFoundError if neither exists

desispec.io.util.columns_to_goodcamword(camword, badcamword, badamps=None, obstype=None, suppress_logging=False)[source]

Takes a camera information and obstype and returns the correct camword of good cameras for that type of observation.

Parameters:
  • camword (type str) – camword of available cameras

  • badcamword (str) – camword of bad cameras

  • badamps (str) – comma seperated list of bad amplifiers [brz][0-9][A-D]

  • obstype (str) –

  • suppress_logging (bool) –

Returns:

Camword for that observation given the obstype and

input camera information.

Return type:

goodcamword (str)

desispec.io.util.create_camword(cameras)[source]

Function that takes in a list of cameras and creates a succinct listing of all spectrographs in the list with cameras. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.

Parameters:

cameras (1-d array or list) – iterable containing strings of cameras, e.g. ‘b0’,’r1’,…

Returns (str):

A string representing all information about the spectrographs/cameras given in the input iterable, e.g. a01234678b59z9

desispec.io.util.decode_camword(camword)[source]

Function that takes in a succinct listing of all spectrographs and outputs a 1-d numpy array with a list of all spectrograph/camera pairs. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.

Parameters:

camword (str) – A string representing all information about the spectrographs/cameras e.g. a01234678b59z9

Returns (np.ndarray, 1d): an array containing strings of

cameras, e.g. ‘b0’,’r1’,…

desispec.io.util.difference_camwords(fullcamword, badcamword, suppress_logging=False)[source]

Returns the difference of two camwords. The second argument cameras are removed from the first argument and the remainer is returned. Smart enough to ignore bad cameras if they don’t exist in full camword list.

Parameters:
  • fullcamword (including the bad ones to be removed) –

  • cameras (str. The camword of all) –

  • badcamword

  • output (str. A camword defining the bad cameras you don't want to include in the final camword that is) –

Returns:

str. A camword of cameras in fullcamword that are not in badcamword.

desispec.io.util.erow_to_goodcamword(erow, suppress_logging=False)[source]

Takes a dictionary-like object with at minimum keys for OBSTYPE (type str), CAMWORD (type str), BADCAMWORD (type str), BADAMPS (str).

Parameters:
  • erow (dict-like) – the exposure table row with columns OBSTYPE, CAMWORD, BADCAMWORD, and BADAMPS.

  • suppress_logging (bool) –

Returns:

Camword for that observation given the obstype and

input camera information.

Return type:

goodcamword (str)

desispec.io.util.fitsheader(header)[source]

Convert header into astropy.io.fits.Header object.

header can be:
  • None: return a blank Header

  • list of (key, value) or (key, (value,comment)) entries

  • dict d[key] -> value or (value, comment)

  • Header: just return it unchanged

Returns fits.Header object

desispec.io.util.get_log_pathname(data_pathname)[source]

Returns unique log pathname for a given data product.

Parameters:

data_pathname (str) – input filename of output data

Returns:

pathname to the log

Return type:

log_pathname (str)

Returns unique filename in same directory

desispec.io.util.get_speclog(nights, rawdir=None)[source]

Scans raw data headers to return speclog of observations. Slow.

Parameters:

nights – list of YEARMMDD nights to scan

Options:

rawdir (str): overrides $DESI_SPECTRO_DATA

Returns:

Table with columns NIGHT,EXPID,MJD,FLAVOR,OBSTYPE,EXPTIME

Scans headers of rawdir/NIGHT/EXPID/desi-EXPID.fits.fz

desispec.io.util.get_tempfilename(filename)[source]

Returns unique tempfile in same directory as filename with same extension

Parameters:

filename (str) – input filename

Returns unique filename in same directory

Example intended usage:

tmpfile = get_tempfile(filename)
table.write(tmpfile)
os.rename(tmpfile, filename)

By keeping the same extension as the input file, this preserves the ability of table.write to derive the format to use, and if something goes wrong with the I/O it doesn’t leave a corrupted partially written file with the final name. The tempfile includes the PID to provide some race condition protection (the last one do do os.rename wins, but at least different processes won’t corrupt each other’s files).

desispec.io.util.header2wave(header)[source]

Converts header keywords into a wavelength grid.

returns wave = CRVAL1 + range(NAXIS1)*CDELT1

if LOGLAM keyword is present and true/non-zero, returns 10**wave

desispec.io.util.healpix_subdirectory(nside, pixel)[source]

Return a fixed directory path for healpix grouped files.

Given an NSIDE and NESTED pixel index, return a directory named after a degraded NSIDE and pixel followed by the original nside and pixel. This function is just to ensure that this subdirectory path is always created by the same code.

Parameters:
  • nside (int) – a valid NSIDE value.

  • pixel (int) – the NESTED pixel index.

Returns (str):

a path containing the low and high resolution directories.

desispec.io.util.is_svn_current(dirname)[source]

Return True/False for if svn checkout dirname is up-to-date with server

Raises ValueError if unable to determine (e.g. dirname isn’t svn checkout)

desispec.io.util.iterfiles(root, prefix, suffix=None)[source]

Returns iterator over files starting with prefix found under root dir Optionally also check if filename ends with suffix

desispec.io.util.makepath(outfile, filetype=None)[source]

Create path to outfile if needed.

If outfile isn’t a string and filetype is set, interpret outfile as a tuple of parameters to locate the file via findfile().

Returns /path/to/outfile

TODO: maybe a different name?

desispec.io.util.native_endian(data)[source]

Convert numpy array data to native endianness if needed.

Returns new array if endianness is swapped, otherwise returns input data

Context: By default, FITS data from astropy.io.fits.getdata() are not Intel native endianness and scipy 0.14 sparse matrices have a bug with non-native endian data.

desispec.io.util.parse_badamps(badamps, joinsymb=',')[source]

Parses badamps string from an exposure or processing table into the (camera,petal,amplifier) sets, with appropriate checking of those values to make sure they’re valid. Returns empty list if badamps is None.

Parameters:
  • badamps – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.

  • (comma (str. A string of {camera}{petal}{amp} entries separated by symbol given with joinsymb) – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.

  • joinsymb

  • badamps. (str. The symbol separating entries in the str list given by) –

Returns:

cam_petal_amps, list. A list where each entry is a length 3 tuple of (camera,petal,amplifier).

Camera is a lowercase string in [b, r, z]. Petal is an int from 0 to 9. Amplifier is an upper case string in [A, B, C, D].

desispec.io.util.parse_cameras(cameras, loglevel='INFO')[source]

Function that takes in a representation of all spectrographs and outputs a string that succinctly lists all spectrograph/camera pairs. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.

Parameters:
  • cameras – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().

  • array (str. 1-d) – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().

  • list – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().

Options:

loglevel, str: use e.g. “WARNING” to avoid INFO-level log messages for just this call

Returns (str):
camword, str. A string representing all information about the spectrographs/cameras

given in the input iterable, e.g. a01234678b59z9

Create a relative symlink from dst -> src, while also handling $DESI_ROOT vs $DESI_ROOT_READONLY

Parameters:
  • src (str) – the pre-existing file to point to

  • dst (str) – the symlink file to create

Options:

pathonly (bool): return the path, but don’t make the link

Returns:

the releative path from dst -> src

desispec.io.util.replace_prefix(filepath, oldprefix, newprefix)[source]

Replace filename prefix even if prefix is elsewhere in path or filename

Parameters:
  • filepath – filename, optionally including path

  • oldprefix – original prefix to filename part of filepath

  • newprefix – new prefix to use for filename

Returns:

new filepath with replaced prefix

e.g. replace_prefix(‘/blat/foo/blat-bar-blat.fits’, ‘blat’, ‘quat’) returns ‘/blat/foo/quat-bar-blat.fits’

desispec.io.util.spectros_to_camword(spectros)[source]

Converts a specification of spectrographs to the equivalent camword

Parameters:

spectros – str (e.g. “0-7,9”) or list of ints spectrographs

Returns: camword str (e.g. a012345679)

desispec.io.util.validate_badamps(badamps, joinsymb=',')[source]

Checks (and transforms) badamps string for consistency with the for need in an exposure or processing table for use in the Pipeline. Specifically ensure they come in (camera,petal,amplifier) sets, with appropriate checking of those values to make sure they’re valid. Returns the input string except removing whitespace and replacing potential character separaters with joinsymb (default ‘,’). Returns None if None is given.

Parameters:
  • badamps – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.

  • (comma (str. A string of {camera}{petal}{amp} entries separated by symbol given with joinsymb) – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.

  • joinsymb

  • badamps. (str. The symbol separating entries in the str list given by) –

Returns:

newbadamps, str. Input badamps string of {camera}{petal}{amp} entries separated by symbol given with

joinsymb (comma by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’. Differs from input in that other symbols used to separate terms are replaced by joinsymb and whitespace is removed.

desispec.io.util.write_bintable(filename, data, header=None, comments=None, units=None, extname=None, clobber=False, primary_extname='PRIMARY')[source]

Utility function to write a fits binary table complete with comments and units in the FITS header too. DATA can either be dictionary, an Astropy Table, a numpy.recarray or a numpy.ndarray.

Parameters:
  • filename – full path to filename to write

  • data – dict or table-like data to write

Options:

header: dict-like header key/value pairs to propagate comments (dict): comments[COLNAME] per column units (dict): units[COLNAME] per column extname (str): extension name for EXTNAME header clobber (bool): if True, overwrite pre-existing file primary_extname (str): EXTNAME to use for primary HDU=0

desispec.io.xytraceset

I/O routines for XYTraceSet objects.

desispec.io.xytraceset.read_xytraceset(filename)[source]

Reads traces in PSF fits file

Parameters:

filename – Path to input fits file which has to contain XTRACE and YTRACE HDUs

Returns:

XYTraceSet object

desispec.io.xytraceset.write_xytraceset(outfile, xytraceset)[source]

Write a traceset fits file and returns path to file written.

Parameters:
  • outfile – full path to output file

  • xytraceset – desispec.xytraceset.XYTraceSet object

Returns:

full filepath of output file that was written

desispec.joincosmics

Utility functions to join detected cosmic rays

class desispec.joincosmics.LinearSelectionElement(n, m, angle)[source]

Define a selection element for morphological binary image processing.

plot()[source]

Return a plot of the selection element (a bitmap).

Returns:

fig – Figure object for plotting/saving.

Return type:

matplotlib.Figure

desispec.joincosmics.downsample_image(image, n)[source]

Downsample 2D input image n x n.

Parameters:
  • image (ndarray) – 2D input image.

  • n (int) – Downsampling factor, applied to both image dimensions.

Returns:

result – Resampled image with shape = image.shape//n.

Return type:

ndarray

desispec.linalg

Some linear algebra functions.

desispec.linalg.cholesky_invert(A)[source]

returns the inverse of a positive definite matrix

Args :

A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray)

Returns:

2D positive definite matrix, inverse of A (numpy.ndarray)

Return type:

cov

desispec.linalg.cholesky_solve(A, B, overwrite=False, lower=False)[source]

Returns the solution X of the linear system A.X=B assuming A is a positive definite matrix

Args :

A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray) B : 1D vector, must have dimension n (numpy.ndarray)

Options :

overwrite: replace A data by cholesky decomposition (faster) lower: cholesky decomposition triangular matrix is lower instead of upper

Returns :

X : 1D vector, same dimension as B (numpy.ndarray)

desispec.linalg.cholesky_solve_and_invert(A, B, overwrite=False, lower=False)[source]

returns the solution X of the linear system A.X=B assuming A is a positive definite matrix

Args :

A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray) B : 1D vector, must have dimension n (numpy.ndarray)

Options :

overwrite: replace A data by cholesky decomposition (faster) lower: cholesky decomposition triangular matrix is lower instead of upper

Returns:

X,cov, where X : 1D vector, same dimension n as B (numpy.ndarray) cov : 2D positive definite matrix, inverse of A (numpy.ndarray)

desispec.linalg.spline_fit(output_wave, input_wave, input_flux, required_resolution, input_ivar=None, order=3, max_resolution=None)[source]

Performs spline fit of input_flux vs. input_wave and resamples at output_wave

Parameters:
  • output_wave – 1D array of output wavelength samples

  • input_wave – 1D array of input wavelengths

  • input_flux – 1D array of input flux density

  • required_resolution (float) – resolution for spline knot placement (same unit as wavelength)

Options:

input_ivar : 1D array of weights for input_flux order (int) : spline order max_resolution (float) : if not None and first fit fails, try once this resolution

Returns:

1D array of flux sampled at output_wave

Return type:

output_flux

desispec.log

This is a transitional dummy wrapper on desiutil.log.

desispec.log.get_logger(*args, **kwargs)[source]

Transitional dummy wrapper on desiutil.log.get_logger().

desispec.magnitude

Broadband flux and magnitudes.

desispec.magnitude.ab_flux_in_ergs_s_cm2_A(wave)[source]
Parameters:

wave – 1D numpy array (Angstrom)

Returns:

ab flux in units of ergs/s/cm2/A

desispec.magnitude.compute_ab_mag(spectrum_wave, spectrum_flux, transmission_wave, transmission_value)[source]

Computes ab mag

Parameters:
  • spectrum_wave – 1D numpy array (Angstrom)

  • spectrum_flux – 1D numpy array (in units of 1e-17 ergs/s/cm2/A), same size as spectrum_wave

  • transmission_wave – 1D numpy array (Angstrom)

  • transmission_value – 1D numpy array , dimensionless, same size as transmission_wave

Returns:

mag (float scalar)

desispec.magnitude.compute_broadband_flux(spectrum_wave, spectrum_flux, transmission_wave, transmission_value)[source]

Computes broadband flux

Parameters:
  • spectrum_wave – 1D numpy array (Angstrom)

  • spectrum_flux – 1D numpy array is some input density unit, same size as spectrum_wave

  • transmission_wave – 1D numpy array (Angstrom)

  • transmission_value – 1D numpy array , dimensionless, same size as transmission_wave

Returns:

integrated flux (unit= A x (input density unit)) , scalar

desispec.maskbits

Mask bits for the spectro pipeline.

Stephen Bailey, LBNL, January 2015

Example:

from desispec.maskbits import ccdmask

#- bit operations
mask |= ccdmask.COSMIC     #- set ccdmask.COSMIC in integer/array `mask`
mask & ccdmask.COSMIC      #- get ccdmask.COSMIC from integer/array `mask`
(mask & ccdmask.COSMIC) != 0  #- test boolean status of ccdmask.COSMIC in integer/array `mask`
ccdmask.COSMIC | specmask.SATURATED  #- Combine two bitmasks.

#- bit attributes
ccdmask.mask('COSMIC')     #- 2**0, same as ccdmask.COSMIC
ccdmask.mask(0)            #- 2**0, same as ccdmask.COSMIC
ccdmask.COSMIC             #- 2**0, same as ccdmask.mask('COSMIC')
ccdmask.bitnum('COSMIC')   #- 0
ccdmask.bitname(0)         #- 'COSMIC'
ccdmask.names()            #- ['COSMIC', 'HOT', 'DEAD', 'SATURATED', ...]
ccdmask.names(3)           #- ['COSMIC', 'HOT']
ccdmask.comment(0)         #- "Cosmic ray"
ccdmask.comment('BADPIX')  #- "Cosmic ray"

desispec.maskedmedian

Utility function to perform a median of images with masks.

desispec.maskedmedian.masked_median(images, masks=None)[source]

Perfomes a median of an list of input images. If a list of mask is provided, the median is performed only on unmasked pixels.

Parameters:

images – 3D numpy array : list of images of same shape

Options:

masks : list of mask images of same shape as the images. Only pixels with mask==0 are considered in the median.

Returns : median image

desispec.mgii_afterburner

Authors: Edmond CHAUSSIDON (CEA Saclay), Corentin RAVOUX (CEA Saclay) Contact: edmond.chaussidon@cea.fr, corentin.ravoux@cea.fr

Note: Please see https://desi.lbl.gov/trac/attachment/wiki/TargetSelectionWG/TSTeleconMinutes/DESITS210113/QSO_completeness_Ravoux_13_01_2021_v2.pdf for details about the MgII fitter

the “optimal”/chosen parameters are (can be modify when desi_qso_mgii_afterburner is called): lambda_width = 250 max_sigma = 200 min_sigma = 10 min_deltachi2 = 16 min_signifiance_A = 3 min_A = 0

desispec.mgii_afterburner.create_mask_fit(fit_results, max_sigma=None, min_sigma=None, min_deltachi2=None, min_A=None, min_signifiance_A=None)[source]

Create a mask based on the results of the MgII fit and some parameters which constraints these parameters

Parameters:
  • fit_results (numpy array) – Array containing the parameters of the fit

  • max_sigma (float) – Maximal value for the standard deviation of the fitted Gaussian peak

  • min_sigma (float) – Minimal value for the standard deviation of the fitted Gaussian peak

  • min_deltachi2 (float) – Minimal value for the difference of chi2 between redrock fit and MgII fitter over the lambda_width interval considered

  • min_A (float) – Minimal value for the amplitude of the fitted Gaussian peak

  • min_signifiance_A (float) – Minimal signifiance for the amplitude of the fitted Gaussian peak. The signifiance is here define as the ratio between peak amplitude and error on the peak amplitude.

Returns:

mask with the same length than

fit_results which indicates the objects validating parameter constraints

Return type:

mask (boolean numpy array)

desispec.mgii_afterburner.create_model(spectra, redshifts, archetype_fit=False, archetypes_dir=None, template_dir=None)[source]

< COPY from prospect.plotframes to avoid to load prospect in desispec >

Returns model_wave[nwave], model_flux[nspec, nwave], row matched to redshifts, which can be in a different order than spectra. - redshifts must be entry-matched to spectra.

desispec.mgii_afterburner.fit_mgii_line(target_id, redshift_redrock, flux, ivar_flux, model_flux, wavelength, lambda_width, add_linear_term=False, gaussian_smoothing_fit=None, mask_mgii=None)[source]

Fitting routine. Fit a Gaussian peak on preselected spectra and return the main parameters of the fit including parameter errors.

Parameters:
  • target_id (numpy array) – Array containing target id of the the object to fit

  • redshift_redrock (numpy array) – Array containing the redshift of the the object to fit

  • flux (numpy array) – Array containing the full flux arrays of every object to fit

  • ivar_flux (numpy array) – Array containing the inverse variance arrays of every object to fit

  • model_flux (numpy array) – Array containing the best fit redrock model for every object to fit

  • wavelength (numpy array) – Array containing the wavelength

  • lambda_width (float) – The width in wavelength (in Angstrom) considered for the fitting arount the MgII peak

  • add_linear_term (boolean) – Add a linear term to the Gaussian peak term to fit the continuum.

  • gaussian_smoothing_fit (float) – If not None, the spectra is smoothed by the given value before the fit

  • mask_mgii (float) – If not None, mask a region of near the MgII peak with the given witdh to fit double MgII peak (in progress)

Returns:

Array containing the parameters of the fit

Return type:

fit_results (numpy array)

desispec.mgii_afterburner.get_spectra(spectra_name, redrock_name, lambda_width, index_to_fit, template_dir=None, archetypes_dir=None)[source]

Get the spectra and the best fit model from a given spectra and redrock file.

Parameters:
  • spectra_name (str) – The name of the spectra file.

  • redrock_name (str) – The name of the redrock file associated to the spectra file.

  • lambda_width (float) – The width in wavelength (in Angstrom) considered for the fitting arount the MgII peak

  • index_to_fit (boolean numpy array) – boolean array of size 500 specifing which spectra have to be used

  • add_linear_term (boolean) – Add a linear term to the Gaussian peak term to fit the continuum.

  • template_dir (str) – If the redrock template variable is not loaded by the desi environment, specify the template path

  • archetypes_dir (str) – If not None, use the archetypes templates in the path specified

Returns:

Array containing target id of the the object to fit

redshift_redrock (numpy array): Array containing the redshift of the the object to fit

flux (numpy array): Array containing the full flux arrays of every object to fit

ivar_flux (numpy array): Array containing the inverse variance arrays of every object to fit

model_flux (numpy array): Array containing the best fit redrock model for every object to fit

wavelength (numpy array): Array containing the wavelength

index_with_fit (boolean numpy array): boolean array of index_to_fit size masking index where mgII fitter is not apply

Return type:

target_id (numpy array)

desispec.mgii_afterburner.load_redrock_templates(template_dir=None)[source]

< COPY from prospect.plotframes to avoid to load prospect in desispec >

Load redrock templates; redirect stdout because redrock is chatty

desispec.mgii_afterburner.mgii_fitter(spectra_name, redrock_name, index_to_fit, lambda_width, add_linear_term=False, gaussian_smoothing_fit=None, template_dir=None, archetypes_dir=None, max_sigma=None, min_sigma=None, min_deltachi2=None, min_A=None, min_signifiance_A=None)[source]

MgII fitter afterburner main function. For a given spectra file and its associated redrock file (redrock output), returns a numpy mask which indicates the objects fitted by redrock as galaxies and considered as QSO by the MgII fitter. The mask is determined following input parameters.

Parameters:
  • spectra_name (str) – The name of the spectra file.

  • redrock_name (str) – The name of the redrock file associated to the spectra file.

  • lambda_width (float) – The width in wavelength (in Angstrom) considered for the fitting arount the MgII peak

  • index_to_fit (boolean numpy array) – boolean array of size 500 specifing which spectra have to be used

  • add_linear_term (boolean) – Add a linear term to the Gaussian peak term to fit the continuum.

  • template_dir (str) – If the redrock template variable is not loaded by the desi environment, specify the template path

  • archetypes_dir (str) – If not None, use the archetypes templates in the path specified

  • max_sigma (float) – Maximal value for the standard deviation of the fitted Gaussian peak

  • min_sigma (float) – Minimal value for the standard deviation of the fitted Gaussian peak

  • min_deltachi2 (float) – Minimal value for the difference of chi2 between redrock fit and MgII fitter over the lambda_width interval considered

  • min_A (float) – Minimal value for the amplitude of the fitted Gaussian peak

  • min_signifiance_A (float) – Minimal signifiance for the amplitude of the fitted Gaussian peak. The signifiance is here define as the ratio between peak amplitude and error on the peak amplitude.

Returns:

mask with the same length than index_with_fit which indicates object considered by MgII fitter as QSO

fit_results (numpy array): Array containing the parameters of the fit to save them

index_with_fit (boolean numpy array): boolean array of index_to_fit size masking index where mgII fitter is not apply

Return type:

mask_fit (boolean numpy array)

desispec.night_qa

desispec.night_qa._javastring()[source]

Return a string that embeds a date in a webpage.

Note

Credits to ADM (desitarget/QA.py).

desispec.night_qa._read_ctedet(night, prod, ctedet_expid, petal, camera)[source]

Internal function used by create_ctedet_pdf(), reading the ctedet_expid preproc info.

Parameters:
  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • ctedet_expid – EXPID for the CTE diagnosis (1s FLAT, or darker science exposure) (int)

  • petal – 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 (int)

  • camera – “b”, “r”, or “z” (string)

Returns:

a dictionary with the IMAGE data, plus various infos, assuming a preproc file is available, otherwise None.

Return type:

mydict

desispec.night_qa._read_dark(fn, night, prod, dark_expid, petal, camera, binning=4)[source]

Internal function used by create_dark_pdf(), to read and bin the 300s dark.

Parameters:
  • fn – full path to the dark image (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • dark_expid – EXPID of the 300s DARK exposure to display (int)

  • petal – 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 (int)

  • camera – “b”, “r”, or “z” (string)

  • binning (optional, defaults to 4) – binning of the image (which will be beforehand trimmed) (int)

Returns:

a dictionary with the binned+masked image, plus various infos, assuming a preproc file is available. Otherwise None.

Return type:

mydict

desispec.night_qa._read_sframesky(night, prod, expid)[source]

Internal function called by create_sframesky_pdf, which generates the per-expid sframesky plot.

Parameters:
  • outpng – output png file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • expid – expid to display (int)

Returns:

mydict, dictionary with per-camera wave, flux, and various infos, assuming sframe files were available, otherwise None.

desispec.night_qa.create_badcol_png(outpng, night, prod, n_previous_nights=10)[source]

For a given night, create a png file with displaying the number of bad columns per {camera}{petal}.

Parameters:
  • outpng – output png file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • n_previous_nights (optional, defaults to 10) – number of previous nights to plot (int)

desispec.night_qa.create_ctedet_pdf(outpdf, night, prod, ctedet_expid, nproc, nrow=21, xmin=None, xmax=None, ylim=(-5, 10), yoffcols={'A': -2.5, 'B': -3.5, 'C': -2.5, 'D': -3.5})[source]

For a given night, create a pdf with a CTE diagnosis (from preproc files).

Parameters:
  • outpdf – output pdf file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • ctedet_expid – EXPID for the CTE diagnosis (1s FLAT, or darker science exposure) (int)

  • nproc – number of processes running at the same time (int)

  • nrow (optional, defaults to 21) – number of rows to include in median (int)

  • xmin (optional, defaults to None) – minimum column to display (float)

  • xmax (optional, defaults to None) – maximum column to display (float)

  • ylim (optional, default to (-5, 10)) – ylim for the median plot (duplet)

  • (optional (yoffcols) – -2.5, “B” : -3.5, “C” : -2.5, “D” : -3.5}): y-values to report the per-amplifier OFFCOLS info, if any (dictionnary of floats)

  • {"A" (defaults to) – -2.5, “B” : -3.5, “C” : -2.5, “D” : -3.5}): y-values to report the per-amplifier OFFCOLS info, if any (dictionnary of floats)

Note

  • Credits to S. Bailey.

  • Copied-pasted-adapted from /global/homes/s/sjbailey/desi/dev/ccd/plot-amp-cte.py

desispec.night_qa.create_dark_pdf(outpdf, night, prod, dark_expid, nproc, binning=4, bkgsub_science_cameras=None, run_preproc=None)[source]

For a given night, create a pdf with the 300s binned dark.

Parameters:
  • outpdf – output pdf file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • dark_expid – EXPID of the 300s or 1200s DARK exposure to display (int)

  • nproc – number of processes running at the same time (int)

  • binning (optional, defaults to 4) – binning of the image (which will be beforehand trimmed) (int)

  • bkgsub_science_cameras (optional, defaults to None) – comma-separated list of the cameras to be additionally processed with the –bkgsub-for-science argument (e.g. “b”) (string)

  • run_preproc (optional, defaults to None) – if None, autoderive if preproc should be run, otherwise override if it should be run or not (None or bool)

Note

If the identified dark image is not processed and if the raw data is there, we do process it (in a temporary folder), so that we can generate the plots.

desispec.night_qa.create_mp4(fns, outmp4, duration=15)[source]

Create an animated .mp4 from a set of input files (usually pngs).

Parameters:
  • fns – list of input filenames, in the correct order (list of strings)

  • outmp4 – output .mp4 filename

  • duration (optional, defaults to 15) – video duration in seconds (float)

Note

  • Requires ffmpeg to be installed.

  • At NERSC, run in the bash command line: “module load ffmpeg”.

  • The movie uses fns in the provided order.

desispec.night_qa.create_petalnz_pdf(outpdf, night, prod, tileids, surveys, dchi2_threshold=25, group='cumulative', newlya_ecsv=None)[source]

For a given night, create a per-petal, per-tracer n(z) pdf file.

Parameters:
  • outpdf – output pdf file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • tileids – list of tileids to consider (list or numpy array)

  • surveys – list of the surveys for each tileid of tileids (list or numpy array)

  • dchi2_threshold (optional, defaults to 25) – DELTACHI2 value to split the sample (float)

  • group (optional, str) – tile group “cumulative” or “pernight”

  • newlya_ecsv (optional, defaults to None) – if set, table saving the per-tile number of newly identified Ly-a and the tile coverage (if no dark tiles, no file will be saved).

Note

  • Only displays:

    • sv1, sv2, sv3, main, as otherwise the plotted tracers are not relevant;

    • FAPRGRM=”bright” or “dark” tileids;

    • for the main survey, tiles with EFFTIME > MINTFRAC * GOALTIME.

  • If the tile-qa-TILEID-thruNIGHT.fits file is missing, that tileid is skipped.

  • For the Lya, work from the zmtl*fits files, trying to mimick what is done in desitarget.mtl.make_mtl().

  • The LRG, ELG, QSO, BGS_BRIGHT, BGS_FAINT bit are the same for sv1, sv2, sv3, main, so ok to simply use the bit mask values from the main.

  • TBD : we query the FAPRGRM of the tile-qa-*.fits header, not sure that properly works for surveys other than main..

desispec.night_qa.create_sframesky_pdf(outpdf, night, prod, expids, nproc)[source]

For a given night, create a pdf from per-expid sframe for the sky fibers only.

Parameters:
  • outpdf – output pdf file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • expids – expids to display (list or np.array)

  • nproc – number of processes running at the same time (int)

desispec.night_qa.create_skyzfiber_png(outpng, night, prod, tileids, dchi2_threshold=9, group='cumulative')[source]
For a given night, create a Z vs. FIBER plot for all SKY fibers, and one for

each of the main backup/bright/dark programs

Parameters:
  • outpdf – output pdf file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • tileids – list of tileids to consider (list or numpy array)

  • dchi2_threshold (optional, defaults to 9) – DELTACHI2 value to split the sample (float)

  • group (str, optional) – tile group “cumulative” or “pernight”

Note

Work from the redrock-*.fits files.

desispec.night_qa.create_tileqa_pdf(outpdf, night, prod, expids, tileids, group='cumulative')[source]

For a given night, create a pdf from the tile-qa*png files, sorted by increasing EXPID.

Parameters:
  • outpdf – output pdf file (string)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • expids – expids of the tiles to display (list or np.array)

  • tileids – tiles to display (list or np.array)

  • group (str, optional) – tile group “cumulative” or “pernight”

desispec.night_qa.get_ctedet_night_expid(night, prod)[source]

Returns the EXPID of the 1s FLAT exposure for a given night. If not present, takes the science exposure with the lowest sky counts.

Parameters:
  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

Returns:

EXPID (int)

Return type:

expid

Note

  • If nothing found, returns None.

  • We look for preproc files.

  • As we are looking for a faint signal, we want the image with the less electrons, thus it could be picking a BRIGHT short exposure against a longer DARK exposure.

desispec.night_qa.get_dark_night_expid(night, prod)[source]

Returns the EXPID of the 300s DARK exposure for a given night.

Parameters:
  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

Returns:

EXPID (int)

Return type:

expid

Note

  • If nothing found, returns None.

  • 20220110 : new method, relying on processing_tables

desispec.night_qa.get_morning_dark_night_expid(night, prod, exptime=1200)[source]

Returns the EXPID of the latest 1200s DARK exposure for a given night.

Parameters:
  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • exptime (optional, defaults to 1200) – exposure time we are looking for (float)

Returns:

EXPID (int)

Return type:

expid

Note

  • If nothing found, returns None.

  • As of now (20221220), those morning darks are not processed by the daily pipeline; hence we do not use the processing_tables, but the exposure_tables.

desispec.night_qa.get_nightqa_outfns(outdir, night)[source]

Utility function to get nightqa file names.

Parameters:
  • outdir – output folder name (string)

  • night – night (int)

Returns:

dictionary with file names

desispec.night_qa.get_surveys_night_expids(night, datadir=None)[source]

List the (EXPIDs, TILEIDs) from a given night for a given survey.

Parameters:
  • night – night (int)

  • surveys – comma-separated list of surveys to consider, in lower-cases, e.g. “sv1,sv2,sv3,main” (str)

  • datadir (optional, defaults to $DESI_SPECTRO_DATA) – full path where the {NIGHT}/desi-{EXPID}.fits.fz files are (str)

Returns:

list of the EXPIDs (np.array()) tileids: list of the TILEIDs (np.array()) surveys: list of the SURVEYs (np.array())

Return type:

expids

Note

Based on: - parsing the OBSTYPE keywords from the SPEC extension header of the desi-{EXPID}.fits.fz files; - for OBSTYPE=”SCIENCE”, parsing the fiberassign-TILEID.fits* header

desispec.night_qa.path_full2web(fn)[source]

Convert full path to web path (needs DESI_ROOT to be defined).

Parameters:

fn – filename full path

Returns:

Web path

desispec.night_qa.plot_newlya(ax, ntilecovs, nnewlyas, tileids=None, title=None, xlabel='Number of observed tiles', ylabel='(N_newlya_obs / N_newlya_expect) - 1', xlim=(0.5, 6.5), ylim=(-2.5, 2.5), nvalifiber_norm=3900)[source]
Plot the ratio of the number of newly identified Ly-a to the expected number,

vs. the tile observations coverage for a list of tiles.

Parameters:
  • ax – pyplot Axes object

  • ntilecovs – average number of previously observed tiles for each number of pass (np.array(len(tiles), n_dark_passids))

  • nnewlyas – list of nb of newly identified Ly-a (with a valid fiber), normalized to nvalifiber_norm (list or np.array() of ints)

  • tileids (optional, defaults to None) – tileids (to report outliers) (list or np.array() of ints)

  • title (optional, defaults to None) – plot title (str)

  • xlabel (optional, defaults to "Number of observed tiles") – plot xlabel (str)

  • ylabel (optional, defaults to "Number of newly identified LYA candidates") – plot ylabel(str)

  • xlim (optional, defaults to (0.5, 6.5)) – plot xlim (tuple)

  • ylim (optional, defaults to (-2.5, 2.5) – plot ylim (tuple)

  • nvalifiber_norm (optional, defaults to 3900) – number of valid fibers to normalize to, for the expected regions (int)

Note

  • ntilecovs[:, 0] lists the fractions of the tiles covered by 1 tile, etc.

  • The plotted y-values are: (N_newlya_observed / N_newlya_expected) - 1.

  • The expected numbers are based on all main dark tiles (from daily) up to May 26th 2022.

  • The 1-2-3-sigma regions reflect the approximate scatter of those data.

desispec.night_qa.write_html_collapse_script(html, classname)[source]

Write the required lines to have the collapsing sections working.

Parameters:
  • html – an html file object.

  • classname – “collapsible” only for now (string)

desispec.night_qa.write_html_today(html)[source]

Write in an html object today’s date.

Parameters:

html – html file object.

desispec.night_qa.write_nightqa_html(outfns, night, prod, css, expids, tileids, surveys)[source]

Write the nightqa-{NIGHT}.html page.

Parameters:
  • outfns – dictionary with filenames generated by desi_night_qa (output from get_nightqa_outfns)

  • night – night (int)

  • prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)

  • css – path to the nightqa.css file

  • expids – EXPIDs for the considered science exposures (np.array() of ints)

  • tileids – TILEIDs for the considered science exposures (np.array() of ints)

  • surveys – SURVEYs for the considered science exposures (np.array() of strings)

Notes

expids, tileids, surveys: fiducial use is that those are the outputs of get_surveys_night_expids()

desispec.parallel

Helper functions and classes for dealing with parallelization and related topics.

desispec.parallel.default_nproc = 1

Default number of multiprocessing processes. Set globally on first import.

desispec.parallel.dist_balanced(nwork, maxworkers)[source]

Distribute items between a flexible number of workers.

This assumes that each item has equal weight, and that they should be divided into contiguous blocks of items and assigned to workers in rank order.

If the number of workers is less than roughly sqrt(nwork), then we do not reduce the number of workers and the result is the same as the dist_uniform function. If there are more workers than this, then the number of workers is reduced until all workers have close to the same number of tasks.

Parameters:
  • nwork (int) – The number of work items.

  • maxworkers (int) – The maximum number of workers. The actual number may be less than this.

Returns:

A list of tuples, one for each worker. The first element of the tuple is the first item assigned to the worker, and the second element is the number of items assigned to the worker.

desispec.parallel.dist_discrete(worksizes, nworkers, workerid, power=1.0)[source]

Distribute indivisible blocks of items between groups.

Given some contiguous blocks of items which cannot be subdivided, distribute these blocks to the specified number of groups in a way which minimizes the maximum total items given to any group. Optionally weight the blocks by a power of their size when computing the distribution.

This is effectively the “Painter”s Partition Problem”.

Parameters:
  • worksizes (list) – The sizes of the indivisible blocks.

  • nworkers (int) – The number of workers.

  • workerid (int) – The worker ID whose range should be returned.

  • power (float) – The power to use for weighting

Returns:

A tuple. The first element of the tuple is the first block assigned to the worker ID, and the second element is the number of blocks assigned to the worker.

desispec.parallel.dist_discrete_all(worksizes, nworkers, power=1.0)[source]

Distribute indivisible blocks of items between groups.

Given some contiguous blocks of items which cannot be subdivided, distribute these blocks to the specified number of groups in a way which minimizes the maximum total items given to any group. Optionally weight the blocks by a power of their size when computing the distribution.

This is effectively the “Painter”s Partition Problem”.

Parameters:
  • worksizes (list) – The sizes of the indivisible blocks.

  • nworkers (int) – The number of workers.

  • pow (float) – The power to use for weighting

Returns:

list of length nworkers; each element is a list of indices of

worksizes assigned to that worker.

desispec.parallel.dist_uniform(nwork, nworkers, id=None)[source]

Statically distribute some number of items among workers.

This assumes that each item has equal weight, and that they should be divided into contiguous blocks of items and assigned to workers in rank order.

This function returns the index of the first item and the number of items for the specified worker ID, or the information for all workers.

Parameters:
  • nwork (int) – the number of things to distribute.

  • nworkers (int) – the number of workers.

  • id (int) – optionally return just the tuple associated with this worker

Returns (tuple):

A tuple of ints, containing the first item and number of items. If id=None, then return a list containing the tuple for all workers.

desispec.parallel.stdouterr_redirected(to=None, comm=None, overwrite=False)[source]

Redirect stdout and stderr to a file.

The general technique is based on:

http://stackoverflow.com/questions/5081657 http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/

One difference here is that each process in the communicator redirects to a different temporary file, and the upon exit from the context the rank zero process concatenates these in order to the file result.

Parameters:
  • to (str) – The output file name.

  • comm (mpi4py.MPI.Comm) – The optional MPI communicator.

  • overwrite (bool) – if True overwrite file, otherwise backup to to.N first

desispec.parallel.take_turns(comm, at_a_time, func, *args, **kwargs)[source]

Processes call a function in groups.

Any extra positional and keyword arguments are passed to the function.

Parameters:
  • comm – mpi4py.MPI.Comm or None.

  • at_a_time (int) – the maximum number of processes to run at a time.

  • func – the function to call.

Returns:

The return value on each process is the return value of the function.

desispec.parallel.use_mpi()[source]

Return whether we can use MPI.

desispec.parallel.weighted_partition(weights, n, groups_per_node=None)[source]

Partition weights into n groups with approximately same sum(weights)

Parameters:
  • weights – array-like weights

  • n – number of groups

Returns list of lists of indices of weights for each group

Notes

compared to dist_discrete_all, this function allows non-contiguous items to be grouped together which allows better balancing.

desispec.pipeline

Tools for pipeline creation and running.

desispec.pipeline.control

Tools for controling pipeline production.

desispec.pipeline.control.chain(tasktypes, nightstr=None, states=None, expid=None, spec=None, pack=False, nosubmitted=False, depjobs=None, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, dryrun=False)[source]

Run a chain of jobs for multiple pipeline steps.

For the list of task types, get all ready tasks meeting the selection criteria. Then either pack all tasks into one job or submit each task type as its own job. Input job dependencies can be specified, and dependencies are tracked between jobs in the chain.

Parameters:
  • tasktypes (list) – list of valid task types.

  • nightstr (str) – Comma separated (YYYYMMDD) or regex pattern. Only nights matching these patterns will be considered.

  • states (list) – list of task states to select.

  • nights (list) – list of nights to select.

  • expid (int) – exposure ID to select.

  • pack (bool) – if True, pack all tasks into a single job.

  • nosubmitted (bool) – if True, do not run jobs that have already been submitted.

  • depjobs (list) – list of job ID dependencies.

  • nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).

  • nersc_queue (str) – the name of the queue to use (regular | debug | realtime).

  • nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.

  • nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.

  • nersc_shifter (str) – the name of the shifter image to use.

  • mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.

  • mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.

  • procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.

  • nodb (bool) – if True, do not use the production DB.

  • out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.

  • debug (bool) – if True, enable DEBUG log level in generated scripts.

  • dryrun (bool) – if True, do not submit the jobs.

Returns:

the job IDs from the final step in the chain.

Return type:

list

desispec.pipeline.control.check_tasks(tasks, db=None)[source]

Check the state of pipeline tasks.

If the database handle is given, use the DB for checking. Otherwise use the filesystem.

Parameters:
  • tasks (list) – list of tasks to check.

  • db (DataBase) – the database to use.

Returns:

dictionary of the state of each task.

Return type:

OrderedDict

desispec.pipeline.control.cleanup(db, tasktypes, failed=False, submitted=False, expid=None)[source]

Clean up stale tasks in the DB.

Parameters:
  • db (DataBase) – the production DB.

  • tasktypes (list) – list of valid task types.

  • failed (bool) – also clear failed states.

  • submitted (bool) – also clear submitted flag.

  • expid (int) – only clean this exposure ID.

desispec.pipeline.control.create(root=None, data=None, redux=None, prod=None, force=False, basis=None, calib=None, db_sqlite=False, db_sqlite_path=None, db_postgres=False, db_postgres_host='nerscdb03.nersc.gov', db_postgres_port=5432, db_postgres_name='desidev', db_postgres_user='desidev_admin', db_postgres_authorized='desidev_ro', nside=64)[source]

Create (or re-create) a production.

Parameters:
  • root (str) – value to use for DESI_ROOT.

  • data (str) – value to use for DESI_SPECTRO_DATA.

  • redux (str) – value to use for DESI_SPECTRO_REDUX.

  • prod (str) – value to use for SPECPROD.

  • force (bool) – if True, overwrite existing production DB.

  • basis (str) – value to use for DESI_BASIS_TEMPLATES.

  • calib (str) – value to use for DESI_SPECTRO_CALIB.

  • db_sqlite (bool) – if True, use SQLite for the DB.

  • db_sqlite_path (str) – override path to SQLite DB.

  • db_postgres (bool) – if True, use PostgreSQL for the DB.

  • db_postgres_host (str) – PostgreSQL hostname.

  • db_postgres_port (int) – PostgreSQL connection port number.

  • db_postgres_name (str) – PostgreSQL DB name.

  • db_postgres_user (str) – PostgreSQL user name.

  • db_postgres_authorized (str) – Additional PostgreSQL users to authorize.

  • nside (int) – HEALPix nside value used for spectral grouping.

desispec.pipeline.control.dryrun(tasks, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, db_postgres_user='desidev_ro', force=False)[source]

Print equivalent command line jobs.

For the specified tasks, print the equivalent stand-alone commands that would be run on each task. A pipeline job calls the internal desispec.scripts entrypoints directly.

Parameters:
  • tasks (list) – list of tasks to run.

  • nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).

  • nersc_queue (str) – the name of the queue to use (regular | debug | realtime).

  • nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.

  • nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.

  • nersc_shifter (str) – the name of the shifter image to use.

  • mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.

  • mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.

  • procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.

  • nodb (bool) – if True, do not use the production DB.

  • db_postgres_user (str) – If using postgres, connect as this user for read-only access”

  • force (bool) – if True, print commands for all tasks, not just the ones in a ready state.

desispec.pipeline.control.gen_scripts(tasks_by_type, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, db_postgres_user='desidev_ro')[source]

Generate scripts to run tasks of one or more types.

If multiple task type keys are contained in the dictionary, they will be packed into a single batch job.

Parameters:
  • tasks_by_type (dict) – each key is the task type and the value is a list of tasks.

  • nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).

  • nersc_queue (str) – the name of the queue to use (regular | debug | realtime).

  • nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.

  • nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.

  • nersc_shifter (str) – the name of the shifter image to use.

  • mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.

  • mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.

  • procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.

  • nodb (bool) – if True, do not use the production DB.

  • out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.

  • debug (bool) – if True, enable DEBUG log level in generated scripts.

  • db_postgres_user (str) – If using postgres, connect as this user for read-only access”

Returns:

the generated script files

Return type:

list

desispec.pipeline.control.get_tasks(db, tasktypes, nights, states=None, expid=None, spec=None, nosubmitted=False, taskfile=None)[source]

Get tasks of multiple types that match certain criteria.

Parameters:
  • db (DataBase) – the production DB.

  • tasktypes (list) – list of valid task types.

  • states (list) – list of task states to select.

  • nights (list) – list of nights to select.

  • expid (int) – exposure ID to select.

  • spec (int) – spectrograph to select.

  • nosubmitted (bool) – if True, ignore tasks that were already submitted.

Returns:

all tasks of all types.

Return type:

list

desispec.pipeline.control.get_tasks_type(db, tasktype, states, nights, expid=None, spec=None)[source]

Get tasks of one type that match certain criteria.

Parameters:
  • db (DataBase) – the production DB.

  • tasktype (str) – a valid task type.

  • states (list) – list of task states to select.

  • nights (list) – list of nights to select.

  • expid (int) – exposure ID to select.

  • spec (int) – spectrograph to select.

Returns:

list of tasks meeting the criteria.

Return type:

(list)

desispec.pipeline.control.getready(db, nightstr=None)[source]

Update forward dependencies in the database.

Update database for one or more nights to ensure that forward dependencies know that they are ready to run.

Parameters:
  • db (DataBase) – the production DB.

  • nightstr (list) – comma separated (YYYYMMDD) or regex pattern.

desispec.pipeline.control.run(taskfile, nosubmitted=False, depjobs=None, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False)[source]

Create job scripts and run them.

This gets tasks from the taskfile and sorts them by type. Then it generates the scripts. Finally, it runs or submits those scripts to the scheduler.

Parameters:
  • taskfile (str) – read tasks from this file (if not specified, read from STDIN).

  • nosubmitted (bool) – if True, do not run jobs that have already been submitted.

  • depjobs (list) – list of job ID dependencies.

  • nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).

  • nersc_queue (str) – the name of the queue to use (regular | debug | realtime).

  • nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.

  • nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.

  • nersc_shifter (str) – the name of the shifter image to use.

  • mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.

  • mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.

  • procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.

  • nodb (bool) – if True, do not use the production DB.

  • out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.

  • debug (bool) – if True, enable DEBUG log level in generated scripts.

Returns:

the job IDs returned by the scheduler.

Return type:

list

desispec.pipeline.control.run_scripts(scripts, deps=None, slurm=False)[source]

Run job scripts with optional dependecies.

This either submits the jobs to the scheduler or simply runs them in order with subprocess.

Parameters:
  • scripts (list) – list of pathnames of the scripts to run.

  • deps (list) – optional list of job IDs which are dependencies for these scripts.

  • slurm (bool) – if True use slurm to submit the jobs.

Returns:

the job IDs returned by the scheduler.

Return type:

list

desispec.pipeline.control.script(taskfile, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, db_postgres_user='desidev_ro')[source]

Generate pipeline scripts for a taskfile.

This gets tasks from the taskfile and sorts them by type. Then it generates the scripts.

Parameters:
  • taskfile (str) – read tasks from this file (if not specified, read from STDIN).

  • nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).

  • nersc_queue (str) – the name of the queue to use (regular | debug | realtime).

  • nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.

  • nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.

  • nersc_shifter (str) – the name of the shifter image to use.

  • mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.

  • mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.

  • procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.

  • nodb (bool) – if True, do not use the production DB.

  • out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.

  • debug (bool) – if True, enable DEBUG log level in generated scripts.

  • db_postgres_user (str) – If using postgres, connect as this user for read-only access”

Returns:

the generated script files

Return type:

list

desispec.pipeline.control.status(task=None, tasktypes=None, nightstr=None, states=None, expid=None, spec=None, db_postgres_user='desidev_ro')[source]

Check the status of pipeline tasks.

Args:

Returns:

None

desispec.pipeline.control.sync(db, nightstr=None, specdone=False)[source]

Synchronize DB state based on the filesystem.

This scans the filesystem for all tasks for the specified nights, and updates the states accordingly.

Parameters:
  • db (DataBase) – the production DB.

  • nightstr (list) – comma separated (YYYYMMDD) or regex pattern.

  • specdone – If true, set spectra to done if files exist.

desispec.pipeline.control.tasks(tasktypes, nightstr=None, states=None, expid=None, spec=None, nosubmitted=False, db_postgres_user='desidev_ro', taskfile=None)[source]

Get tasks of multiple types that match certain criteria.

Parameters:
  • tasktypes (list) – list of valid task types.

  • nightstr (list) – comma separated (YYYYMMDD) or regex pattern.

  • states (list) – list of task states to select.

  • expid (int) – exposure ID to select.

  • spec (int) – spectrograph to select.

  • nosubmitted (bool) – if True, ignore tasks that were already submitted.

  • db_postgres_user (str) – If using postgres, connect as this user for read-only access”

  • taskfile (str) – if set write to this file, else write to STDOUT.

desispec.pipeline.control.update(nightstr=None, nside=64, expid=None)[source]

Update a production.

Parameters:
  • nightstr (str) – Comma separated (YYYYMMDD) or regex pattern. Only nights matching these patterns will be considered.

  • nside (int) – HEALPix nside value used for spectral grouping.

  • expid (int) – Only update the production for a single exposure ID.

desispec.pipeline.db

Pipeline processing database

class desispec.pipeline.db.DataBase[source]

Class for tracking pipeline processing objects and state.

cleanup(tasktypes=None, expid=None, cleanfailed=False, cleansubmitted=False)[source]

Reset states of tasks.

Any tasks that are marked as “running” will have their state reset to “ready”. This can be called if a job dies before completing all tasks.

Parameters:
  • tasktypes (list) – if not None, clean up only tasks of these types.

  • expid (int) – if not None, only clean tasks related to this exposure ID. Note that tasks which are independent of an expid (psfnight, fiberflatnight, spectra, redshift) will be ignored if this option is given.

  • cleanfailed (bool) – if True, also reset failed tasks to ready.

  • cleansubmitted (bool) – if True, set submitted flag to False.

count_task_states(tasktype)[source]

Return a dictionary of how many tasks are in each state

Parameters:

tasktype (str) – the type of these tasks.

Returns:

keyed by state, values are number of tasks in that state0

Return type:

dict

get_states(tasks)[source]

Efficiently get the state of many tasks at once.

Parameters:

tasks (list) – list of task names.

Returns:

the state of each task.

Return type:

dict

get_states_type(tasktype, tasks)[source]

Efficiently get the state of many tasks of a single type.

Parameters:
  • tasktype (str) – the type of these tasks.

  • tasks (list) – list of task names.

Returns:

the state of each task.

Return type:

dict

get_submitted(tasks)[source]

Return the submitted flag for the list of tasks.

Parameters:

tasks (list) – list of task names.

Returns:

the boolean submitted state of each task (True means that

the task has been submitted).

Return type:

(dict)

getready(night=None)[source]

Update DB, changing waiting to ready depending on status of dependencies .

Parameters:

night (str) – The night to process.

set_states(tasks)[source]

Efficiently set the state of many tasks at once.

Parameters:

tasks (list) – list of tuples containing the task name and the state to set.

Returns:

Nothing.

set_states_type(tasktype, tasks, postprocessing=True)[source]

Efficiently get the state of many tasks of a single type.

Parameters:
  • tasktype (str) – the type of these tasks.

  • tasks (list) – list of tuples containing the task name and the state to set.

Returns:

Nothing.

set_submitted(tasks, unset=False)[source]

Flag a list of tasks as submitted.

Parameters:
  • tasks (list) – list of task names.

  • unset (bool) – if True, invert the behavior and unset the submitted flag for these tasks.

Returns:

Nothing.

set_submitted_type(tasktype, tasks, unset=False)[source]

Flag a list of tasks of a single type as submitted.

Parameters:
  • tasktype (str) – the type of these tasks.

  • tasks (list) – list of task names.

  • unset (bool) – if True, invert the behavior and unset the submitted flag for these tasks.

Returns:

Nothing.

sync(night, specdone=False)[source]

Update states of tasks based on filesystem.

Go through all tasks in the DB for the given night and determine their state on the filesystem. Then update the DB state to match.

Parameters:
  • night (str) – The night to scan for updates.

  • specdone – If true, set spectra to done if files exist.

update(night, nside, expid=None)[source]

Update DB based on raw data.

This will use the usual io.meta functions to find raw exposures. For each exposure, the fibermap and all following objects will be added to the DB.

Parameters:
  • night (str) – The night to scan for updates.

  • nside (int) – The current NSIDE value used for pixel grouping.

  • expid (int) – Only update the DB for this exposure.

class desispec.pipeline.db.DataBasePostgres(host, port, dbname, user, schema=None, authorize=None)[source]

Pipeline database using PostgreSQL as the backend.

Parameters:
  • host (str) – The database server.

  • port (int) – The connection port.

  • dbname (str) – The database to connect.

  • user (str) – The user name for the connection. The password should be stored in the ~/.pgpass file.

  • schema (str) – The schema within the database. If this is specified, then the database is assumed to exist. Otherwise the schema is computed from a hash of the production location and will be created.

  • authorize (str) – If creating the schema, this is the list of additional roles that should be granted access.

initdb()[source]

Create DB tables for all tasks if they do not exist.

class desispec.pipeline.db.DataBaseSqlite(path, mode)[source]

Pipeline database using sqlite3 as the backend.

Parameters:
  • path (str) – the filesystem path of the database to open. If None, then a temporary database is created in memory.

  • mode (str) – if “r”, the database is open in read-only mode. If “w”, the database is open in read-write mode and created if necessary.

initdb()[source]

Create DB tables for all tasks if they do not exist.

desispec.pipeline.db.all_task_types()[source]

Get the list of possible task types that are supported.

Returns:

The list of supported task types.

Return type:

list

desispec.pipeline.db.all_tasks(night, nside, expid=None)[source]

Get all possible tasks for a single night.

This uses the filesystem to query the raw data for a particular night and return a dictionary containing all possible tasks for each task type. For objects which span multiple nights (e.g. spectra, redrock), this returns the tasks which are touched by the given night.

Parameters:
  • night (str) – The night to scan for tasks.

  • nside (int) – The HEALPix NSIDE value to use.

  • expid (int) – Only get tasks for this single exposure.

Returns:

a dictionary whose keys are the task types and where each value

is a list of task properties.

Return type:

dict

desispec.pipeline.db.check_tasks(tasklist, db=None, inputs=None)[source]

Check a list of tasks and return their state.

If the database is specified, it is used to check the state of the tasks and their dependencies. Otherwise the filesystem is checked.

Parameters:
  • tasklist (list) – list of tasks.

  • db (pipeline.db.DB) – The optional database to use.

  • inputs (dict) – optional dictionary containing the only input dependencies that should be considered.

Returns:

The current state of all tasks.

Return type:

dict

desispec.pipeline.db.load_db(dbstring, mode='w', user=None)[source]

Load a database from a connection string.

This instantiates either an sqlite or postgresql database using a string. If this string begins with “postgresql:”, then it is taken to be the information needed to connect to a postgres server. Otherwise it is assumed to be a filesystem path to use with sqlite. The mode is only meaningful when using sqlite. Postgres permissions are controlled through the user permissions.

Parameters:
  • dbstring (str) – either a filesystem path (sqlite) or a colon-separated string of connection properties in the form “postresql:<host>:<port>:<dbname>:<user>:<schema>”.

  • mode (str) – for sqlite, the mode.

  • user (str) – for postgresql, an alternate user name for opening the DB. This can be used to connect as a user with read-only access.

Returns:

a derived database class of the appropriate type.

Return type:

DataBase

desispec.pipeline.db.task_sort(tasks)[source]

Sort a list of tasks by type.

This takes a list of arbitrary tasks and sorts them by type. The result is placed in an ordered dictionary of lists in run order.

Parameters:

tasks (list) – the list of input tasks.

Returns:

ordered dictionary of tasks sorted by type.

Return type:

(OrderedDict)

desispec.pipeline.defs

Common definitions needed by pipeline modules.

desispec.pipeline.defs.prod_options_name = 'options.yaml'

The name of the options file inside the run directory.

desispec.pipeline.defs.state_colors = {'done': '#00ff00', 'failed': '#ff0000', 'ready': '#0000ff', 'running': '#ffff00', 'waiting': '#000000'}

State colors used for visualization.

desispec.pipeline.defs.task_name_sep = '_'

The separator string used for building object names.

desispec.pipeline.defs.task_states = ['waiting', 'ready', 'running', 'done', 'failed']

The valid states of each pipeline task.

desispec.pipeline.plan

Tools for planning pipeline runs.

desispec.pipeline.plan.compute_nodes(nworker, taskproc, nodeprocs)[source]

Compute number of nodes for the number of workers.

Parameters:
  • nworker (int) – The number of workers.

  • taskproc (int) – The number of processes per task.

  • nodeprocs (int) – The number of processes per node.

Returns:

The number of required nodes.

Return type:

(int)

desispec.pipeline.plan.compute_worker_tasks(tasktype, tasklist, tfactor, nworker, workersize, startup=0.0, db=None, num_nodes=None)[source]

Compute the distribution of tasks for specified workers.

Parameters:
  • tasktype (str) – The task type.

  • tasklist (list) – List of tasks, all of type tasktype.

  • tfactor (float) – Additional runtime scaling factor.

  • nworker (int) – The number of workers.

  • workersize (int) – The number of processes in each worker.

  • startup (float, optional) – Startup overhead in minutes for each worker.

  • db (DataBase, optional) – the database to pass to the task runtime calculation.

  • num_nodes (int, optional) – number of nodes over which the workers are distributed

Returns:

The (sorted tasks, sorted runtime weights, dist) results where dist is the a list of tuples (one per worker) indicating the indices of tasks for that worker in the returned sorted list of tasks.

Return type:

(tuple)

desispec.pipeline.plan.nersc_job_size(tasktype, tasklist, machine, queue, maxtime, maxnodes, nodeprocs=None, db=None, balance=False)[source]

Compute the NERSC job parameters based on constraints.

Given the list of tasks, query their estimated runtimes and determine the “best” job size to use. If the job is too big to fit the constraints then split up the tasks into multiple jobs.

If maxtime or maxnodes is zero, then the defaults for the queue are used.

The default behavior is to create jobs that are as large as possible- i.e. to run all tasks simultaneously in parallel. In general, larger jobs with a shorter run time will move through the queue faster. If the job size exceeds the maximum number of nodes, then the job size is fixed to this maximum and the runtime is extended. If the runtime exceeds maxtime, then the job is split.

Parameters:
  • tasktype (str) – the type of these tasks.

  • tasklist (list) – the list of tasks.

  • machine (str) – the nersc machine name, e.g. cori-haswell, cori-knl

  • queue (str) – the nersc queue name, e.g. regular or debug

  • maxtime (int) – the maximum run time in minutes.

  • maxnodes (int) – the maximum number of nodes.

  • nodeprocs (int) – the number of processes per node. If None, estimate this based on the per-process memory needs of the task and the machine properties.

  • db (DataBase) – the database to pass to the task runtime calculation.

  • balance (bool) – if True, change the number of workers to load balance the job.

Returns:

List of tuples (nodes, nodeprocs, runtime, nworker, workersize,

tasks) containing one entry per job. Each entry specifies the number of nodes to use, the expected total runtime, number of workers, and the list of tasks for that job.

Return type:

list

desispec.pipeline.plan.nersc_machine(name, queue)[source]

Return the properties of the specified NERSC host.

Parameters:
  • name (str) – the name of the host. Allowed values are: cori-haswell and cori-knl.

  • queue (str) – the queue on the machine (regular, debug, etc)

Returns:

properties of this machine.

Return type:

dict

desispec.pipeline.plan.worker_times(tasktimes, workerdist, startup=0.0)[source]

Compute the time needed for each worker.

Parameters:
  • tasktimes (array) – array of individual task times.

  • workerdist (list) – List of tuples of indices in taskstimes.

  • startup (float) – Startup overhead in minutes for each worker.

Returns:

The (worker times, min, max).

Return type:

(tuple)

Notes / Examples:

len(tasktimes) = number of tasks len(workerdist) = number of workers workerdist[i] = tuple of tasktime indices assigned to worker i sum(tasktimes[workerdist[i]]) = expected total time for worker i

desispec.pipeline.prod

Functions for updating and loading a production.

desispec.pipeline.prod.load_prod(mode='w', user=None)[source]

Load the database and options for a production.

This loads the database from the production location defined by the usual DESI environment variables. It also loads the global options file for the production.

Parameters:
  • mode (str) – open mode for sqlite database (“r” or “w”).

  • user (str) – for postgresql, an alternate user name for opening the DB. This can be used to connect as a user with read-only access.

Returns:

(pipeline.db.DataBase, dict) The database for the production

and the global options dictionary.

Return type:

tuple

desispec.pipeline.prod.select_nights(allnights, nightstr)[source]

Select nights based on regex matches.

Given a list of nights, select all nights matching the specified patterns and return this subset.

Parameters:
  • allnights (list) – list of all nights as strings

  • nightstr (str) – comma-separated list of regex patterns.

Returns:

list of nights that match the patterns.

Return type:

list

desispec.pipeline.prod.task_read(path)[source]

Read a task list from a text file or STDIN.

Lines that begin with ‘#’ are ignored as comments. If the path is None, lines are read from STDIN until an EOF marker is received.

Parameters:

path (str) – the input file name.

Returns:

the list of tasks.

Return type:

list

desispec.pipeline.prod.task_write(path, tasklist)[source]

Write a task list to a text file or STDOUT.

If the path is None, write lines to STDOUT. In all cases, write a special termination line so that this stream or file can be passed into the task_read function.

Parameters:
  • path (str) – the output file name.

  • tasklist (list) – the data.

Returns:

nothing.

desispec.pipeline.prod.update_prod(nightstr=None, hpxnside=64, expid=None)[source]

Create or update a production directory tree.

For a given production, create the directory hierarchy and the starting default options.yaml file if it does not exist. Also initialize the production DB if it does not exist. Then update the DB with one or more nights from the raw data. Nights to update may be specified by a list of simple regex matches.

Parameters:
  • nightstr (str) – comma-separated list of regex patterns.

  • hpxnside (int) – The nside value to use for spectral grouping.

  • expid (int) – Only update a single exposure. If this is specified, then nightstr must contain only a single night.

desispec.pipeline.prod.yaml_read(path)[source]

Read a dictionary from a file.

Parameters:

path (str) – the input file name.

Returns:

the data.

Return type:

dict

desispec.pipeline.prod.yaml_write(path, input)[source]

Write a dictionary to a file.

Parameters:
  • path (str) – the output file name.

  • input (dict) – the data.

Returns:

nothing.

desispec.pipeline.run

Tools for running the pipeline.

exception desispec.pipeline.run.TimeoutError[source]
desispec.pipeline.run.dry_run(tasktype, tasklist, opts, procs, procs_per_node, db=None, launch='mpirun -np', force=False)[source]

Compute the distribution of tasks and equivalent commands.

This function takes similar arguments as run_task_list() except simulates the data distribution and commands that would be run if given the specified number of processes and processes per node.

This can be used to debug issues with the runtime concurrency or the actual options that will be passed to the underying main() entry points for each task.

This function requires that the DESI environment variables are set to point to the current production directory.

Only tasks that are ready to run (based on the filesystem checks or the database) will actually be attempted.

NOTE: Since this function is just informative and for interactive use, we print information directly to STDOUT rather than logging.

Parameters:
  • tasktype (str) – the pipeline step to process.

  • tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.

  • opts (dict) – the global options (for example, as read from the production options.yaml file).

  • procs (int) – the number of processes to simulate.

  • procs_per_node (int) – the number of processes per node to simulate.

  • db (pipeline.db.DB) – The optional database to update.

  • launch (str) – The launching command for a job. This is just a convenience and prepended to each command before the number of processes.

  • force (bool) – If True, ignore database and filesystem state and just run the tasks regardless.

Returns:

Nothing.

desispec.pipeline.run.run_dist(tasktype, tasklist, db, nproc, procs_per_node, force=False)[source]

Compute the runtime distribution of tasks.

For a given number of processes, parse job environment variables and compute the number of workers to use and the remaining tasks to process. Divide the processes into groups, and associate some (or all) of those groups to workers. Assign tasks to these groups of processes. Some groups may have zero tasks if there are more groups than workers needed.

Returns:

The (groupsize, groups, tasks, dist) information. Groupsize

is the processes per group. Groups is a list of tuples (one per process) giving the group number and rank within the group. The tasks are a sorted list of tasks containing the subset of the inputs that needs to be run. The dist is a list of tuples (one per group) containing the indices of tasks assigned to each group.

Return type:

tuple

desispec.pipeline.run.run_task(name, opts, comm=None, logfile=None, db=None)[source]

Run a single task.

Based on the name of the task, call the appropriate run function for that task. Log output to the specified file. Run using the specified MPI communicator and optionally update state to the specified database.

Note: This function DOES NOT check the database or filesystem to see if the task has been completed or if its dependencies exist. It assumes that some higher-level code has done that if necessary.

Parameters:
  • name (str) – the name of this task.

  • opts (dict) – options to use for this task.

  • comm (mpi4py.MPI.Comm) – optional MPI communicator.

  • logfile (str) – output log file. If None, do not redirect output to a file.

  • db (pipeline.db.DB) – The optional database to update.

Returns:

the total number of processes that failed.

Return type:

int

desispec.pipeline.run.run_task_list(tasktype, tasklist, opts, comm=None, db=None, force=False)[source]

Run a collection of tasks of the same type.

This function requires that the DESI environment variables are set to point to the current production directory.

This function first takes the communicator and uses the maximum processes per task to split the communicator and form groups of processes of the desired size. It then takes the list of tasks and uses their relative run time estimates to assign tasks to the process groups. Each process group loops over its assigned tasks.

If the database is not specified, no state tracking will be done and the filesystem will be checked as needed to determine the current state.

Only tasks that are ready to run (based on the filesystem checks or the database) will actually be attempted.

Parameters:
  • tasktype (str) – the pipeline step to process.

  • tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.

  • opts (dict) – the global options (for example, as read from the production options.yaml file).

  • comm (mpi4py.Comm) – the full communicator to use for whole set of tasks.

  • db (pipeline.db.DB) – The optional database to update.

  • force (bool) – If True, ignore database and filesystem state and just run the tasks regardless.

Returns:

the number of ready tasks, number that are done, and the number

that failed.

Return type:

tuple

desispec.pipeline.run.run_task_list_db(tasktype, tasklist, comm=None)[source]

Run a list of tasks using the pipeline DB and options.

This is a wrapper around run_task_list which uses the production database and global options file.

Parameters:
  • tasktype (str) – the pipeline step to process.

  • tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.

  • comm (mpi4py.Comm) – the full communicator to use for whole set of tasks.

Returns:

the number of ready tasks, and the number that failed.

Return type:

tuple

desispec.pipeline.run.run_task_simple(name, opts, comm=None)[source]

Run a single task with no DB or log redirection.

This a wrapper around run_task() for use without a database and with no log redirection. See documentation for that function.

Parameters:
  • name (str) – the name of this task.

  • opts (dict) – options to use for this task.

  • comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the total number of processes that failed.

Return type:

int

desispec.pipeline.scriptgen

Tools for generating shell and slurm scripts.

desispec.pipeline.scriptgen.batch_nersc(tasks_by_type, outroot, logroot, jobname, machine, queue, maxtime, maxnodes, nodeprocs=None, openmp=False, multiproc=False, db=None, shifterimg=None, debug=False)[source]

Generate slurm script(s) to process lists of tasks.

Given sets of task lists and constraints about the machine, generate slurm scripts for use at NERSC.

Parameters:
  • tasks_by_type (OrderedDict) – Ordered dictionary of the tasks for each type to be written to a single job script.

  • outroot (str) – root output script name.

  • logroot (str) – root output log name.

  • jobname (str) – the name of the job.

  • machine (str) – the NERSC machine name.

  • queue (str) – the name of the queue

  • maxtime (int) – the maximum run time in minutes.

  • maxnodes (int) – the maximum number of nodes to use.

  • nodeprocs (int) – the number of processes to use per node.

  • openmp (bool) – if True, set OMP_NUM_THREADS to the correct value.

  • multiproc (bool) – if True, use OMP_NUM_THREADS=1 and disable core binding of processes.

  • db (DataBase) – the pipeline database handle.

  • shifter (str) – the name of the shifter image to use.

  • debug (bool) – if True, set DESI log level to DEBUG in the script.

Returns:

list of generated slurm files.

Return type:

(list)

desispec.pipeline.scriptgen.batch_shell(tasks_by_type, outroot, logroot, mpirun='', mpiprocs=1, openmp=1, db=None)[source]

Generate bash script(s) to process lists of tasks.

Given sets of task lists, generate a script that processes each in order.

Parameters:
  • tasks_by_type (OrderedDict) – Ordered dictionary of the tasks for each type to be written to a single job script.

  • outroot (str) – root output script name.

  • logroot (str) – root output log name.

  • mpirun (str) – optional command to use for launching MPI programs.

  • mpiprocs (int) – if mpirun is specified, use this number of processes.

  • openmp (int) – value to set for OMP_NUM_THREADS.

  • db (DataBase) – the pipeline database handle.

Returns:

list of generated script files.

Return type:

(list)

desispec.pipeline.scriptgen.dump_job_env(fh, tfactor, startup, nworker, workersize)[source]

Write parameters needed at runtime to an open filehandle.

desispec.pipeline.scriptgen.nersc_job(jobname, path, logroot, desisetup, commands, machine, queue, nodes, cnodes, ppns, minutes, nworker, workersize, multisrun=False, openmp=False, multiproc=False, shifterimg=None, debug=False)[source]

Create a SLURM script for use at NERSC.

Args:

desispec.pipeline.scriptgen.parse_job_env()[source]

Retrieve job parameters from the environment.

desispec.pipeline.tasks

Classes that describe pipeline tasks.

desispec.pipeline.tasks.base

Common operations for pipeline tasks.

class desispec.pipeline.tasks.base.BaseTask[source]

Base class for tasks.

This defines the interfaces for the classes representing pipeline tasks. This class should not be instantiated directly.

_create(db)[source]

See BaseTask.create.

_insert(cursor, props)[source]

See BaseTask.insert.

_retrieve(db, name)[source]

See BaseTask.retrieve.

_run_max_mem_proc(name, db)[source]

Return zero (i.e. not a limit)

_run_max_mem_task(name, db)[source]

Return zero (i.e. no memory requirement)

_state_get(db, name, cur=None)[source]

See BaseTask.state_get.

_state_set(db, name, state, cur=None)[source]

See BaseTask.state_set.

create(db)[source]

Initialize a database for this task type.

This may include creating one or more tables.

Parameters:

db (pipeline.DB) – the database instance.

deps(name, db=None, inputs=None)[source]

Get the dependencies for a task.

This gets a list of other tasks which are required.

Parameters:
  • name (str) – the task name.

  • db (pipeline.DB) – the optional database instance.

  • inputs (dict) – optional dictionary containing the only input dependencies that should be considered.

Returns:

a dictionary of dependencies. The keys are arbitrary and

the values can be either scalar task names or lists of tasks.

Return type:

dict

getready(db, name, cur)[source]

Checks whether dependencies are ready

insert(cursor, props)[source]

Insert a task into a database.

This uses the name and extra keywords to update one or more task-specific tables.

Parameters:
  • cursor (DB cursor) – the database cursor of an open connection.

  • props (dict) – dictionary of properties for the task.

name_join(props)[source]

Construct a task name from its properties.

Parameters:

props (dict) – dictionary of properties.

Returns:

the task name.

Return type:

str

name_split(name)[source]

Split a task name into its properties.

Parameters:

name (str) – the task name.

Returns:

dictionary of properties.

Return type:

dict

paths(name)[source]

The filesystem path(s) associated with this task.

Parameters:

name (str) – the task name.

Returns:

the list of output files generated by this task.

Return type:

list

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

retrieve(db, name)[source]

Retrieve all task information from the DB.

This may include additional information beyond the contents of the task name (e.g. from other tables).

Parameters:
  • db (pipeline.DB) – the database instance.

  • name (str) – the task name.

Returns:

dictionary of properties for the task.

Return type:

dict

run(name, opts, comm=None, db=None)[source]

Run the task.

Parameters:
  • name (str) – the name of this task.

  • opts (dict) – options to use for this task.

  • comm (mpi4py.MPI.Comm) – optional MPI communicator.

  • db (pipeline.db.DB) – The database.

Returns:

the number of processes that failed.

Return type:

int

run_and_update(db, name, opts, comm=None)[source]

Run the task and update DB state.

The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.

Parameters:
  • db (pipeline.db.DB) – The database.

  • name (str) – the name of this task.

  • opts (dict) – options to use for this task.

  • comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the number of processes that failed.

Return type:

int

run_cli(name, opts, procs, launch=None, log=None, db=None)[source]

Return the equivalent command-line interface.

Parameters:
  • name (str) – the name of the task.

  • opts (dict) – dictionary of runtime options.

  • procs (int) – The number of processes to use.

  • launch (str) – optional launching command.

  • log (str) – optional log file for output.

  • db (pipeline.db.DB) – The database.

Returns:

a command line.

Return type:

str

run_defaults()[source]

Default options.

This dictionary of default options will be written to the options.yaml file in a production directory. The options will then be loaded from that file at run time.

Changes to this function will only impact newly-created productions, and these options will be overridden by any changes to the options.yaml file.

Returns:

dictionary of default options.

Return type:

dict

run_max_mem_proc(name, db=None)[source]

Maximum memory in GB per process required.

If zero is returned, it indicates that the memory requirement is so small that the code can run fully-packed on any system.

Parameters:
  • name (str) – the name of the task.

  • db (pipeline.DB) – the optional database instance.

Returns:

the required RAM in GB per process.

Return type:

float

run_max_mem_task(name, db=None)[source]

Maximum memory in GB per task required.

If zero is returned, it indicates that the memory requirement is so small that the code can run on a single node.

Parameters:
  • name (str) – the name of the task.

  • db (pipeline.DB) – the optional database instance.

Returns:

the required RAM in GB per process.

Return type:

float

run_max_procs()[source]

Maximum number of processes supported by this task type.

Parameters:

procs_per_node (int) – the number of processes running per node.

Returns:

the maximum number of processes. Zero indicates no limit.

Return type:

int

run_time(name, procs, db=None)[source]

Estimated runtime for a task at maximum concurrency.

Parameters:
  • name (str) – the name of the task.

  • procs (int) – the total number of processes used for this task.

  • db (pipeline.DB) – the optional database instance.

Returns:

estimated minutes of run time.

Return type:

int

state_get(db, name, cur=None)[source]

Get the state of a task.

This should not be called repeatedly for many tasks- it is more efficient to get the state of many tasks in a single custom SQL query.

Parameters:
  • db (pipeline.DB) – the database instance.

  • name (str) – the task name.

Returns:

the state.

Return type:

str

state_set(db, name, state, cur=None)[source]

Set the state of a task.

This should not be called repeatedly if you are setting the state of many tasks. It is more efficient to do that in a single SQL command.

Parameters:
  • db (pipeline.DB) – the database instance.

  • name (str) – the task name.

desispec.pipeline.tasks.base.task_type(name)[source]

Given a task name, find the type from the list of available ones.

Parameters:

name (str) – the name of the task.

Returns:

the type of the task.

Return type:

str

desispec.pipeline.tasks.cframe

class desispec.pipeline.tasks.cframe.TaskCFrame[source]

Class containing the properties of a sky fit task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

_run_max_procs()[source]

See BaseTask.run_max_procs.

_run_time(name, procs, db)[source]

See BaseTask.run_time.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.extract

class desispec.pipeline.tasks.extract.TaskExtract[source]

Class containing the properties of one extraction task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.fiberflat

class desispec.pipeline.tasks.fiberflat.TaskFiberflat[source]

Class containing the properties of one extraction task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.fiberflatnight

class desispec.pipeline.tasks.fiberflatnight.TaskFiberflatNight[source]

Class containing the properties of one fiberflat combined night task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

getready(db, name, cur)[source]

Checks whether dependencies are ready

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.fibermap

class desispec.pipeline.tasks.fibermap.TaskFibermap[source]

Class containing the properties of one fibermap.

Since fibermaps have no dependencies and are not created by the pipeline, this class is just used to specify names, etc.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

desispec.pipeline.tasks.fluxcalib

class desispec.pipeline.tasks.fluxcalib.TaskFluxCalib[source]

Class containing the properties of a sky fit task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.preproc

class desispec.pipeline.tasks.preproc.TaskPreproc[source]

Class containing the properties of one preprocessed pixel file.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.psf

class desispec.pipeline.tasks.psf.TaskPSF[source]

Class containing the properties of one PSF task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.psfnight

class desispec.pipeline.tasks.psfnight.TaskPSFNight[source]

Class containing the properties of one PSF combined night task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_dict(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

getready(db, name, cur)[source]

Checks whether dependencies are ready

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.qadata

class desispec.pipeline.tasks.qadata.TaskQAData[source]

Class containing the properties of a sky fit task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.rawdata

class desispec.pipeline.tasks.rawdata.TaskRawdata[source]

Class containing the properties of one rawdata.

Since rawdatas have no dependencies and are not created by the pipeline, this class is just used to specify names, etc.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

desispec.pipeline.tasks.redshift

class desispec.pipeline.tasks.redshift.TaskRedshift[source]

Class containing the properties of one spectra task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

run_and_update(db, name, opts, comm=None)[source]

Run the task and update DB state.

The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.

It is specific for redshift because the healpix_frame table has to be updated

Parameters:
  • db (pipeline.db.DB) – The database.

  • name (str) – the name of this task.

  • opts (dict) – options to use for this task.

  • comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the number of processes that failed.

Return type:

int

desispec.pipeline.tasks.sky

class desispec.pipeline.tasks.sky.TaskSky[source]

Class containing the properties of a sky fit task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.spectra

class desispec.pipeline.tasks.spectra.TaskSpectra[source]

Class containing the properties of one spectra task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

run_and_update(db, name, opts, comm=None)[source]

Run the task and update DB state.

The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.

It is specific for spectra because the healpix_frame table has to be updated

Parameters:
  • db (pipeline.db.DB) – The database.

  • name (str) – the name of this task.

  • opts (dict) – options to use for this task.

  • comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the number of processes that failed.

Return type:

int

desispec.pipeline.tasks.starfit

class desispec.pipeline.tasks.starfit.TaskStarFit[source]

Class containing the properties of one extraction task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pipeline.tasks.traceshift

class desispec.pipeline.tasks.traceshift.TaskTraceShift[source]

Class containing the properties of one trace shift task.

_deps(name, db, inputs)[source]

See BaseTask.deps.

_option_list(name, opts)[source]

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)[source]

See BaseTask.paths.

_run(name, opts, comm, db)[source]

See BaseTask.run.

_run_cli(name, opts, procs, db)[source]

See BaseTask.run_cli.

_run_defaults()[source]

See BaseTask.run_defaults.

postprocessing(db, name, cur)[source]

For successful runs, postprocessing on DB

desispec.pixflat

Routines for pixel flat fielding.

desispec.pixflat.convolve2d(image, k, weight=None)[source]

Return a 2D convolution of image with kernel k, optionally with a weight image

Parameters:
  • image – 2D np.array image

  • k – 2D np.array kernel, each dimension must be odd and greater than 1

  • weight – 2D np.array of same shape as image

  • optional – 2D np.array of same shape as image

Returns:

2D np.array convolved image of same shape as input image

Return type:

cimage

desispec.pixgroup

Tools to regroup spectra in individual exposures by healpix on the sky.

class desispec.pixgroup.FrameLite(wave, flux, ivar, mask, resolution_data, fibermap, header, scores=None)[source]

Lightweight Frame object for regrouping

This is intended for I/O without the overheads of float32 -> float64 conversion, correcting endianness, etc.

classmethod read(filename)[source]

Return FrameLite read from filename

class desispec.pixgroup.SpectraLite(bands, wave, flux, ivar, mask, resolution_data, fibermap, exp_fibermap=None, scores=None)[source]

Lightweight spectra I/O object for regrouping

num_spectra()[source]

Get the number of spectra contained in this group.

Returns (int):

Number of spectra contained in this group.

num_targets()[source]

Get the number of distinct targets.

Returns (int):

Number of unique targets with spectra in this object.

classmethod read(filename)[source]

Return a SpectraLite object read from filename

target_ids()[source]

Return list of unique target IDs.

The target IDs are sorted by the order that they first appear.

Returns (array):

an array of integer target IDs.

write(filename, header=None)[source]

Write this SpectraLite object to filename

desispec.pixgroup.add_missing_frames(frames)[source]

Adds any missing frames with ivar=0 FrameLite objects with correct shape to match those that do exist.

Parameters:

frames – dict of FrameLite objects, keyed by (night,expid,camera)

Modifies frames in-place.

Example: if frames has keys (2020,1,’b0’) and (2020,1,’r0’) but not (2020,1,’z0’), this will add a blank FrameLite object for z0.

The purpose of this is to facilitate frames2spectra, which needs something for every spectro camera for every exposure that is included.

desispec.pixgroup.fibermap2tilepix(fibermap, nside=64)[source]

Maps fibermap to which healpix are covered by which petals

Parameters:

fibermap – table with columns TARGET_RA, TARGET_DEC, PETAL_LOC

Options:

nside (int): nested healpix nside (must be power of 2)

Returns dict petalpix[petal] = list(healpix covered by that petal)

desispec.pixgroup.frames2spectra(frames, pix=None, nside=64)[source]

Combine a dict of FrameLite into a SpectraLite for healpix pix

Parameters:

frames – dict of FrameLight, keyed by (night, expid, camera)

Options:

pix: only include targets in this NESTED healpix pixel number nside: Healpix nside, must be power of 2

Returns:

SpectraLite object with subset of spectra from frames that are in the requested healpix pixel pix

desispec.pixgroup.get_exp2healpix_map(expfile, survey=None, program=None, specprod_dir=None, nights=None, expids=None)[source]

Maps exposures to healpixels using preproc/NIGHT/EXPID/tilepix*.json files

Parameters:

expfile (str) – filepath to production exposure file with EXPOSURES and FRAMES HDUs

Options:

survey (str): filter by this survey (main, sv3, sv1, …) program (str): filter by this FAPRGRM (dark, bright, backup, other) specprod_dir (str): override $DESI_SPECTRO_REDUX/$SPECPROD nights (array-like): filter to only these nights expids (array-like): filter to only these exposure IDs

Returns table with columns NIGHT EXPID TILEID SURVEY PROGRAM SPECTRO HEALPIX

desispec.pixgroup.update_frame_cache(frames, framekeys, specprod_dir=None)[source]

Update a cache of FrameLite objects to match requested frameskeys

Parameters:
  • frames – dict of FrameLite objects, keyed by (night, expid, camera)

  • framekeys – list of desired (night, expid, camera)

Updates frames in-place

Notes

frames is dictionary, framekeys is list. When finished, the keys of frames match the entries in framekeys

desispec.preproc

Preprocess raw DESI exposures.

desispec.preproc._background(image, header, patch_width=200, stitch_width=10, stitch=False)[source]

determine background using a 2D median with square patches of width = width that are interpolated and optionnally try and match the level of amplifiers (does not subtract the background)

Parameters:
  • image (ndarray) –

  • = (( image) –

  • CCDSEC (header is used to read) –

Options:

patch_width (integer) size in pixels of the median square patches stitch_width (integer) width in pixels of amplifier edges to match level of amplifiers stitch : do match level of amplifiers

Returns background image with same shape as input image

desispec.preproc._clipped_std_bias(nsigma)[source]

Returns the bias on the standard deviation of a sigma-clipped dataset

Divide by the returned bias to get a corrected value:

a = nsigma
bias = sqrt((integrate x^2 exp(-x^2/2), x=-a..a) / (integrate exp(-x^2/2), x=-a..a))
     = sqrt(1 - 2a exp(-a^2/2) / (sqrt(2pi) erf(a/sqrt(2))))

See http://www.wolframalpha.com/input/?i=(integrate+x%5E2+exp(-x%5E2%2F2),+x+%3D+-a+to+a)+%2F+(integrate+exp(-x%5E2%2F2),+x%3D-a+to+a)

desispec.preproc._global_background(image, patch_width=200)[source]

determine background using a 2D median with square patches of width = width that are interpolated (does not subtract the background)

Parameters:
  • image (ndarray) –

  • = (( image) –

Options:

patch_width (integer) size in pixels of the median square patches

Returns background image with same shape as input image

desispec.preproc._overscan(pix, nsigma=5, niter=3)[source]

DEPRECATED: See calc_overscan

desispec.preproc._savgol_clipped(data, window=15, polyorder=5, niter=0, threshold=3.0)[source]

Simple method to iteratively do a SavGol filter with rejection and replacing rejected pixels by nearest neighbors

Parameters:
  • data (ndarray) –

  • window (int) – Window parameter for savgol

  • polyorder (int) –

  • niter (int) –

  • threshold (float) –

Returns:

desispec.preproc.calc_overscan(pix, nsigma=5, niter=3)[source]

Calculates overscan, readnoise from overscan image pixels

Parameters:

pix (ndarray) – overscan pixels from CCD image

Optional:

nsigma (float) : number of standard deviations for sigma clipping niter (int) : number of iterative refits

Returns:

Mean, sigma-clipped value readnoise (float):

Return type:

overscan (float)

desispec.preproc.compute_background_between_fiber_blocks(image, xyset)[source]

Computes CCD background between blocks of fibers

Parameters:
  • image – desispec.image.Image object

  • xyset – desispec.xytraceset.XYTraceSet object

Returns (model, qadict):

model: np.array of same shape as image.pix qadict: dictionary of keywords for QA with min/max per amplifier

Notes

Has hardcoded number of blocks and fibers and typical spacing between blocked tuned to DESI spectrographs.

desispec.preproc.compute_overscan_step(overscan_col, median_size=7, edge_margin=50)[source]

Compute the overscan step score ‘OSTEP’ from an array of overscan values averaged per CCD row

Parameters:

overscan_col – 1D numpy.array

Options:

median_size (int): window size for median pre-filter of overscan_col edge_margin (int): ignore this number of rows at the CCD edges

Returns:

OSTEP value (float scalar)

desispec.preproc.find_overscan_cosmic_trails(rawimage, ov_col, overscan_values, col_width=300, threshold=25000.0, smooth=100)[source]

Find overscan columns that might be impacted by a trail from bright cosmic

Parameters:
  • rawimage – numpy 2D array of raw image

  • ov_col – tuple(yslice, xslice) from parse_sec_keyword(‘BIASSECx’) defining overscan region

Options:

col_width: number of pixels from overscan region to consider threshold: ADU threshold for what might cause a problematic trail smooth: median filter smoothing scale

Returns (badrows, active_col_val) where badrows is a boolean array of whether each row is bad or not, and active_col_val is an array of column-summed and row median-filtered from the active region of the CCD next to the overscan region.

desispec.preproc.get_amp_ids(header)[source]

Return list of amp names based upon header keywords

desispec.preproc.get_calibration_image(cfinder, keyword, entry, header=None)[source]

Reads a calibration file

Parameters:
  • cfinder – None or CalibFinder object

  • keyword – BIAS, MASK, or PIXFLAT

  • entry – boolean or filename or image if entry==False return False if entry==True use calibration filename from calib. config and read it if entry==str use this for the filename if entry==image return input

Options:

header : if not None, update header[‘CAL…’] = calib provenance

Returns:

2D numpy array with calibration image

For the case of keyword=’BIAS’, check for nightly bias before using default bias in $DESI_SPECTRO_CALIB

desispec.preproc.get_readout_mode(header)[source]

Derive CCD readout mode from CCD header

Parameters:

header – dict-like FITS header object with BIASSEC keywrds

Returns “4Amp”, “2AmpLeftRight”, or “2AmpUpDown”

“4Amp” means all 4 amps (ABCD) were used for CCD readout; “2AmpLeftRight” means 1 left amp (AC) and 1 right amp (BD) were used; “2AmpUpDown” means 1 upper amp (CD) and one lower (AB) were used.

desispec.preproc.numba_mean(image_flux, image_ivar, x, hw=3)

Returns mean of pixels vs. row about x+-hw

Parameters:
  • image_flux – 2D array of CCD image pixels

  • image_ivar – 2D array of inverse variance of image_flux

  • x – 1D array of x location per row, len(x) = image_flux.shape[0]

Options:

hw (int): halfwidth over which to average

Returns (flux, ivar) 1D arrays with weighted mean and inverse variance of pixels[i, int(x-hw):int(x+hw)+1] per row i

desispec.preproc.parse_sec_keyword(value)[source]

parse keywords like BIASSECB=’[7:56,51:4146]’ into python slices

python and FITS have almost opposite conventions,
  • FITS 1-indexed vs. python 0-indexed

  • FITS upperlimit-inclusive vs. python upperlimit-exclusive

  • FITS[x,y] vs. python[y,x]

i.e. BIASSEC2=’[7:56,51:4146]’ -> (slice(50,4146), slice(6,56))

desispec.preproc.preproc(rawimage, header, primary_header, bias=True, dark=True, pixflat=True, mask=True, bkgsub_dark=False, nocosmic=False, cosmics_nsig=6, cosmics_cfudge=3.0, cosmics_c2fudge=None, ccd_calibration_filename=None, nocrosstalk=False, nogain=False, overscan_per_row=False, use_overscan_row=False, use_savgol=None, nodarktrail=False, remove_scattered_light=False, psf_filename=None, bias_img=None, model_variance=False, no_traceshift=False, bkgsub_science=False, keep_overscan_cols=False, no_overscan_per_row=False, no_ccd_region_mask=False)[source]

preprocess image using metadata in header

image = ((rawimage-bias-overscan)*gain)/pixflat

Parameters:
  • rawimage – 2D numpy array directly from raw data file

  • header – dict-like metadata, e.g. from FITS header, with keywords CAMERA, BIASSECx, DATASECx, CCDSECx where x = A, B, C, D for each of the 4 amplifiers (also supports old naming convention 1, 2, 3, 4).

  • primary_header – dict-like metadata fit keywords EXPTIME, DOSVER DATE-OBS is also required if bias, pixflat, or mask=True

Optional bias, pixflat, and mask can each be:

False: don’t apply that step True: use default calibration data for that night ndarray: use that array filename (str or unicode): read HDU 0 and use that

Optional overscan features:
overscan_per_rowbool, Subtract the overscan_col values

row by row from the data.

use_overscan_rowbool, Subtract off the overscan_row

from the data (default: False). Requires ORSEC in the Header

use_savgolbool, Specify whether to use Savitsky-Golay filter for

the overscan. (default: False). Requires use_overscan_row=True to have any effect.

Optional variance model if model_variance=True Optional background subtraction with median filtering accross the whole CCD if bkgsub_dark=True Optional background subtraction with median filtering between groups of fiber traces if bkgsub_science=True

Optional disabling of cosmic ray rejection if nocosmic=True Optional disabling of dark trail correction if nodarktrail=True

Optional bias image (testing only) may be provided by bias_img=

Optional tuning of cosmic ray rejection parameters:

cosmics_nsig: number of sigma above background required cosmics_cfudge: number of sigma inconsistent with PSF required cosmics_c2fudge: fudge factor applied to PSF (default is 0.5, set in code)

Optional fit and subtraction of scattered light

Optional disabling of overscan subtraction per row if no_overscan_per_row=True

Returns Image object with member variables:

pix : 2D preprocessed image in units of electrons per pixel ivar : 2D inverse variance of image mask : 2D mask of image (0=good) readnoise : 2D per-pixel readnoise of image meta : metadata dictionary TODO: define what keywords are included

preprocessing includes the following steps:
  • bias image subtraction

  • overscan subtraction (from BIASSEC* keyword defined regions)

  • readnoise estimation (from BIASSEC* keyword defined regions)

  • gain correction (from GAIN* keywords)

  • pixel flat correction

  • cosmic ray masking

  • propagation of input known bad pixel mask

  • inverse variance estimation

Notes:

The bias image is subtracted before any other calculation to remove any non-uniformities in the overscan regions prior to calculating overscan levels and readnoise.

The readnoise is an image not just one number per amp, because the pixflat image also affects the interpreted readnoise.

The inverse variance is estimated from the readnoise and the image itself, and thus is biased.

desispec.preproc.read_bias(filename=None, camera=None, dateobs=None)[source]

Return calibration bias filename for camera on dateobs or night

Options:

filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

Notes

must provide filename, or both camera and dateobs

desispec.preproc.read_dark(filename=None, camera=None, dateobs=None, exptime=None)[source]

Return dark current 2D image ( accounting for exposure time)

Parameters:
  • filename – input filename to read

  • camera – e.g. ‘b0’, ‘r1’, ‘z9’

  • dateobs – DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

  • exptime – the exposure time of the image in seconds

Notes

must provide filename

desispec.preproc.read_mask(filename=None, camera=None, dateobs=None)[source]

Read bad pixel mask image for camera on dateobs.

Options:

filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

Notes

must provide filename, or both camera and dateobs

desispec.preproc.read_pixflat(filename=None, camera=None, dateobs=None)[source]

Read calibration pixflat image for camera on dateobs.

Options:

filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

Notes

must provide filename, or both camera and dateobs

desispec.preproc.subtract_peramp_overscan(image, hdr)[source]

Subtract per-amp overscan using BIASSEC* keywords

Parameters:
  • image – 2D image array, modified in-place

  • hdr – FITS header with BIASSEC[ABCD] or BIASSEC[1234] keywords

Note: currently used in desispec.ccdcalib.compute_bias_file to model bias image, but not preproc itself (which subtracts that bias, and has more complex support for row-by-row, col-overscan, etc.)

desispec.qa

desispec.qa.html

Module for generating QA HTML.

desispec.qa.html.calib(qaprod_dir=None, specprod_dir=None)[source]

Generate HTML to orgainze calib HTML

desispec.qa.html.calib_exp(night, expid, qaprod_dir=None)[source]

Geneate HTML for calib exposure PNGs :param night: :param expid:

Returns:

desispec.qa.html.finish(f, body, links=None)[source]

Fill in the HTML file and end it :param f: :type f: file :param body: :type body: str :param links: :type links: str, optional

desispec.qa.html.header(title)[source]
Parameters:

title (str, optional) –

desispec.qa.html.make_exposure(night, expid, qaprod_dir=None)[source]

Generate HTML for exposure PNGs

Parameters:
  • setup (str) –

  • cbset (str) –

  • det (int) –

Returns:

  • links (str)

  • body (str)

desispec.qa.html.make_exposures(qaprod_dir=None)[source]

Generate HTML to organize exposure HTML

Returns:

  • links (str)

  • body (str)

desispec.qa.html.toplevel(qaprod_dir=None)[source]

Generate HTML to top level QA Mainly generates the highest level HTML file which has links to the Exposure and Calib QA.

This also slurps any .png files in the top-level

Parameters:
  • setup (str) –

  • cbset (str) –

  • det (int) –

Returns:

  • links (str)

  • body (str)

desispec.qa.qa_brick

Classes to organize and execute QA for a DESI exposure.

desispec.qa.qa_exposure

Classes to organize and execute QA for a DESI exposure.

desispec.qa.qa_frame

Classes to organize and execute QA for a DESI exposure.

desispec.qa.qa_frame.qaframe_from_frame(frame_file, specprod_dir=None, make_plots=False, qaprod_dir=None, output_dir=None, clobber=True)[source]

Generate a qaframe object from an input frame_file name (and night)

Write QA to disk Will also make plots if directed :param frame_file: str :param specprod_dir: str, optional :param qa_dir: str, optional – Location of QA :param make_plots: bool, optional :param output_dir: str, optional

Returns:

desispec.qa.qa_multiexp

Class to organize QA for multiple exposures. Likely to only be used as parent of QA_Night or QA_Prod.

desispec.qa.qa_night

Class to organize QA for one night of DESI exposures.

desispec.qa.qa_plots

Module for QA plots.

desispec.qa.qa_plots.brick_redrock(outfil, zf, qabrick)[source]

QA plots for redrock fits

Parameters:
  • outfil

  • qabrick

  • zf – ZfindBase object

Returns:

Stuff?

desispec.qa.qa_plots.exposure_fiberflat(channel, expid, metric, outfile=None)[source]

Generate an Exposure level plot of a FiberFlat metric :param channel: str, e.g. ‘b’, ‘r’, ‘z’ :param expid: int :param metric: str, allowed entires are: [‘meanflux’]

Returns:

desispec.qa.qa_plots.exposure_fluxcalib(outfil, qa_data)[source]

QA plots for Flux calibration in an Exposure

Parameters:
  • outfil – str – Name of PDF file

  • qa_data – dict – QA data, including that of the individual frames

desispec.qa.qa_plots.exposure_map(x, y, metric, mlbl=None, outfile=None, title=None, ax=None, fig=None, psz=9.0, cmap=None, vmnx=None)[source]

Generic method used to generated Exposure level QA One channel at a time

Parameters:
  • x – list or ndarray

  • y – list or ndarray

  • metric – list or ndarray

  • mlbl – str, optional

  • outfile – str, optional

  • title – str, optional

desispec.qa.qa_plots.exposure_s2n(qa_exp, metric, outfile='exposure_s2n.png', verbose=True, specprod_dir=None)[source]

Generate an Exposure level plot of a S/N metric :param qa_exp: QA_Exposure :param metric: str, allowed entires are: [‘resid’] :param specprod_dir: str, optional

Returns:

desispec.qa.qa_plots.frame_fiberflat(outfil, qaframe, frame, fiberflat)[source]

QA plots for fiber flat

Parameters:
  • outfil

  • qaframe

  • frame

  • fiberflat

  • clobber – bool, optional

Returns:

Stuff?

desispec.qa.qa_plots.frame_fluxcalib(outfil, qaframe, frame, fluxcalib)[source]

QA plots for Flux calibration in a Frame

Parameters:
  • outfil – str, name of output file

  • qaframe – dict containing QA info

  • frame – frame object containing extraction of standard stars

  • fluxcalib – fluxcalib object containing flux calibration

Returns:

desispec.qa.qa_plots.frame_s2n(s2n_dict, outfile, rescut=0.2, verbose=True)[source]

Plot S/N diagnostics for a given frame Replaces a previous-QL script

Parameters:
  • s2n_dict (dict) – dictionary of qa outputs repackaged a bit

  • outfile (str) – output png filename

  • rescut (float, optional) – only plot residuals (+/-) less than rescut

desispec.qa.qa_plots.frame_skyres(outfil, frame, skymodel, qaframe, quick_look=False)[source]

Generate QA plots and files for sky residuals of a given frame

Parameters:
  • outfil (str) – Name of output file

  • frame (Frame object) –

  • skymodel (SkyModel object) –

  • qaframe (QAFrame object) –

desispec.qa.qa_plots.get_channel_clrs()[source]

Simple dict to organize styles for channels :returns: dict :rtype: channel_dict

desispec.qa.qa_plots.get_sty_otype()[source]

Styles for plots

desispec.qa.qa_plots.prod_ZP(qa_prod, outfile=None, channels=('b', 'r', 'z'), xaxis='MJD')[source]

Generate a plot summarizing the ZP for a production

Parameters:
  • qa_prod – QA_Prod object

  • outfile – str, optional Output file name

  • channels – tuple, optional List of channels to show

  • xaxis – str, optional Designate x-axis. Options are: MJD, expid, texp

Returns:

desispec.qa.qa_plots.prod_avg_s2n(qa_prod, outfile=None, optypes=['ELG'], xaxis='MJD', fiducials=None)[source]

Generate a plot summarizing average S/N in a production for a few object types in a few cameras

Parameters:
  • qa_prod – QA_Prod object

  • outfile – str, optional Output file name

  • optypes – list, optional List of fiducial objects to show Options are: ELG, LRG, QSO

  • xaxis – str, optional Designate x-axis. Options are: MJD, expid, texp

  • fiducials

Returns:

desispec.qa.qa_plots.prod_channel_hist(qa_prod, qatype, metric, xlim=None, outfile=None, pp=None, close=True)[source]

Generate a series of histrograms (one per channel)

Parameters:
  • qa_prod – QA_Prod class

  • qatype – str

  • metric – str

  • xlim – tuple, optional

  • outfile – str, optional

  • pp – PdfPages, optional

  • close – bool, optional

Returns:

desispec.qa.qa_plots.prod_time_series(qa_multi, qatype, metric, outfile=None, close=True, pp=None, bright_dark=0, exposures=False, night=None, horiz_line=None)[source]

Generate a time series plot for a production Can be MJD or Exposure number

Parameters:
  • qa_multi – QA_Prod or QA_Night

  • qatype – str

  • metric – str

  • outfile – str, optional

  • close – bool, optional

  • pp

  • bright_dark – int, optional; (flag: 0=all; 1=bright; 2=dark)

  • night – str, optional Only used for the Title

  • horiz_line – float, optional Draw a horizontal line at input value

desispec.qa.qa_plots.show_meta(ax, qaframe, qaflavor, outfil)[source]

Show meta data on the figure

Parameters:
  • ax – matplotlib.ax

  • qaframe – QA_Frame

  • qaflavor – str

Returns:

desispec.qa.qa_plots.skyline_resid(channel, sky_wave, sky_flux, sky_res, sky_ivar, outfile=None, pp=None, close=True, dpi=700)[source]

QA plot for residuals on sky lines ala Julien Guy :param sky_wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close: :param nslices: :param dpi:

Returns:

desispec.qa.qa_plots.skysub_gauss(sky_wave, sky_flux, sky_res, sky_ivar, outfile=None, pp=None, close=True, binsz=0.1, dpi=700, nfbin=4)[source]

Generate a plot examining the Gaussianity of the residuals Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param sky_ivar: :param outfile: :param pp: :param close:

Returns:

desispec.qa.qa_plots.skysub_resid_dual(sky_wave, sky_flux, sky_res, outfile=None, pp=None, close=True, nslices=20, dpi=700)[source]

Generate a plot of sky subtraction residuals Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close:

Returns:

desispec.qa.qa_plots.skysub_resid_series(sky_dict, xtype, outfile=None, pp=None, close=True, nslices=20, dpi=700)[source]

Generate a plot of sky subtraction residuals for a series of inputs Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close:

Returns:

desispec.qa.qa_plots_ql

This includes routines to make pdf plots on the qa outputs from quicklook.

For information on QA dictionaries used here as input, visit wiki page: https://desi.lbl.gov/trac/wiki/Pipeline/QuickLook/QuicklookQAOutputs/Science

desispec.qa.qa_plots_ql.plot_RMS(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot RMS

Parameters:
  • qa_dict – dictionary of qa outputs from running qa_quicklook.Get_RMS

  • outfile – Name of plot output file

desispec.qa.qa_plots_ql.plot_SNR(qa_dict, outfile, objlist, fitsnr, rescut=0.2, sigmacut=2.0, plotconf=None, hardplots=False)[source]

Plot SNR

Parameters:
  • qa_dict – dictionary of qa outputs from running qa_quicklook.Calculate_SNR

  • outfile – output png file

  • objlist – list of objtype for log(snr**2) vs. mag plots

  • badfibs – list of fibers with infs or nans to remove for plotting

  • fitsnr – list of snr vs. mag fitting coefficients # JXP – THIS IS NOT TRUE!!

  • rescut – only plot residuals (+/-) less than rescut (default 0.2)

  • sigmacut – only plot residuals (+/-) less than sigma cut (default 2.0)

  • NOTE – rescut taken as default cut parameter

desispec.qa.qa_plots_ql.plot_XWSigma(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot XWSigma

Parameters:
  • qa_dict – qa dictionary from countpix qa

  • outfile – file of the plot

desispec.qa.qa_plots_ql.plot_bias_overscan(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Map of bias from overscan from 4 regions of CCD

Parameters:
  • qa_dict – qa dictionary from bias_from_overscan qa

  • outfile – pdf file of the plot

desispec.qa.qa_plots_ql.plot_countpix(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot pixel counts above some threshold

Parameters:
  • qa_dict – qa dictionary from countpix qa

  • outfile – pdf file of the plot

desispec.qa.qa_plots_ql.plot_countspectralbins(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot count spectral bins.

Parameters:
  • qa_dict – dictionary of qa outputs from running qa_quicklook.CountSpectralBins

  • outfile – Name of figure.

desispec.qa.qa_plots_ql.plot_lpolyhist(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot histogram for each legendre polynomial coefficient in WSIGMA array.

Parameters:
  • qa_dict – Dictionary of qa outputs from running qa_quicklook.Check_Resolution

  • outfile – Name of figure.

desispec.qa.qa_plots_ql.plot_sky_continuum(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot mean sky continuum from lower and higher wavelength range for each fiber and accross amps.

Parameters:
  • qa_dict – dictionary from sky continuum QA

  • outfile – pdf file to save the plot

desispec.qa.qa_plots_ql.plot_sky_peaks(qa_dict, outfile, plotconf=None, hardplots=False)[source]

Plot rms of sky peaks for smy fibers across amps

Parameters:
  • qa_dict – dictionary from sky peaks QA

  • outfile – pdf file to save the plot

desispec.qa.qa_prod

Class to organize QA for a full DESI production run.

desispec.qa.qa_quicklook

Monitoring algorithms for Quicklook pipeline.

class desispec.qa.qa_quicklook.Bias_From_Overscan(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Calc_XWSigma(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Calculate_SNR(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Check_FiberFlat(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Check_HDUs(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Check_Resolution(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.CountSpectralBins(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Count_Pixels(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Get_RMS(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Integrate_Spec(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Sky_Continuum(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Sky_Peaks(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Sky_Rband(name, config, logger=None)[source]
class desispec.qa.qa_quicklook.Sky_Residual(name, config, logger=None)[source]

Use offline sky_residual function to calculate sky residuals

class desispec.qa.qa_quicklook.Trace_Shifts(name, config, logger=None)[source]
desispec.qa.qa_quicklook.get_frame(filetype, night, expid, camera, specdir)[source]

Make frame object from file if in development mode

desispec.qa.qa_quicklook.get_image(filetype, night, expid, camera, specdir)[source]

Make image object from file if in development mode

desispec.qa.qa_quicklook.get_inputs(*args, **kwargs)[source]

Get inputs required for each QA

desispec.qa.qalib

Simple low level library functions for offline and online QAs.

desispec.qa.qalib.SN_ratio(flux, ivar)[source]

SN Ratio median snr for the spectra, flux should be sky subtracted.

Parameters:
  • flux (array) – 2d [nspec,nwave] the signal (typically for spectra, this comes from frame object

  • ivar (array) – 2d [nspec,nwave] corresponding inverse variance

Returns:

1d [nspec]

Return type:

medsnr (array)

desispec.qa.qalib.SignalVsNoise(frame, params, fidboundary=None)[source]

Signal vs. Noise

Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera.

Parameters:
  • flux (array) – 2d [nspec,nwave] the signal (typically for spectra, this comes from frame object

  • ivar (array) – 2d [nspec,nwave] corresponding inverse variance

  • fidboundary – list of slices indicating where to select in fiber and wavelength directions for each amp (output of slice_fidboundary function)

desispec.qa.qalib._get_mags(frame)[source]

Extract frame.fibermap fluxes into mags depending upon camera

Parameters:

frame – Frame object

Returns array of magnitudes, using 99.0 when flux<0

b camera frames return g-band magnitudes; r camera -> r-mags; z camera -> z-mags

desispec.qa.qalib.ampregion(image)[source]

Get the pixel boundary regions for amps

Parameters:

image – desispec.image.Image object

desispec.qa.qalib.continuum(wave, flux, wmin=None, wmax=None)[source]

Find the median continuum of the spectrum inside a wavelength region.

Parameters:
  • wave – 1d wavelength array

  • flux – 1d counts/flux array

  • wmax (wmin and) – region to consider for the continuum

desispec.qa.qalib.countbins(flux, threshold=0)[source]

Count the number of bins above a given threshold on each fiber

Parameters:
  • flux – 2d (nspec,nwave)

  • threshold – threshold counts

desispec.qa.qalib.countpix(image, nsig=None)[source]

Count the pixels above a given threshold in units of sigma.

Parameters:
  • image – 2d image array

  • nsig – threshold in units of sigma, e.g 2 for 2 sigma

desispec.qa.qalib.fiducialregion(frame, psf)[source]

Get the fiducial amplifier regions on the CCD pixel to fiber by wavelength space

Parameters:
  • frame – desispec.frame.Frame object

  • psf – desispec.psf.PSF like object

desispec.qa.qalib.gauss(x, a, mu, sigma)[source]

Gaussian fit of input data

desispec.qa.qalib.getrms(image)[source]

Calculate the rms of the pixel values)

Parameters:

image – 2d array

desispec.qa.qalib.integrate_spec(wave, flux)[source]

Calculate the integral of the spectrum in the given range using trapezoidal integration

Note: limits of integration are min and max values of wavelength

Parameters:
  • wave – 1d wavelength array

  • flux – 1d flux array

desispec.qa.qalib.orig_SNRFit(frame, night, camera, expid, params, fidboundary=None, offline=False)[source]

Signal vs. Noise With fitting

Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera. then fit the log(snr)=a+b*mag or log(snr)=poly(mag)

see http://arXiv.org/abs/0706.1062v2 for proper fitting of power-law distributions it is not implemented here!

qadict has the following data model

“MAGNITUDES” : ndarray - Depends on camera (DECAM_G, DECAM_R, DECAM_Z) “MEDIAN_SNR” : ndarray (nfiber) “NUM_NEGATIVE_SNR” : int “SNR_MAG_TGT” “FITCOEFF_TGT” : list “SNR_RESID” : list, can be trimmed down during the fitting “FIDSNR_TGT” “RA” : ndarray (nfiber) “DEC” : ndarray (nfiber) “OBJLIST” : list - Save a copy to make sense of the list order later “EXPTIME” : float “FIT_FILTER” : str “r2” : float - Fitting parameter

Parameters:
  • frame – desispec.Frame object

  • night

  • camera

  • expid – int

  • params – parameters dictionary

  • { – “Func”: “linear”, # Fit function type one of [“linear”,”poly”,”astro”] “FIDMAG”: 22.0, # magnitude to evaluate the fit “Filter”:”DECAM_R”, #filter name

  • }

  • fidboundary – list of slices indicating where to select in fiber and wavelength directions for each amp (output of slice_fidboundary function)

  • offline – bool, optional If True, save things differently for offline

Returns:

dict

Return type:

qadict

desispec.qa.qalib.s2n_flux_astro(flux, A, B)[source]

Function for a normalized (by texp**1/2) curve to flux vs S/N

Parameters:
  • flux (float or np.ndarray) – Flux value(s)

  • A (float) – Scale coefficient

  • B (float) – Offset coefficient

Returns:

S/N at the input flux

desispec.qa.qalib.s2n_funcs(exptime=None)[source]

Functions for fitting S/N

Parameters:

exptime – float, optional

Returns:

dict

Return type:

funcMap

desispec.qa.qalib.s2nfit(frame, camera, params)[source]

Signal vs. Noise With fitting

Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera. then fit snr=A*mag/sqrt(A*mag+B)

see http://arXiv.org/abs/0706.1062v2 for proper fitting of power-law distributions it is not implemented here!

Instead we use scipy.optimize.curve_fit

Parameters:
  • frame – desispec.Frame object

  • camera – str, name of the camera

  • params – parameters dictionary for S/N

Returns:

dict

MEDIAN_SNR (ndarray, nfiber): Median S/N of light in each fiber FIT_FILTER (str): Filter used for the fluxes EXPTIME (float): Exposure time XXX_FIBERID (list): Fibers matching ELG, LRG, BGS, etc. SNR_MAG_TGT (list): List of lists with S/N and mag of ELG, LRG, BGS, etc. FITCOEFF_TGT (list): List of fitted coefficients. Junk fits have np.nan OBJLIST (list): List of object types analyzed (1 or more fiber)

Return type:

qadict

desispec.qa.qalib.sky_continuum(frame, wrange1, wrange2)[source]

QA Algorithm for sky continuum.

To be called from desispec.sky.qa_skysub and desispec.qa.qa_quicklook.Sky_Continuum.run_qa

Parameters:
  • frame

  • wrange1

  • wrange2

Returns:

skyfiber, contfiberlow, contfiberhigh, meancontfiber, skycont

desispec.qa.qalib.sky_resid(param, frame, skymodel, quick_look=False)[source]

QA Algorithm for sky residual To be called from desispec.sky.qa_skysub and desispec.qa.qa_quicklook.Sky_residual.run_qa :param param: dict of QA parameters :param frame: desispec.Frame object after sky subtraction :param skymodel: desispec.SkyModel object

Returns a qa dictionary for sky resid

desispec.qa.qalib.slice_fidboundary(frame, leftmax, rightmin, bottommax, topmin)[source]

leftmax,rightmin,bottommax,topmin - Indices in spec-wavelength space for different amps (e.g output from fiducialregion function) #- This could be merged to fiducialregion function

Returns (list):

list of tuples of slices for spec- wavelength boundary for the amps.

desispec.qa.utils

Module for QA support.

desispec.qa.utils.get_skyres(cframes, sub_sky=False, flatten=True)[source]
Parameters:
  • cframes – str or list Single cframe or a list of them

  • sub_sky – bool, optional Subtract the sky? This should probably not be done

  • flatten – bool, optional Return a flat, 1D array for each variable

  • combine – bool, optional combine the individual sky fibers? Median ‘smash’

Returns:

ndarray flux : ndarray res : ndarray ivar : ndarray

Return type:

wave

desispec.qproc

desispec.qproc.flavor

desispec.qproc.flavor.check_qframe_flavor(qframe, input_flavor=None)[source]

Tool to check the flavor of a qframe

Parameters:

qframe – DESI QFrame object

Optional:

input_flavor

Returns:

flavor string

desispec.qproc.io

I/O routines for qproc objects

desispec.qproc.io.read_qframe(filename, nspec=None, skip_resolution=False)[source]

Reads a frame fits file and returns its data.

Parameters:
  • filename – path to a file

  • skip_resolution – bool, option Speed up read time (>5x) by avoiding the Resolution matrix

Returns:

desispec.Frame object with attributes wave, flux, ivar, etc.

desispec.qproc.io.write_qframe(outfile, qframe, header=None, fibermap=None, units=None)[source]

Write a frame fits file and returns path to file written.

Parameters:
  • outfile – full path to output file, or tuple (night, expid, channel)

  • qframe – desispec.qproc.QFrame object with wave, flux, ivar…

Optional:

header: astropy.io.fits.Header or dict to override frame.header fibermap: table to store as FIBERMAP HDU

Returns:

full filepath of output file that was written

Note

to create a QFrame object to pass into write_qframe, qframe = QFrame(wave, flux, ivar)

desispec.qproc.qarc

desispec.qproc.qarc.process_arc(qframe, xytraceset, linelist=None, npoly=2, nbins=2)[source]

qframe: desispec.qframe.QFrame object xytraceset : desispec.xytraceset.XYTraceSet object linelist: line list to fit npoly: polynomial order for sigma expansion nbins: no of bins for the half of the fitting window return: xytraceset (with ysig vs wave)

desispec.qproc.qarc.sigmas_from_arc(wave, flux, ivar, linelist, n=2)[source]

Gaussian fitting of listed arc lines and return corresponding sigmas in pixel units Args: linelist: list of lines (A) for which fit is to be done n: fit region half width (in bin units): n=2 bins => (2*n+1)=5 bins fitting window.

desispec.qproc.qextract

desispec.qproc.qextract.qproc_boxcar_extraction(xytraceset, image, fibers=None, width=7, fibermap=None, save_sigma=True)[source]

Fast boxcar extraction of spectra from a preprocessed image and a trace set

Parameters:
  • xytraceset – DESI XYTraceSet object

  • image – DESI preprocessed Image object

Optional:

fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : extraction boxcar width, default is 7 fibermap : table

Returns:

QFrame object

desispec.qproc.qfiberflat

desispec.qproc.qfiberflat.qproc_apply_fiberflat(qframe, fiberflat, return_flat=False)[source]

Apply a fiber flat to a qframe.

Inputs:

qframe: desispec.qproc.qframe.QFrame object which will be modified fiberflat: desispec.fiberflat.FiberFlat object with the flat to apply

Optional:

return_flat : if True, returns the flat field that has been applied

Returns nothing or the flat that has been applied.

desispec.qproc.qfiberflat.qproc_compute_fiberflat(qframe, niter_meanspec=4, nsig_clipping=3.0, spline_res_clipping=20.0, spline_res_flat=5.0)[source]

Fast estimation of fiberflat

desispec.qproc.qframe

Lightweight wrapper class for spectra extract with qextract, row-by-row extraction (boxcar or profile)

desispec.qproc.qsky

desispec.qproc.qsky.qproc_sky_subtraction(qframe, return_skymodel=False)[source]

Fast sky subtraction directly applied to the input qframe.

Args:

qframe : DESI QFrame object

Optional:

return_skymodel returns the skymodel as an array of same shape as qframe.flux

desispec.qproc.rowbyrowextract

Row-by-row extraction routines (Horne 1986).

desispec.qproc.rowbyrowextract.build_atcinva(atcinva, xstart, xend, profiles, ivar)

Build A^T C^-1 A for the row-by-row extraction.

Modifies input atcinva in place. Assumes spectra are ordered, so xstart[i+1] > xstart[i].

Parameters:
  • atcinvb (np.ndarray) – matrix to fill in (modified in place)

  • xstart (int) – x coordinate of start of each profile

  • xend (int) – x coordinate of end of each profile

  • profiles (np.ndarray) – cross-dispersion profile for each trace

  • ivar (np.ndarray) – ivar for appropriate row of image

desispec.qproc.rowbyrowextract.build_atcinvb(atcinvb, xstart, profiles, data)

Build A^T C^-1 b for the row-by-row extraction.

Modifies input atcinvb in place.

Parameters:
  • atcinvb (np.ndarray) – matrix to fill in (modified in place)

  • xstart (int) – x coordinate of start of each profile

  • profiles (np.ndarray) – cross-dispersion profile for each trace

  • data (np.ndarray) – flux * ivar for appropriate row of image

desispec.qproc.rowbyrowextract.extract(image, psf, blocksize=25, fibermap=None, nspec=500, return_model=False, tails=False)[source]

Extract spectra from image using an optimal row-by-row extraction.

The algorithm extracts the spectra row-by-row, finding the fluxes associated with the 1D PSF that best explain the data.

This only works for Gauss-Hermite PSFs at present, since it does some trickery to efficiently get 1D PSFs for large groups of wavelengths and spectra.

Parameters:
  • image (Image) – image from which to extract the spectra.

  • psf (specter.psf.gausshermite.GaussHermitePSF) – PSF to use

  • blocksize (int) – number of spectra to fit simultaneously. Spectra in different blocks will be fit independently.

  • fibermap (astropy.Table) – fibermap for image

  • nspec (int) – total number of spectra to extract

  • return_model (bool) – if True, return also the model of the image, the profiles, and the locations in the image corresponding to the profiles.

  • tails (bool) – include PSF tails in the modeling. These are usually identically zero for DESI and so default to off.

Returns:

frame – The 1D extracted spectra as a QFrame

Return type:

desispec.qproc.qframe.QFrame

desispec.qproc.rowbyrowextract.model(frame, profile, profilepix, shape)[source]

Create model image for frame.

Parameters:
  • frame (desispec.qproc.qframe.QFrame) – frame for which to create model

  • profile (np.ndarray) – cross-dispersion profile at each row for each spectrum

  • profilepix (np.ndarray) – x-coordinates in image corresponding to each entry in profile

  • shape ((int, int)) – shape of output image

desispec.qproc.rowbyrowextract.onedprofile_gaussian(psf, ypix, ispec, nsig=4)[source]

Compute the cross-dispersion profile for a Gaussian.

This was warm-up code for the Gauss-Hermite implementation below, but may have independent value? It will work with any specter PSF, albeit using a Gaussian approximation to the cross-dispersion profile.

Parameters:
  • psf (specter.psf.PSF) – psf

  • ypix (np.ndarray) – rows for which to compute profile

  • ispec (np.ndarray) – indices of spectra for which to compute profiles

  • nsig (np.ndarray) – how many sigma out to compute the Gaussian; profile is treated as zero beyond roughly here.

Returns:

  • profilepix, profile, lambda

  • profilepix (np.ndarray) – the x-coordinates corresponding to profile on the detector

  • profile (np.ndarray) – the cross dispersion profiles

  • lambda (np.ndarray) – the wavelengths of the spectra at the requested locations

desispec.qproc.rowbyrowextract.onedprofile_gh(psf, ypix, ispec, tails=True)[source]

Compute cross-dispersion profile for Gauss-Hermite PSFs

Parameters:
  • psf (specter.psf.gausshermite.GaussHermitePSF) – psf

  • ypix (np.ndarray) – rows for which to compute profile

  • ispec (np.ndarray) – indices of spectra for which to compute profiles

Returns:

  • profilepix, profile, lambda

  • profilepix (np.ndarray) – the x-coordinates corresponding to profile on the detector

  • profile (np.ndarray) – the cross dispersion profiles

  • lambda (np.ndarray) – the wavelengths of the spectra at the requested locations

desispec.qproc.rowbyrowextract.pgh(x, m=0, xc=0.0, sigma=1.0)

Pixel-integrated (probabilist) Gauss-Hermite function. :param x: pixel-center baseline array :param m: order of Hermite polynomial multiplying Gaussian core :param xc: sub-pixel position of Gaussian centroid relative to x baseline :param sigma: sigma parameter of Gaussian core in units of pixels

Uses the relationship Integral{ H_k(x) exp(-0.5 x^2) dx} = -H_{k-1}(x) exp(-0.5 x^2) + const Written: Adam S. Bolton, U. of Utah, fall 2010 Adapted for efficiency by S. Bailey while dropping generality modified from the orig _pgh to enable jit-compiling –> no longer passing in self, calling custom numba functions in util E. Schlafly added some broadcasting bits to make it play more nicely with large arrays. This should probably go to specter instead.

desispec.qproc.rowbyrowextract.tail1d(core, xx, index)[source]

Compute the cross-dispersion profile of the tail model.

Parameters:
  • core (np.ndarray) – core scale (pixels)

  • xx (np.ndarray) – distance in x from center

  • index (np.ndarray) – power law index of tail model

Returns:

  • np.ndarray

  • tail amplitude at given location

desispec.quicklook

desispec.quicklook.arcprocess

desispec.quicklook.arcprocess.process_arc(frame, linelist=None, npoly=2, nbins=2, domain=None)[source]

frame: desispec.frame.Frame object, preumably resolution not evaluated. linelist: line list to fit npoly: polynomial order for sigma expansion nbins: no of bins for the half of the fitting window return: coefficients of the polynomial expansion

desispec.quicklook.arcprocess.sigmas_from_arc(wave, flux, ivar, linelist, n=2)[source]

Gaussian fitting of listed arc lines and return corresponding sigmas in pixel units Args: linelist: list of lines (A) for which fit is to be done n: fit region half width (in bin units): n=2 bins => (2*n+1)=5 bins fitting window.

desispec.quicklook.arcprocess.write_psffile(infile, wcoeffs, wcoeffs_wavemin, wcoeffs_wavemax, outfile, wavestepsize=None)[source]

extract psf file, add wcoeffs, and make a new psf file preserving the traces etc. psf module will load this

desispec.quicklook.merger

A class to merge quicklook qa outputs.

desispec.quicklook.palib

Low level functions to be from top level PAs.

desispec.quicklook.palib.apply_flux_calibration(frame, fluxcalib)[source]

Apply flux calibration to sky subtracted qframe Use offline algorithm, but assume qframe object is input and that it is on native ccd wavelength grid Calibration vector is resampled to frame wavelength grid

frame: QFrame object fluxcalib: FluxCalib object

Modifies frame.flux and frame.ivar

desispec.quicklook.palib.get_resolution(wave, nspec, tset, usesigma=False)[source]

Calculates approximate resolution values at given wavelengths in the format that can directly feed resolution data of desispec.frame.Frame object.

wave: wavelength array nsepc: no of spectra (int) tset: desispec.xytraceset like object usesigma: allows to use sigma from psf file for resolution computation.

returns : resolution data (nspec,nband,nwave); nband = 1 for usesigma = False, otherwise nband=21

desispec.quicklook.palib.project(x1, x2)[source]

return a projection matrix so that arrays are related by linear interpolation x1: Array with one binning x2: new binning

Return Pr: x1= Pr.dot(x2) in the overlap region

desispec.quicklook.palib.resample_spec(wave, flux, outwave, ivar=None)[source]

rebinning conserving S/N Algorithm is based on http://www.ast.cam.ac.uk/%7Erfc/vpfit10.2.pdf Appendix: B.1

Args: wave : original wavelength array (expected (but not limited) to be native CCD pixel wavelength grid outwave: new wavelength array: expected (but not limited) to be uniform binning flux : df/dx (Flux per A) sampled at x ivar : ivar in original binning. If not None, ivar in new binning is returned.

Note: Full resolution computation for resampling is expensive for quicklook.

desispec.interpolation.resample_flux using weights by ivar does not conserve total S/N. Tests with arc lines show much narrow spectral profile, thus not giving realistic psf resolutions This algorithm gives the same resolution as obtained for native CCD binning, i.e, resampling has insignificant effect. Details,plots in the arc processing note.

desispec.quicklook.pas

class desispec.quicklook.pas.PipelineAlg(name, inptype, outtype, config, logger=None)[source]

Simple base class for Pipeline algorithms

get_default_config()[source]

return a dictionary of 3-tuples, field 0 is the name of the parameter field 1 is the default value of the parameter field 2 is the comment for human readeable format. Field 2 can be used for QLF to dynamically setup the display

desispec.quicklook.procalgs

Pipeline Preprocessing algorithms for Quicklook.

class desispec.quicklook.procalgs.ApplyFiberFlat(name, config, logger=None)[source]

PA to Apply the fiberflat field to the given frame

class desispec.quicklook.procalgs.ApplyFiberFlat_QL(name, config, logger=None)[source]

PA to Apply the fiberflat field (QL) to the given frame

class desispec.quicklook.procalgs.ApplyFiberFlat_QP(name, config, logger=None)[source]

PA to Apply the fiberflat field (QP) to the given qframe

class desispec.quicklook.procalgs.ApplyFluxCalibration(name, config, logger=None)[source]

PA to apply flux calibration to the given sframe

class desispec.quicklook.procalgs.BoxcarExtract(name, config, logger=None)[source]
do_boxcar(tset, outwave, boxwidth=2.5, nspec=500, maskFile=None, usesigma=False, quick_resolution=False)

Extracts spectra row by row, given the centroids

Parameters:
  • image – desispec.image object

  • tset – desispec.xytraceset like object

  • outwave – wavelength array for the final spectra output

  • boxwidth – HW box size in pixels

  • usesigma – if True, use sigma from psf file (ysigma) to calculate resolution data.

  • quick_resolution – whether to calculate the resolution matrix or use QuickResolution object

Returns flux, ivar, resolution

get_default_config()[source]

return a dictionary of 3-tuples, field 0 is the name of the parameter field 1 is the default value of the parameter field 2 is the comment for human readeable format. Field 2 can be used for QLF to dynamically setup the display

class desispec.quicklook.procalgs.ComputeFiberflat(name, config, logger=None)[source]

PA to compute fiberflat field correction from a DESI continuum lamp frame

class desispec.quicklook.procalgs.ComputeFiberflat_QL(name, config, logger=None)[source]

PA to compute fiberflat field correction from a DESI continuum lamp frame

class desispec.quicklook.procalgs.ComputeFiberflat_QP(name, config, logger=None)[source]
class desispec.quicklook.procalgs.ComputeSky(name, config, logger=None)[source]

PA to compute sky model from a DESI frame

class desispec.quicklook.procalgs.ComputeSky_QL(name, config, logger=None)[source]

PA to compute sky model from a DESI frame

class desispec.quicklook.procalgs.Extract_QP(name, config, logger=None)[source]
get_default_config()[source]

return a dictionary of 3-tuples, field 0 is the name of the parameter field 1 is the default value of the parameter field 2 is the comment for human readeable format. Field 2 can be used for QLF to dynamically setup the display

class desispec.quicklook.procalgs.Extraction_2d(name, config, logger=None)[source]

Offline 2D extraction for offline QuickLook

class desispec.quicklook.procalgs.Flexure(name, config, logger=None)[source]

Use desi_compute_trace_shifts to output modified psf file

class desispec.quicklook.procalgs.Initialize(name, config, logger=None)[source]

This PA takes information from the fibermap and raw header and adds it to the general info section of the merged dictionary

class desispec.quicklook.procalgs.Preproc(name, config, logger=None)[source]
class desispec.quicklook.procalgs.ResolutionFit(name, config, logger=None)[source]

Fitting of Arc lines on extracted arc spectra, polynomial expansion of the fitted sigmas, and updating the coefficients to the new traceset file

class desispec.quicklook.procalgs.SkySub(name, config, logger=None)[source]
class desispec.quicklook.procalgs.SkySub_QL(name, config, logger=None)[source]

This is for QL Sky subtraction. The input frame object should be fiber flat corrected. Unlike offline, if no skymodel file is given as input, a sky compute method is called to create a skymodel object and then subtraction is performed. Outputing that skymodel to a file is optional and can be configured.

class desispec.quicklook.procalgs.SkySub_QP(name, config, logger=None)[source]

Sky subtraction. The input frame object should be fiber flat corrected. No sky model is saved for now

desispec.quicklook.qas

class desispec.quicklook.qas.MonitoringAlg(name, inptype, config, logger=None)[source]

Simple base class for monitoring algorithms

class desispec.quicklook.qas.QASeverity(value)[source]

An enumeration.

desispec.quicklook.ql_plotlib

Generic plotting algorithms for QuickLook QAs.

desispec.quicklook.ql_plotlib.ql_2dplot(ax, xvals, yvals, plottitle, xtitle, ytitle, xlim=None, ylim=None)[source]

Make 2d plot of specific metrics provided in configuration file

Parameters:
  • ax – matplotlib subplot

  • xvals – QA metric to be plotted along the xaxis

  • yvals – QA metric to be plotted along the yaxis

  • plottitle – plot title from configuration file

  • xtitle – x axis label

  • ytitle – y axis label

Optional:

xlim: list containing x range (i.e. [x_lo,x_hi]) ylim: list containing y range (i.e. [y_lo,y_hi])

Returns:

matplotlib sublot containing plotted metrics

desispec.quicklook.ql_plotlib.ql_3dplot(ax, xvals, yvals, zvals, plottitle, xtitle, ytitle, zlim=None, heatmap=None)[source]

Make 3d scatter plot of specific metrics provided in configuration file

Parameters:
  • ax – matplotlib subplot

  • xvals – QA metric to be plotted along the xaxis

  • yvals – QA metric to be plotted along the yaxis

  • zvals – QA metric to be plotted

  • plottitle – plot title from configuration file

  • xtitle – x axis label

  • ytitle – y axis label

Optional:

zlim: list containing scatter plot range (i.e. [z_lo,z_hi])

Returns:

matplotlib sublot containing plotted metrics

desispec.quicklook.ql_plotlib.ql_patchplot(ax, vals, plottitle, grid, heatmap=None)[source]

Make patch plot of specific metrics provided in configuration file

Parameters:
  • ax – matplotlib subplot

  • vals – QA metric to be plotted

  • plottitle – plot title from configuration file

  • grid – shape of patch plot

Optional:

heat: specify color of heatmap (must conform to matplotlib)

Returns:

matplotlib sublot containing plotted metrics

desispec.quicklook.ql_plotlib.ql_qaplot(fig, plotconf, qadict, camera, expid, outfile)[source]

Get plotting configuration info and setup plots

Parameters:
  • fig – matplotlib figure

  • plotconf – list of config info for each plot

  • qadict – QA metrics dictionary

  • camera – to be used in output png title

  • expid – to be used in output png title

  • outfile – output png file

Returns:

png file containing all desired plots

desispec.quicklook.qlboxcar

Boxcar extraction for Spectra from Desi Image.

desispec.quicklook.qlboxcar.do_boxcar(image, tset, outwave, boxwidth=2.5, nspec=500, maskFile=None, usesigma=False, quick_resolution=False)[source]

Extracts spectra row by row, given the centroids

Parameters:
  • image – desispec.image object

  • tset – desispec.xytraceset like object

  • outwave – wavelength array for the final spectra output

  • boxwidth – HW box size in pixels

  • usesigma – if True, use sigma from psf file (ysigma) to calculate resolution data.

  • quick_resolution – whether to calculate the resolution matrix or use QuickResolution object

Returns flux, ivar, resolution

desispec.quicklook.qlconfig

class desispec.quicklook.qlconfig.Config(configfile, night, camera, expid, singqa, amps=True, rawdata_dir=None, specprod_dir=None, outdir=None, qlf=False, psfid=None, flatid=None, templateid=None, templatenight=None, qlplots=False, store_res=None)[source]

A class to generate Quicklook configurations for a given desi exposure. expand_config will expand out to full format as needed by quicklook.setup

dump_pa(paname)[source]

dump the PA outputs to respective files. This has to be updated for fframe and sframe files as QL anticipates for dumpintermediate case.

dump_qa()[source]

yaml outputfile for the set of qas for a given pa Name and default locations of files are handled by desispec.io.meta.findfile

expand_config()[source]

config: desispec.quicklook.qlconfig.Config object

io_qa(qaname)[source]

Specify the filenames: json and png for the given qa output

io_qa_pa(paname)[source]

Specify the filenames: json and png of the pa level qa files”

property paargs

Many arguments for the PAs are taken default. Some of these may need to be variable psfspfile is for offline extraction case

property palist

palist for this config see :class: Palist for details.

property qalist

qalist for the given palist

class desispec.quicklook.qlconfig.Palist(thislist=None, algorithms=None)[source]

Generate PA list and QA list for the Quicklook Pipeline for the given exposure

desispec.quicklook.qlconfig.check_config(outconfig, singqa)[source]

Given the expanded config, check for all possible file existence etc….

desispec.quicklook.qlexceptions

Exception classes for Quicklook.

exception desispec.quicklook.qlexceptions.ParameterException(value)[source]

desispec.quicklook.qlheartbeat

desispec.quicklook.qllogger

class desispec.quicklook.qllogger.QLLogger(name=None, loglevel=20)[source]

Simple logger class using logging

desispec.quicklook.qlpsf

Given a psf output file e.g. output from bootcalib.write_psf or desimodel/data/specpsf/PSF files this defines an interface that other codes can use the trace and wavelength solutions.

Mostly making parallel to specter.psf.PSF baseclass and inheriting as needed, but only xtrace, ytrace and wavelength solution available for this case. No resolution information yet.

class desispec.quicklook.qlpsf.PSF(filename)[source]

Base class for 2D psf

angstroms_per_pixel(ispec, wavelength)[source]

Return CCD pixel width in Angstroms for spectrum ispec at given wavlength(s). Wavelength may be scalar or array.

wavelength(ispec=None, y=None)[source]

returns wavelength evaluated at y

x(ispec=None, wavelength=None)[source]

returns CCD x centroids for the spectra ispec can be None, scalar or a vector wavelength can be None or a vector

y(ispec=None, wavelength=None)[source]

returns CCD y centroids for the spectra ispec can be None, scalar or a vector wavelength can be a vector but not allowing None #- similar as in specter.psf.PSF.y

desispec.quicklook.qlresolution

Quicklook version of resolution object that can calculate resolution efficiently from psf information

Author: Sami Kama

class desispec.quicklook.qlresolution.QuickResolution(*args: Any, **kwargs: Any)[source]

Quicklook version of the resolution mimicking desispec.resolution.Resolution with some reduction in dimentionality. Contains code from Resolution implementation Note that this is similar to desispec.resolution.Resolution, though faster and differing in implementation details that should be cross checked before merging these or replacing one with the other

desispec.quicklook.quickfiberflat

Here will be the fiberflat routines specific to quicklook.

  1. Dhungana, 2016

desispec.quicklook.quickfiberflat.apply_fiberflat(frame, fiberflat)[source]
Args: frame: desispec.frame.Frame object

fiberflat: desispec.fiberflat.Fiberflat object

desispec.quicklook.quickfiberflat.compute_fiberflat()[source]

computes fiberflat: A boss like algorithm writing in progress and will fit in here.

Args:

desispec.quicklook.quicklook

desispec.quicklook.quicklook.get_chan_spec_exp(inpname, camera=None)[source]

Get channel, spectrograph and expid from the filename itself

Parameters:
  • inpname – can be raw or pix, or frame etc filename

  • camera – is required for raw case, eg, r0, b5, z8 irrelevant for others

desispec.quicklook.quicklook.mapkeywords(kw, kwmap)[source]

Maps the keyword in the configuration to the corresponding object returned by the desispec.io module. e.g Bias Image file is mapped to biasimage object… for the same keyword “BiasImage”

desispec.quicklook.quicklook.runpipeline(pl, convdict, conf)[source]

Runs the quicklook pipeline as configured

Parameters:
  • pl – is a list of [pa,qas] where pa is a pipeline step and qas the corresponding qas for that pa

  • convdict – converted dictionary e.g : conf[“IMAGE”] is the real psf file but convdict[“IMAGE”] is like desispec.image.Image object and so on. details in setup_pipeline method below for examples.

  • conf – a configured dictionary, read from the configuration yaml file. e.g: conf=configdict=yaml.safe_load(open(‘configfile.yaml’,’rb’))

desispec.quicklook.quicklook.setup_pipeline(config)[source]

Given a configuration from QLF, this sets up a pipeline [pa,qa] and also returns a conversion dictionary from the configuration dictionary so that Pipeline steps (PA) can take them. This is required for runpipeline.

desispec.quicklook.quicksky

Here will be the sky computing and sky subtraction routines for QL.

desispec.quicklook.quicksky.compute_sky(fframe, fibermap=None, nsig_clipping=4.0, apply_resolution=False)[source]

Adding in the offline algorithm here to be able to apply resolution for sky compute. We will update this here as needed for quicklook. The original weighted sky compute still is the default.

Args: fframe: fiberflat fielded frame object

fibermap: fibermap object apply_resolution: if True, uses the resolution in the frame object to evaluate sky allowing fiber to fiber variation of resolution.

desispec.quicklook.quicksky.subtract_sky(fframe, skymodel)[source]

skymodel: skymodel object. fframe: frame object to do the sky subtraction, should be already fiber flat fielded need same number of fibers and same wavelength grid

desispec.resolution

Standardized handling of sparse wavelength resolution matrices.

Use python -m desispec.resolution to run unit tests.

class desispec.resolution.Resolution(*args: Any, **kwargs: Any)[source]

Canonical representation of a resolution matrix.

Inherits all of the method of scipy.sparse.dia_matrix, including todense() for converting to a dense 2D numpy array of matrix elements, most of which will be zero (so you generally want to avoid this).

Parameters:

data – Must be in one of the following formats listed below.

Options:
offsets: list of diagonals that the data represents. Only used if

data is a 2D dense array.

Raises:

ValueError – Invalid input for initializing a sparse resolution matrix.

Data formats:

  1. a scipy.sparse matrix in DIA format with the required diagonals (but not necessarily in the canoncial order);

  2. a 2D square numpy arrray (i.e., a dense matrix) whose non-zero values beyond default_ndiag will be silently dropped; or

  3. a 2D numpy array[ndiag, nwave] that encodes the sparse diagonal values in the same format as scipy.sparse.dia_matrix.data .

The last format is the one used to store resolution matrices in FITS files.

to_fits_array()[source]

Convert to an array of sparse diagonal values.

This is the format used to store resolution matrices in FITS files. Note that some values in the returned rectangular array do not correspond to actual matrix elements since the diagonals get smaller as you move away from the central diagonal. As long as you treat this array as an opaque representation for FITS I/O, you don’t care about this. To actually use the matrix, create a Resolution object from the fits array first.

Returns:

An array of (num_diagonals,nbins) sparse matrix

element values close to the diagonal.

Return type:

numpy.ndarray

desispec.resolution._gauss_pix(x, mean=0.0, sigma=1.0)[source]

Utility function to integrate Gaussian density within pixels

Parameters:
  • x (1D array) – pixel centers

  • mean (float) – mean of Gaussian

  • sigma (float) – sigma of Gaussian

Returns:

array of integals of the Gaussian density in the pixels.

Note

All pixels must be the same size

desispec.resolution._sort_and_symmeterize(data, offsets)[source]

Sort data,offsets and pad to ensure equal number of upper/lower diagonals

Parameters:
  • data – 2D array of diagonals, following scipy.sparse.dia_matrix.data ordering

  • offsets – 1D array of offsets; must be complete from min to max but doesn’t have to be sorted

Returns:

fulldata, fulloffsets

desispec.scatteredlight

Try to model and remove the scattered light.

desispec.scatteredlight.model_scattered_light(image, xyset)[source]

Model the scattered light in a preprocessed image. The method consist in convolving the “direct” light image (image * mask along spectral traces) and calibrating this convolved image using the data between the fiber bundles.

Parameters:
  • image – desispec.image.Image object

  • xyset – desispec.xytraceset.XYTraceSet object

Returns:

np.array of same shape as image.pix

Return type:

model

desispec.scripts

Main functions and commandline parsing.

Utility script to remove archive tiles for the merged target list (MTL).

desispec.scripts.archive_tilenight.archivetile(tiledir, archivedir, badpetals=[], dryrun=False)[source]

Archive tiledir to archivedir, leaving symlink behind

Parameters:
  • tiledir – full path to tiles/cumulative/TILEID/LASTNIGHT

  • archivedir – full path to tiles/archive/TILEID/ARCHIVEDATE

Options:

dryrun: if True, print messages but don’t move directories.

Returns int error code, non-zero if there was a problem

desispec.scripts.archive_tilenight.check_missing_zmtl(tiledir, archivetileroot)[source]

Check for zmtl file that exist in previous archives but not latest prod

Parameters:
  • tiledir – full path to tiles/cumulative/TILEID/LASTNIGHT

  • archivetileroot – full path to tiles/archive/TILEID

Returns: dict missing[petal] /path/to/zmtl/in/archive/not/in/prod

desispec.scripts.archive_tilenight.create_badpetal_zmtl(inzmtl, outzmtl)[source]

Using inzmtl as template, create outzmtl with ZWARN BAD_PETALQA set

Parameters:
  • inzmtl – full path to input zmtl file

  • outzmtl – full path to output zmtl file

desispec.scripts.archive_tilenight.freezedir(path, dryrun=False, force=False)[source]

Remove write permission from path unless dryrun

Parameters:

path (str) – path to directory to remove

Options:

dryrun: if True, print info but don’t actually remove write access force: if True, rerun chmod even if directory already appears frozen

Returns non-zero error code upon failure (not an exception)

Note: dryrun overrides force

Move src to dst, and create link from src -> dst. If src is already a link, recursively copy instead.

Parameters:
  • src – full path to a source directory

  • dst – full path to a destination directory that doesn’t yet exist

After this is run:

  • dst will have the contents originally in src

  • src will be a link to dst

desispec.scripts.assemble_fibermap

desispec.scripts.autocalib_fiberflat

desispec.scripts.average_fiberflat

desispec.scripts.badcolumn_mask

desispec.scripts.bootcalib

Utility functions to perform a quick calibration of DESI data

desispec.scripts.calibrate_tsnr_ensemble

Generate Master TSNR ensemble DFLUX files. See doc. 4723. Note: in this instance, ensemble avg. of flux is written, in order to efficiently generate tile depths.

desispec.scripts.calibrate_tsnr_ensemble.tsnr_efftime(exposures_table_filename, tsnr_table_filename, tracer, plot=True, exclude=None, use_sv1=False)[source]

Given an external calibration, e.g. /global/cfs/cdirs/desi/survey/observations/SV1/sv1-exposures.fits

with e.g. EFFTIME_DARK and

a tsnr afterburner run, e.g. /global/cfs/cdirs/desi/spectro/redux/cascades/tsnr-cascades.fits

Compute linear coefficient to convert TSNR2_TRACER_BRZ to EFFTIME_DARK or EFFTIME_BRIGHT.

desispec.spectra.coadd_frames

Coadd spectra.

desispec.scripts.coadd_preproc

desispec.scripts.coadd_spectra

Coadd spectra

desispec.scripts.compute.tsnr_ensemble

Generate Master TSNR ensemble DFLUX files. See doc. 4723. Note: in this instance, ensemble avg. of flux is written, in order to efficiently generate tile depths.

desispec.scripts.daily_processing

desispec.scripts.daily_processing.daily_processing_manager(specprod=None, exp_table_path=None, proc_table_path=None, path_to_data=None, expobstypes=None, procobstypes=None, z_submit_types=None, camword=None, badcamword=None, badamps=None, override_night=None, tab_filetype='csv', queue='realtime', exps_to_ignore=None, data_cadence_time=300, queue_cadence_time=1800, exp_cadence_time=2, dry_run_level=0, dry_run=False, no_redshifts=False, continue_looping_debug=False, dont_check_job_outputs=False, dont_resubmit_partial_jobs=False, verbose=False, use_specter=False, use_tilenight=False)[source]

Generates processing tables for the nights requested. Requires exposure tables to exist on disk.

Parameters:
  • specprod – str. The name of the current production. If used, this will overwrite the SPECPROD environment variable.

  • exp_table_path – str. Full path to where to exposure tables are stored, WITHOUT the monthly directory included.

  • proc_table_path – str. Full path to where to processing tables to be written.

  • path_to_data – str. Path to the raw data.

  • expobstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the exposure table.

  • procobstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the processing table.

  • z_submit_types – list of str’s or comma separated list of string. The “group” types of redshifts that should be submitted with each exposure. If not specified, default for daily processing is [‘cumulative’, ‘pernight-v0’]. If false, ‘false’, or [], then no redshifts are submitted.

  • camword – str. Camword that, if set, alters the set of cameras that will be set for processing. Examples: a0123456789, a1, a2b3r3, a2b3r4z3.

  • badcamword – str. Camword that, if set, will be removed from the camword defined in camword if given, or the camword inferred from the data if camword is not given.

  • badamps – str. Comma seperated list of bad amplifiers that should not be processed. Should be of the form “{camera}{petal}{amp}”, i.e. “[brz][0-9][ABCD]”. Example: ‘b7D,z8A’

  • override_night – str or int. 8 digit night, e.g. 20200314, of data to run on. If None, it runs on the current night.

  • tab_filetype – str. The file extension (without the ‘.’) of the exposure and processing tables.

  • queue – str. The name of the queue to submit the jobs to. Default is “realtime”.

  • exps_to_ignore – list. A list of exposure id’s that should not be processed. Each should be an integer.

  • data_cadence_time – int. Wait time in seconds between loops in looking for new data. Default is 30 seconds.

  • queue_cadence_time – int. Wait time in seconds between loops in checking queue statuses and resubmitting failures. Default is 1800s.

  • exp_cadence_time – int. Wait time in seconds between loops over each science exposure. Default 2.

  • dry_run_level – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • int – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • nonzero (If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • dry_run – this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.

  • defined (bool. When to run without submitting scripts or not. If dry_run_level is) – this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.

  • over-rides (then it) – this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.

  • no_redshifts

  • True (optional. Default is False. If) –

  • submitted. (redshifts are not) –

  • continue_looping_debug – bool. FOR DEBUG purposes only. Will continue looping in search of new data until the process is terminated. Default is False.

  • dont_check_job_outputs – data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.

  • False (bool. Default is False. If) – data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.

  • final (the code checks for the existence of the expected) – data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.

  • dont_resubmit_partial_jobs – False, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • is (bool. Default is False. Must be used with dont_check_job_outputs=False. If this flag) – False, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • verbose – bool. True if you want more verbose output, false otherwise. Current not propagated to lower code, so it is only used in the main daily_processing script itself.

  • use_specter

  • bool

  • True

  • specter (use) –

  • default. (otherwise use gpu_specter by) –

  • use_tilenight (bool, optional) – Default is False. If True, use desi_proc_tilenight for prestdstar, stdstar, and poststdstar steps for science exposures.

Returns: Nothing

Notes

Generates both exposure table and processing tables ‘on the fly’ and saves them at various checkpoints. These should be capable of being reloaded in case of interuption or accidental termination of the manager’s process.

desispec.scripts.editexptable

desispec.scripts.editexptable.change_exposure_table_rows(exptable, exp_str, colname, value, include_comment='', append_string=False, overwrite_value=False, joinsymb=',')[source]

Changes the column named colname to value of value for rows of exposure table in exptable that correspond to the exposures defined in exp_str.

Note: This edits and returns the exptable given in the inputs.

Parameters:
  • exptable

  • exposure. (astropy.table.Table. An exposure table defined in desispec.workflow.exptable. Each column is an) –

  • exp_str – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.

  • value. (bool. Default is False. Must be set to True if you want to overwrite a non-default) – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.

  • colname

  • values. (str. The column name in the exptable where you want to change) –

  • value

  • to. (any scalar type. The value you want to change the column value of each exp_str exposure row) –

  • include_comment – value for the given exp_str exposures.

  • to (str. A user specified comment to be added to the COMMENTS column after setting colname) – value for the given exp_str exposures.

  • append_string

  • string. (bool. True if you want to append your input value to the end of an existing) –

  • overwrite_value – If current value is a default value for that column for that row, this doesn’t need to be set.

  • value. – If current value is a default value for that column for that row, this doesn’t need to be set.

  • joinsymb – Default is ‘,’.

  • '|'. (str. The symbol used to separate string elements that are being appended. Shouldn't be) – Default is ‘,’.

Returns:

exptable, astropy.table.Table. The exposure table given in the input, with edits made to the column colname

for the rows corresponding to the exposure ID’s in exp_str.

desispec.scripts.editexptable.columns_not_to_append()[source]

Defines column names that shouldn’t be edited.

desispec.scripts.editexptable.columns_not_to_edit()[source]

Defines column names that shouldn’t be edited.

desispec.scripts.editexptable.columns_not_to_report()[source]

Returns list of column names that shouldn’t have reporting information saved because they are user-defined values.

desispec.scripts.editexptable.document_in_comments(tablerow, colname, value, comment_col='HEADERERR')[source]

Places “reporting” string in the appropriate comment column of the exposure table to document the edits being made.

Note: This alters and returns the input tablerow. How astropy handles this may vary. As of Jan 2021, I believe a copy is made in memory upon altering of a tablerow object. So the output here should be returned and assigned to overwrite the old value in the input table.

Parameters:
  • tablerow

  • array. (astropy.table.Row. A table row with columns colname and comment_col. Comment_col must be a numpy) –

  • colname

  • edited. (str. The name of the column that is being) –

  • value

  • to. (any scalar type. The value that the column's current value should be changed) –

  • comment_col

  • HEADERERR. (str. The name of the comment column where the change reporting should be placed. Default is) –

Returns:

tablerow, astropy.table.Row. A table row with columns colname and comment_col. Comment_col is a numpy array

with the new reporting string included.

desispec.scripts.editexptable.edit_exposure_table(exp_str, colname, value, night=None, include_comment='', tablepath=None, append_string=False, overwrite_value=False, use_spec_prod=True, read_user_version=False, write_user_version=False, overwrite_file=True, joinsymb=',')[source]

Edits the exposure table on disk to change the column named colname to value of value for rows of exposure table that correspond to the exposures defined in exp_str. The table on disk can be defined using night given directly with tablepath.

Note: This overwrites an exposure table file on disk by default.

Parameters:
  • exp_str – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.

  • value. (bool. Default is False. Must be set to True if you want to overwrite a non-default) – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.

  • colname

  • values. (str. The column name in the exptable where you want to change) –

  • value

  • to. (any scalar type. The value you want to change the column value of each exp_str exposure row) –

  • night

  • table. (str or int. The night the exposures were acquired on. This uniquely defines the exposure) –

  • include_comment – value for the given exp_str exposures.

  • to (str. A user specified comment to be added to the COMMENTS column after setting colname) – value for the given exp_str exposures.

  • tablepath – in desispec.workflow.exptable.

  • file (str. A relative or absolute path to the exposure table) – in desispec.workflow.exptable.

  • default (if named differently from the) – in desispec.workflow.exptable.

  • append_string

  • string. (bool. True if you want to append your input value to the end of an existing) –

  • overwrite_value – If current value is a default value for that column for that row, this doesn’t need to be set.

  • value. – If current value is a default value for that column for that row, this doesn’t need to be set.

  • use_spec_prod – defined SPECPROD as opposed to the exposure table repository location. Default is True.

  • currently (bool. True if you want to read in the exposure table defined by night from the) – defined SPECPROD as opposed to the exposure table repository location. Default is True.

  • read_user_version – USER name. Meant for test editing of a file multiple times. If the file doesn’t exist, the non-user value is loaded. Default is False.

  • user's (bool. True if you want to write in an exposure table saved including the current) – USER name. Meant for test editing of a file multiple times. If the file doesn’t exist, the non-user value is loaded. Default is False.

  • write_user_version – USER name. Meant for test editing of a file without overwriting the true exposure table. Default is False.

  • user's – USER name. Meant for test editing of a file without overwriting the true exposure table. Default is False.

  • overwrite_file

  • True. (bool. True if you want to overwrite the file on disk. Default is) –

  • joinsymb – Default is ‘,’.

  • '|'. (str. The symbol used to separate string elements that are being appended. Shouldn't be) – Default is ‘,’.

desispec.scripts.editexptable.parse_int_list(input_string, allints=None, only_unique=True)[source]

Given a str this determines what integer values it represents. Whether that be “all” indicating all ints in the table column, a range of integers specified with ‘:’, ‘-’, or ‘..’, a single integer, or an indeterminant number of them in a comma separated list.

Parameters:
  • input_string – multiple of these separated by a comma. ‘all’ requires allints.

  • range (str. String with either integer) – multiple of these separated by a comma. ‘all’ requires allints.

  • integer (single) – multiple of these separated by a comma. ‘all’ requires allints.

  • of (or 'all'. It can have a combination) – multiple of these separated by a comma. ‘all’ requires allints.

  • allints

  • specified. (np.array. One dimensional array of all integers. Returns if 'all' is) –

  • only_unique – are kept.

  • string (bool. True if you want a unique set returned. Otherwise repeated entries in the input) – are kept.

Returns:

out_array, np.array. Array of ints for the string specified.

desispec.scripts.editexptable.parse_int_list_term(input_string, allints=None)[source]

Given a str this determines what integer values it represents. Whether that be “all” indicating all ints in the table column, a range of integers specified with ‘:’, ‘-’, or ‘..’, or a single integer. This should not be a list.

Parameters:
  • input_string

  • range (str. String with either integer) –

  • integer (single) –

  • allints (or 'all'. 'all' requires) –

  • allints

  • specified. (np.array. One dimensional array of all integers. Returns if 'all' is) –

Returns:

out_array, np.array. Array of ints for the string specified.

desispec.scripts.editexptable.process_int_range_inclusive(input_string)[source]

Given a str indicating a range of integers, this auto-detects the symbol used and returns that range as an INCLUSIVE numpy array of ints. Symbol can be ‘:’, ‘-’, or ‘..’.

Parameters:
  • input_string – 102 returns 100,101,102.

  • 100 (str. String with integer range with the upper value being included in the output. E.g.) – 102 returns 100,101,102.

Returns:

np.array. Array of ints for the range specified in the input_string.

desispec.scripts.editexptable.validate_value(colname, value, joinsymb)[source]

Checks that the value provided matches the syntax of the colname given. If the syntax is incorrect an error is raised.

Warning: may change the value of “value” and returns it.

Parameters:
  • colname

  • edited. (str. The name of the column that is being) –

  • value

  • to. (any scalar type. The value that the column's current value should be changed) –

Returns:

value, any scalar type. The value that the column’s current value should be changed to. This is verified to

have the proper syntax for the colname given.

desispec.scripts.emlinefit

desispec.scripts.exposure_qa

This script computes QA scores per exposure, after the cframe are done

desispec.scripts.exposure_qa._func(arg)[source]

Wrapper function to compute_exposure_qa for multiprocessing

desispec.scripts.exposure_qa.func(night, expid, specprod_dir, outfile=None)[source]

Wrapper function to compute_exposure_qa for multiprocessing

desispec.scripts.exposuretable

desispec.scripts.exposuretable.create_exposure_tables(nights=None, night_range=None, path_to_data=None, exp_table_path=None, obstypes=None, exp_filetype='csv', cameras=None, bad_cameras=None, badamps=None, verbose=False, no_specprod=False, overwrite_files=False)[source]

Generates processing tables for the nights requested. Requires exposure tables to exist on disk.

Parameters:
  • nights – str, int, or comma separated list. The night(s) to generate procesing tables for.

  • night_range – str, comma separated pair of nights in form YYYYMMDD,YYYYMMDD for first_night,last_night specifying the beginning and end of a range of nights to be generated. last_night should be inclusive.

  • path_to_data – str. The path to the raw data and request*.json and manifest* files.

  • exp_table_path – str. Full path to where to exposure tables should be saved, WITHOUT the monthly directory included.

  • obstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the exposure table.

  • exp_filetype – str. The file extension (without the ‘.’) of the exposure tables.

  • verbose – boolean. Whether to give verbose output information or not. True prints more information.

  • no_specprod – boolean. Create exposure table in repository location rather than the SPECPROD location

  • overwrite_files – boolean. Whether to overwrite processing tables if they exist. True overwrites.

  • cameras – str. Explicitly define the cameras for which you want to reduce the data. Should be a comma separated list. Only numbers assumes you want to reduce r, b, and z for that camera. Otherwise specify separately [brz][0-9].

  • bad_cameras – str. Explicitly define the cameras that you don’t want to reduce the data. Should be a comma separated list. Only numbers assumes you want to reduce r, b, and z for that camera. Otherwise specify separately [brz][0-9].

  • badamps – str. Define amplifiers that you know to be bad and should not be processed. Should be a list separated by comma or semicolon. Saved list will converted to semicolons. Each entry should be of the form {camera}{spectrograph}{amp}, i.e. [brz][0-9][A-D].

Returns: Nothing

desispec.scripts.extract

Extract spectra from DESI pre-processed raw data.

desispec.scripts.extract._extract_and_save(img, psf, bspecmin, bnspec, specmin, wave, raw_wave, fibers, fibermap, outbundle, outmodel, bundlesize, args, log)[source]

Performs the main extraction and saving of extracted frames found in the body of the main loop. Refactored to be callable by both MPI and non-MPI versions of the code. This should be viewed as a shorthand for the following commands.

desispec.scripts.extract.barycentric_correction_multiplicative_factor(header)[source]

Returns mult. barycentric correction factor using coords in header

header must contrain MJD or MJD-OBS; and TARGTRA,TARGTDEC or SKYRA,SKYDEC or TELRA,TELDEC or RA,DEC

desispec.scripts.extract.gpu_specter_check_input_options(args)[source]

Perform pre-flight checks on input options

returns ok(True/False), message

desispec.scripts.fibercrosstalk

desispec.scripts.fiberflat

Utility functions to compute a fiber flat correction and apply it We try to keep all the (fits) io separated.

desispec.scripts.fluxcalibration

desispec.scripts.group_spectra

Regroup spectra by healpix.

desispec.scripts.healpix_redshifts

Script for running healpix-based coadds+redshifts, assuming that the spectral regrouping into healpix has already happened.

desispec.scripts.humidity_corrected_fiberflat

desispec.scripts.inspect_dark

desispec.scripts.interpolate_fiber_psf

desispec.scripts.makezmtl

desispec.scripts.master_nea

This script generates a master NEA (Noise Equivalent Area) file for a given camera.

desispec.scripts.master_nea.process_one(w, psf, ifiber)[source]

Compute the 1D NEA for a given wavelength, fiber and desispec psf instance.

input:

w: wavelength, Angstroms. psf: desispec psf instance. ifiber: fiber indexing integer [0,500].

Returns:

list of 1D nea value [pixles] and angstroms per pix. for this fiber and wavelength.

desispec.scripts.mergebundles

exspec extracts individual bundles of spectra with one bundle per output file. This script merges them back together into a single file combining all bundles.

This workflow is hacky. Release early, release often, but also refactor often.

Stephen Bailey, LBL March 2014

desispec.scripts.night

Automated nightly processing.

desispec.scripts.nightly_bias

functions for bin/desi_compute_nightly_bias script

desispec.scripts.pipe

Interactive control of the pipeline

desispec.scripts.pipe_exec

Run one or more pipeline tasks.

desispec.scripts.preproc

Command line wrappers for pre-processing a DESI raw exposure

desispec.scripts.preproc._preproc_file_kwargs_wrapper(opts)[source]

This function just unpacks opts dict for preproc_file so that it can be used with multiprocessing.Pool.map

desispec.scripts.preproc.preproc_file(infile, camera, outfile=None, outdir=None, fibermap=None, zero_masked=False, **preproc_opts)[source]

Preprocess a single camera from a single input file

Parameters:
  • infile – input raw data file

  • camera – camera, e.g. ‘b0’, ‘r1’, ‘z9’

Options:

outfile: output preprocessed image file to write outdir: output directory; derive filename from infile NIGHT and EXPID fibermap: fibermap filename to include in output zero_masked (bool): set masked pixels to 0 preproc_opts: dictionary to pass to preproc

Returns error code (1=error, 0=success) but will not raise exception if there is an I/O or preprocessing failure (allows other parallel procs to proceed).

Note: either outfile or outdir must be provided

desispec.scripts.proc

One stop shopping for processing a DESI exposure

Examples at NERSC:

# ARC: 18 min on 2 nodes
time srun -N 2 -n 60 -C haswell -t 25:00 --qos realtime desi_proc --mpi -n 20191029 -e 22486

# FLAT: 13 min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22487

# TWILIGHT: 8min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22497

# SKY: 11 min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22536

# ZERO: 2 min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22561

desispec.scripts.proc_joint_fit

desispec.scripts.proc_tilenight

Script for science processing of a given DESI tile and night

desispec.scripts.procdashboard

desispec.scripts.procdashboard.main(args=None)[source]

Code to generate a webpage for monitoring of desi_dailyproc production status Usage: -n can be ‘all’ or series of nights separated by comma or blank like 20200101,20200102 or 20200101 20200102 Normal Mode: desi_proc_dashboard -n 3 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/ desi_proc_dashboard -n 20200101,20200102 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/

desispec.scripts.procdashboard.parse(options)[source]

Initialize the parser to read input

desispec.scripts.procdashboard.populate_night_info(night, check_on_disk=False, night_json_info=None, skipd_expids=None)[source]

For a given night, return the file counts and other other information for each exposure taken on that night input: night output: a dictionary containing the statistics with expid as key name FLAVOR: FLAVOR of this exposure OBSTYPE: OBSTYPE of this exposure EXPTIME: Exposure time SPECTROGRAPHS: a list of spectrographs used n_spectrographs: number of spectrographs n_psf: number of PSF files n_ff: number of fiberflat files n_frame: number of frame files n_sframe: number of sframe files n_cframe: number of cframe files n_sky: number of sky files

desispec.scripts.processingtable

desispec.scripts.processingtable.create_processing_tables(nights=None, night_range=None, exp_table_path=None, proc_table_path=None, obstypes=None, overwrite_files=False, verbose=False, no_specprod_exptab=False, exp_filetype='csv', prod_filetype='csv', joinsymb='|')[source]

Generates processing tables for the nights requested. Requires exposure tables to exist on disk.

Parameters:
  • nights – str, int, or comma separated list. The night(s) to generate procesing tables for.

  • night_range – str, comma separated pair of nights in form YYYYMMDD,YYYYMMDD for first_night,last_night specifying the beginning and end of a range of nights to be generated. last_night should be inclusive.

  • exp_table_path – str. Full path to where to exposure tables are stored, WITHOUT the monthly directory included.

  • proc_table_path – str. Full path to where to processing tables to be written.

  • obstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the processing table.

  • overwrite_files – boolean. Whether to overwrite processing tables if they exist. True overwrites.

  • verbose – boolean. Whether to give verbose output information or not. True prints more information.

  • no_specprod_exptab – boolean. Read exposure table in repository location rather than the SPECPROD location.

  • exp_filetype – str. The file extension (without the ‘.’) of the exposure tables.

  • prod_filetype – str. The file extension (without the ‘.’) of the processing tables.

  • joinsymb – str. Symbol to use to indicate the separation of array values when converting to and from strings for saving to csv. Default is highly advised and is ‘|’. Using a comma will break many things.

Returns: Nothing

Notes

Requires exposure tables to exist on disk. Either in the default location or at the location specified using the function arguments.

desispec.scripts.procexp

This script processes an exposure by applying fiberflat, sky subtraction, spectro-photometric calibration depending on input. Optionally, includes tsnr in the scores hdu.

desispec.scripts.qa_exposure

Script for generating QA for a full exposure

desispec.scripts.qa_frame

Script for generating QA for a single Frame

desispec.scripts.qa_night

Script for analyzing QA from a Night

desispec.scripts.qa_prod

Script for generating QA from a Production run

desispec.scripts.qproc

Run DESI qproc on a given exposure

desispec.scripts.qsomgii

desispec.scripts.qsomgii.save_dataframe_to_fits(dataframe, filename, DESI_TARGET, clobber=True)[source]

Save info from pandas dataframe in a fits file. Need to write the dtype array because of the list in the pandas dataframe (no other solution found)

Parameters:
  • dataframe (pandas dataframe) – dataframe containg the all the necessary QSO info

  • filename (str) – name of the fits file

  • DESI_TARGET (str) – name of DESI_TARGET for the wanted version of the target selection

  • clobber (bool) – overwrite the fits file defined by filename ?

Returns:

None

desispec.scripts.qsomgii.select_targets_with_mgii_fitter(redrock, fibermap, sel_to_mgii, spectra_name, redrock_name, param_mgii_fitter, DESI_TARGET, save_target)[source]

Run QuasarNet to the object with index_to_QN == True from spectra_name. Then, Re-Run RedRock for the targetids which are selected by QN as a QSO.

Parameters:
  • redrock – fitsio hdu ‘REDSHIFTS’ from redrock file

  • fibermap – fitsio hdu ‘FIBERMAP’ from redrock file

  • sel_to_mgii (bool array) – size 500. Select on which objects mgii will be apply (index based on redrock table)

  • redrock_name (spectra_name /) – The name of the spectra / associated redrock file

  • param_mgii_fitter (dict) – contains info for the MgII fitter as lambda_width, max_sigma min_sigma, min_deltachi2, min_signifiance_A, min_A

  • DESI_TARGET (str) – name of DESI_TARGET for the wanted version of the target selection

  • save_target (str) – restricted (save only IS_QSO_MGII==true targets) / all (save all the sample)

Returns:

contains all the information useful to build the QSO cat

Return type:

QSO_sel (pandas dataframe)

desispec.scripts.qsoqn

desispec.scripts.qsoqn.collect_redshift_with_new_RR_run(spectra_name, targetid, z_prior, param_RR, comm=None)[source]

Wrapper to run Redrock on targetid (numpy array) from the spectra_name_file with z_prior using the template contained in template_file

Parameters:
  • spectra_name (str) – The name of the spectra file.

  • targetid (int array) – Array of the targetid (contained in the spectra_name_file) on which RR will be rerun with prior and qso template.

  • z_prior (float array) – Array of the same size than targetid with the redshift estimated by QN for the associated targetid

  • param_RR (dict) – Contains info to re-run RR as the template_filename, filename_priors, filename_output_rerun_RR, filename_redrock_rerun_RR

  • comm (object, optional) – MPI communicator to pass to redrock; must be size=1

Returns:

A tuple containing:

  • redshift (numpy array): Array containing best redshift estimation by the new run of RR

  • err_redshift (numpy array): Array containing the associated error for the redshift

  • coeffs (numpy array): array containing the coefficient for the best fit given by RR even we work only with QSO template, it has a “shape” of redshift.size x 10; warning: they have to be converted into a list (with .tolist()) to be added in the pandas dataframe

Return type:

tuple

desispec.scripts.qsoqn.save_dataframe_to_fits(dataframe, filename, DESI_TARGET, clobber=True)[source]

Save info from pandas dataframe in a fits file. Need to write the dtype array because of the list in the pandas dataframe (no other solution found)

Parameters:
  • dataframe (pandas dataframe) – dataframe containg the all the necessary QSO info

  • filename (str) – name of the fits file

  • DESI_TARGET (str) – name of DESI_TARGET for the wanted version of the target selection

  • clobber (bool) – overwrite the fits file defined by filename ?

Returns:

None

desispec.scripts.qsoqn.selection_targets_with_QN(redrock, fibermap, sel_to_QN, DESI_TARGET, spectra_name, param_QN, param_RR, save_target, comm=None)[source]

Run QuasarNet to the object with index_to_QN == True from spectra_name. Then, Re-Run RedRock for the targetids which are selected by QN as a QSO.

Parameters:
  • redrock – fitsio hdu ‘REDSHIFTS’ from redrock file

  • fibermap – fitsio hdu ‘FIBERMAP’ from redrock file

  • sel_to_QN (bool array) – Select on which objects QN will be apply (index based on redrock)

  • DESI_TARGET (str) – name of DESI_TARGET for the wanted version of the target selection

  • spectra_name (str) – The name of the spectra file

  • param_QN (dict) – contains info for QN as n_thresh and c_thresh

  • param_RR (dict) – contains info to re-run RR as the template_filename, filename_priors, filename_output_rerun_RR, filename_redrock_rerun_RR

  • save_target (str) – restricted (save only IS_QSO_QN_NEW_RR==true targets) / all (save all the sample)

  • comm – MPI communicator to pass to redrock; must be size=1

  • optional – MPI communicator to pass to redrock; must be size=1

Returns:

contains all the information useful to build the QSO cat

Return type:

QSO_sel (pandas dataframe)

desispec.scripts.quicklook

Command line wrapper for running a QL pipeline

QuickLook team @Southern Methodist University (SMU) First version Spring 2016 Latest revision July 2018

Running QuickLook:

desi_quicklook -i qlconfig_science.yaml -n 20191001 -c r0 -e 3577

This requires having necessary input files and setting the following environment variables:

QL_SPEC_DATA: directory containing raw/fibermap files (full path: $QL_SPEC_DATA/night/expid)
QL_SPEC_REDUX: directory for QL output (full path: $QL_SPEC_REDUX/exposures/night/expid)
DESI_CALIBRATION_DATA: directory containing calibration files

Necessary Quicklook command line arguments:

-i,--config_file : path to QL configuration file
-n,--night : night to be processed
-c,--camera : camera to be processed
-e,--expid : exposure ID to be processed

Optional QuickLook arguments:

--rawdata_dir : directory containing raw/fibermap files (overrides $QL_SPEC_DATA)
--specprod_dir : directory for QL output (overrides $QL_SPEC_REDUX)

Plotting options:

-p (including path to plotting configuration file) : generate configured plots
-p (only using -p with no configuration file) : generate QL hardcoded plots
desispec.scripts.quicklook.parse()[source]

Should have either a pre existing config file, or need to generate one using config module

desispec.scripts.rejectcosmics

This script finds cosmics in a pre-processed image and write the result in the mask extension of an output image (output can be same as input).

desispec.scripts.repair_mask

Patch holes in spectrograph 2D cosmic-ray masks using morphological binary image analysis.

desispec.scripts.select_calib_stars

desispec.scripts.sky

desispec.scripts.skysubresid

Script for generating plots on SkySub residuals

desispec.scripts.specex

Run PSF estimation.

desispec.scripts.specex.compatible(head1, head2)[source]

Return bool for whether two FITS headers are compatible for merging PSFs

desispec.scripts.specex.mean_psf(inputs, output)[source]

Average multiple input PSF files into an output PSF file

Parameters:
  • inputs – list of input PSF files

  • output – output filename

desispec.scripts.specex.merge_psf(inputs, output)[source]

Merge individual per-bundle PSF files into full PSF

Parameters:
  • inputs – list of input PSF filenames

  • output – output filename

desispec.scripts.specex.run(comm, cmds, cameras)[source]

Run PSF fits with specex on a set of ccd images in parallel using the run method of the desispec.workflow.schedule.Schedule (Schedule) class.

Parameters:
  • comm – MPI communicator containing all processes available for work and scheduling (usually MPI_COMM_WORLD); at least 21 processes should be available, one for scheduling and (group_size=) 20 to fit all bundles for a given ccd image. Otherwise there is no constraint on the number of ranks available, but (comm.Get_size()-1)%group_size will be unused, since every job is assigned exactly group_size=20 ranks. The variable group_size is set at the number of bundles on a ccd, and there is currently no support for any other number, due to the way merging of bundles is currently done.

  • cmds – dictionary keyed by a camera string (e.g. ‘b0’, ‘r1’, …) with values being the ‘desi_compute_psf …’ string that one would run on the command line.

  • cameras – list of camera strings identifying the entries in cmds to be run as jobs in parallel jobs, one entry per ccd image to be fit. Processes assigned to cameras not present as keys in cmds will write a message to the log instead of running a PSF fit.

The function first defines the procedure to call specex for a given ccd image with the “fitframe” inline function, passes the fitframe function to the Schedule initialization method, and then calls the run method of the Schedule class to call fitframe len(cameras) times, each with group_size = 20 processes.

desispec.scripts.specscore

Compute some information scores on spectra in frames

desispec.scripts.stdstars

Get the normalized best template to do flux calibration.

desispec.scripts.stdstars.get_gaia_ab_correction()[source]

Get the dictionary with corrections from AB magnitudes to Vega magnitudes (as the official gaia catalog is in vega)

desispec.scripts.stdstars.get_magnitude(stdwave, model, model_filters, cur_filt)[source]

Obtain magnitude for a filter taking into account the ab/vega correction if needed. Wwe assume the flux is in units of 1e-17 erg/s/cm^2/A

desispec.scripts.stdstars.main(args=None, comm=None)[source]

finds the best models of all standard stars in the frame and normlize the model flux. Output is written to a file and will be called for calibration.

desispec.scripts.submit_night

desispec.scripts.submit_night.get_completed_tiles(specstatus_path=None, complete_tiles_thrunight=None)[source]

Uses a tiles-specstatus.ecsv file and selection criteria to determine what tiles have beeen completed. Takes an optional argument to point to a custom specstatus file. Returns an array of TILEID’s.

Parameters:
  • specstatus_path – Location of the surveyops specstatus table.

  • str – Location of the surveyops specstatus table.

  • $DESI_SURVEYOPS/ops/tiles-specstatus.ecsv. (optional. Default is) – Location of the surveyops specstatus table.

  • complete_tiles_thrunight – on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None.

  • int – on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None.

  • completed (optional. Default is None. Only tiles) – on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None.

Returns:

array-like. The tiles from the specstatus file determined by the selection criteria to be completed.

desispec.scripts.submit_night.submit_night(night, proc_obstypes=None, z_submit_types=None, queue='realtime', reservation=None, system_name=None, exp_table_path=None, proc_table_path=None, tab_filetype='csv', dry_run_level=0, dry_run=False, no_redshifts=False, error_if_not_available=True, append_to_proc_table=False, ignore_proc_table_failures=False, dont_check_job_outputs=False, dont_resubmit_partial_jobs=False, tiles=None, surveys=None, laststeps=None, use_tilenight=False, all_tiles=False, specstatus_path=None, use_specter=False, do_cte_flat=False, complete_tiles_thrunight=None, all_cumulatives=False)[source]

Creates a processing table and an unprocessed table from a fully populated exposure table and submits those jobs for processing (unless dry_run is set).

Parameters:
  • night (int) – The night of data to be processed. Exposure table must exist.

  • proc_obstypes (list or np.array, optional) – A list of exposure OBSTYPE’s that should be processed (and therefore added to the processing table).

  • z_submit_types (list of str or comma-separated list of str, optional) – The “group” types of redshifts that should be submitted with each exposure. If not specified, default for daily processing is [‘cumulative’, ‘pernight-v0’]. If false, ‘false’, or [], then no redshifts are submitted.

  • queue (str, optional) – The name of the queue to submit the jobs to. Default is “realtime”.

  • reservation (str, optional) – The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • system_name (str) – batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu

  • exp_table_path (str) – Full path to where to exposure tables are stored, WITHOUT the monthly directory included.

  • proc_table_path (str) – Full path to where to processing tables to be written.

  • tab_filetype (str, optional) – The file extension (without the ‘.’) of the exposure and processing tables.

  • dry_run_level (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written but not submitted. If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • dry_run (bool, optional) – When to run without submitting scripts or not. If dry_run_level is defined, then it over-rides this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.

  • no_redshifts (bool, optional) – Whether to submit redshifts or not. If True, redshifts are not submitted.

  • error_if_not_available (bool, optional) – Default is True. Raise as error if the required exposure table doesn’t exist, otherwise prints an error and returns.

  • append_to_proc_table (bool, optional) – True if you want to submit jobs even if a processing table already exists. Otherwise jobs will be appended to it. Default is False

  • ignore_proc_table_failures (bool, optional) – True if you want to submit other jobs even the loaded processing table has incomplete jobs in it. Use with caution. Default is False.

  • dont_check_job_outputs (bool, optional) – Default is False. If False, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.

  • dont_resubmit_partial_jobs (bool, optional) – Default is False. Must be used with dont_check_job_outputs=False. If this flag is False, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • tiles (array-like, optional) – Only submit jobs for these TILEIDs.

  • surveys (array-like, optional) – Only submit science jobs for these surveys (lowercase)

  • laststeps (array-like, optional) – Only submit jobs for exposures with LASTSTEP in these laststeps (lowercase)

  • use_tilenight (bool, optional) – Default is False. If True, use desi_proc_tilenight for prestdstar, stdstar, and poststdstar steps for science exposures.

  • all_tiles (bool, optional) – Default is False. Set to NOT restrict to completed tiles as defined by the table pointed to by specstatus_path.

  • specstatus_path (str, optional) – Default is $DESI_SURVEYOPS/ops/tiles-specstatus.ecsv. Location of the surveyops specstatus table.

  • use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.

  • do_cte_flat (bool, optional) – Default is False. If True, one second flat exposures are processed for cte identification.

  • complete_tiles_thrunight (int, optional) – Default is None. Only tiles completed on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None or all_tiles is True.

  • all_cumulatives (bool, optional) – Default is False. Set to run cumulative redshifts for all tiles even if the tile has observations on a later night.

desispec.scripts.submit_prod

desispec.scripts.submit_prod.assign_survey(night, conf)[source]

Takes a desi production configuration (yaml) dictionary and determines the survey corresponding to a given night based on the contents of the conf dictionary, if psosible. Otherwise returns None.

Parameters:
  • night (int) – The night you want to know the survey it corresponds to.

  • conf (dict) – Dictionary that returned when the configuration yaml file was read in.

Returns:

survey, str. The survey the night was taken under, according to the conf file.

desispec.scripts.submit_prod.get_all_nights()[source]

Returns a full list of all nights availabel in the DESI Raw data directory.

Returns:

nights, list. A list of nights on or after Jan 1 2020 in which data exists at NERSC.

desispec.scripts.submit_prod.submit_production(production_yaml, dry_run=False, error_if_not_available=False)[source]

Interprets a production_yaml file and submits the respective nights for processing within the defined production.

Parameters:
  • production_yaml (str) – Pathname of the yaml file that defines the production.

  • dry_run (bool, optional) – Default is False. Should the jobs written to the processing table actually be submitted for processing.

  • error_if_not_available (bool, optional) – Default is True. Raise as error if the required exposure table doesn’t exist, otherwise prints an error and returns.

Returns:

None.

desispec.scripts.tile_redshifts

desispec.scripts.tile_redshifts.batch_tile_redshifts(tileid, exptable, group, camword=None, submit=False, queue='regular', reservation=None, max_gpuprocs=None, no_gpu=False, dependency=None, system_name=None, run_zmtl=False, no_afterburners=False)[source]

Generate batch script for spectra+coadd+redshifts for a tile

Parameters:
  • tileid (int) – Tile ID

  • exptable (Table) – has columns NIGHT EXPID to use; ignores other columns. Doesn’t need to be full pipeline exposures table (but could be)

  • group (str) – cumulative, pernight, perexp, or a custom name

Options:

camword (str): camword of cameras to include submit (bool): also submit batch script to queue queue (str): batch queue name reservation (str): batch reservation name max_gpuprocs (int): Number of gpu processes no_gpu (bool): Default false. If true it doesn’t use GPU’s even if available. dependency (str): passed to sbatch –dependency upon submit system_name (str): batch system name, e.g. cori-haswell, perlmutter-gpu run_zmtl (bool): if True, also run make_zmtl_files no_afterburners (bool): if True, do not run QSO afterburners

Returns tuple (scriptpath, error):

scriptpath (str): full path to generated script err (int): return code from submitting job (0 if submit=False)

By default this generates the script but don’t submit it

desispec.scripts.tile_redshifts.generate_tile_redshift_scripts(group, nights=None, tileid=None, expids=None, explist=None, spectrographs=None, camword=None, max_gpuprocs=None, no_gpu=False, run_zmtl=False, no_afterburners=False, batch_queue='regular', batch_reservation=None, batch_dependency=None, system_name=None, nosubmit=False)[source]

Creates a slurm script to run redshifts per tile. By default it also submits the job to Slurm. If nosubmit is True, the script is created but not submitted to Slurm.

Parameters:
  • group (str) – Type of coadd redshifts to run. Options are cumulative, pernight, perexp, or a custom name.

  • nights (int, or list or np.array of int's) – YEARMMDD nights.

  • tileid (int) – Tile ID.

  • expids (int, or list or np.array of int's) – Exposure IDs.

  • explist (str) – File with columns TILE NIGHT EXPID to use

  • spectrographs (str or list of int) – spectrographs to include

  • camword (str) – camword of cameras to include

  • max_gpuprocs (int) – Number of gpu processes

  • no_gpu (bool) – Default false. If true it doesn’t use GPU’s even if available.

  • run_zmtl (bool) – If True, also run make_zmtl_files

  • no_afterburners (bool) – If True, do not run QSO afterburners

  • batch_queue (str) – Batch queue name. Default is ‘regular’.

  • batch_reservation (str) – Batch reservation name.

  • batch_dependency (str) – Job dependencies passed to sbatch –dependency .

  • system_name (str) – Batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu.

  • nosubmit (bool) – Generate scripts but don’t submit batch jobs. Default is False.

Returns:

The path names of the scripts created during the function call

that returned a null batcherr.

failed_jobs (list of str): The path names of the scripts created during the function call

that returned a batcherr.

Return type:

batch_scripts (list of str)

Note: specify spectrographs or camword but not both

desispec.scripts.tile_redshifts_bash

desispec.scripts.tile_redshifts_bash.batch_tile_redshifts(tileid, exptable, group, spectrographs=None, submit=False, queue='realtime', reservation=None, dependency=None, system_name=None, run_zmtl=False, noafterburners=False)[source]

Generate batch script for spectra+coadd+redshifts for a tile

Parameters:
  • tileid (int) – Tile ID

  • exptable (Table) – has columns NIGHT EXPID to use; ignores other columns. Doesn’t need to be full pipeline exposures table (but could be)

  • group (str) – cumulative, pernight, perexp, or a custom name

Options:

spectrographs (list of int): spectrographs to include submit (bool): also submit batch script to queue queue (str): batch queue name reservation (str): batch reservation name dependency (str): passed to sbatch –dependency upon submit system_name (str): batch system name, e.g. cori-haswell, perlmutter-gpu run_zmtl (bool): if True, also run make_zmtl_files noafterburners (bool): if True, do not run QSO afterburners

Returns tuple (scriptpath, error):

scriptpath (str): full path to generated script err (int): return code from submitting job (0 if submit=False)

By default this generates the script but don’t submit it

desispec.scripts.tile_redshifts_bash.generate_tile_redshift_scripts(group, night=None, tileid=None, expid=None, explist=None, spectrographs=None, run_zmtl=False, noafterburners=False, batch_queue='realtime', batch_reservation=None, batch_dependency=None, system_name=None, nosubmit=False)[source]

Creates a slurm script to run redshifts per tile. By default it also submits the job to Slurm. If nosubmit is True, the script is created but not submitted to Slurm.

Parameters:
  • group (str) – Type of coadd redshifts to run. Options are cumulative, pernight, perexp, or a custom name.

  • night (int, or list or np.array of int's) – YEARMMDD nights.

  • tileid (int) – Tile ID.

  • expid (int, or list or np.array of int's) – Exposure IDs.

  • explist (str) – File with columns TILE NIGHT EXPID to use

  • spectrographs (str or list of int) – spectrographs to include

  • run_zmtl (bool) – If True, also run make_zmtl_files

  • noafterburners (bool) – If True, do not run QSO afterburners

  • batch_queue (str) – Batch queue name. Default is ‘realtime’.

  • batch_reservation (str) – Batch reservation name.

  • batch_dependency (str) – Job dependencies passed to sbatch –dependency .

  • system_name (str) – Batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu.

  • nosubmit (bool) – Generate scripts but don’t submit batch jobs. Default is False.

Returns:

The path names of the scripts created during the function call

that returned a null batcherr.

failed_jobs (list of str): The path names of the scripts created during the function call

that returned a batcherr.

Return type:

batch_scripts (list of str)

desispec.scripts.tile_redshifts_bash.write_redshift_script(batchscript, outdir, jobname, num_nodes, group, spectro_string, suffix, frame_glob=None, expfile=None, healpix=None, extra_header=None, queue='regular', system_name=None, onetile=True, tileid=None, night=None, expid=None, nexps=0, run_zmtl=False, noafterburners=False, redrock_nodes=1, redrock_cores_per_rank=1)[source]

Write a batch script for running coadds, redshifts, and afterburners

Parameters:
  • batchscript (str) – filepath to batch script to write

  • outdir (str) – output directory to write data

  • jobname (str) – slurm job name

  • num_nodes (int) – number of nodes to allocate

  • group (str) – used for tile redshifts, e.g. ‘cumulative’

  • spectro_string (str) – e.g. ‘0 1 2 3’ spectrographs to run

  • suffix (str) – filename suffix (e.g. TILEID-thruNIGHT)

Options:

frame_glob (str): glob for finding input cframes expfile (str): filename with NIGHT EXPID SPECTRO healpix (int): healpix number (to use with group=healpix) extra_header (dict): extra key/value pairs to add to header queue (str): queue name system_name (str): e.g. cori-haswell, cori-knl, perlmutter-gpu onetile (bool): coadd assuming input is for a single tile? tileid (int): tileid to process; only needed for group=’cumulative’ night (int): process through or on night YEARMMDD; for group=’cumulative’ and ‘pernight’ expid (int): expid for group=’perexp’ nexps (int): number of exposures to be fit run_zmtl (bool): if True, also run zmtl noafterburners (bool): if True, skip QSO afterburners redrock_nodes (int): number of nodes for each redrock call redrock_cores_per_rank (int): number of cores/rank to use for redrock

Note: some of these options are hacked to also be used by healpix_redshifts, e.g. by providing spectro_string=’sv3’ instead of list of spectrographs.

Note: Use redrock_cores_per_rank > 1 to reserve extra memory per rank for large input coadd files (e.g. sv3 healpix).

Note: must specify frame_glob for tile-based groups, and expfile for group=healpix.

desispec.scripts.tileqa

This script computes QA scores per exposure, after the cframe are done

desispec.scripts.tileqa._func(arg)[source]

Wrapper function to compute_tile_qa for multiprocessing

desispec.scripts.tileqa._wrap_make_tile_qa_plot(qafitsfile, specprod_dir=None)[source]

Utility wrapper to make qa plot with try/except/log wrappers

Parameters:

qafitsfile (str) – full path to tile-qa-*.fits data

Options:

specprod_dir (str): full path to production directory

Returns: full path to qapngfile, or None upon failure

Notes

if plotting fails, this prints traceback and logs error but doesn’t raise and exception itself, returning None instead

desispec.scripts.tileqa.func(night, tileid, specprod_dir, exposure_qa_dir, outfile=None, group='cumulative')[source]

Wrapper function to compute_tile_qa for multiprocessing

desispec.scripts.trace_shifts

desispec.scripts.update_spectra

Update healpix-grouped spectra from a set of input cframe files

desispec.scripts.updateexptables

desispec.scripts.updateexptables.update_exposure_tables(nights=None, night_range=None, path_to_data=None, exp_table_path=None, obstypes=None, orig_filetype='csv', out_filetype='csv', verbose=False, no_specprod=False, dry_run=False)[source]

Generates updated exposure tables for the nights requested. Requires exposure tables to exist on disk.

Parameters:
  • nights – str, int, or comma separated list. The night(s) to generate procesing tables for.

  • night_range – str. comma separated pair of nights in form YYYYMMDD,YYYYMMDD for first_night,last_night specifying the beginning and end of a range of nights to be generated. last_night should be inclusive.

  • path_to_data – str. The path to the raw data and request*.json and manifest* files.

  • exp_table_path – str. Full path to where to exposure tables should be saved, WITHOUT the monthly directory included.

  • obstypes – str. The exposure OBSTYPE’s that you want to include in the exposure table. Can be a comma separated list.

  • orig_filetype – str. The file extension (without the ‘.’) of the exposure tables.

  • out_filetype – str. The file extension for the outputted exposure tables (without the ‘.’).

  • verbose – boolean. Whether to give verbose output information or not. True prints more information.

  • no_specprod – boolean. Create exposure table in repository location rather than the SPECPROD location

Returns:

Nothing

Combine individual redrock files into a single zcatalog

Stephen Bailey Lawrence Berkeley National Lab Fall 2015

substantially updated Fall 2023

desispec.scripts.zcatalog._wrap_read_redrock(optdict)[source]

read_redrock wrapper to expand dictionary of named args for multiprocessing

desispec.scripts.zcatalog.load_sv1_ivar_w12(hpix, targetids)[source]

Load FLUX_IVAR_W1/W2 from sv1 target files for requested targetids

Parameters:
  • hpix (int) – nside=8 nested healpix

  • targetids (array) – TARGETIDs to include

Returns table of TARGETID, FLUX_IVAR_W1, FLUX_IVAR_W2

Note: this is only for the special case of sv1 dark/bright and the FLUX_IVAR_W1/W2 columns which were not included in fiberassign for tiles designed before 20201212.

Note: nside=8 nested healpix is hardcodes for simplicity because that is what was used for sv1 target selection and this is not trying to be a more generic targetid lookup function.

desispec.scripts.zcatalog.read_redrock(rrfile, group=None, recoadd_fibermap=False, minimal=False, pertile=False, counter=None)[source]

Read a Redrock file, combining REDSHIFTS, FIBERMAP, and TSNR2 HDUs

Parameters:

rrfile (str) – full path to redrock filename

Options:

group (str): add group-specific columns for cumulative, pernight, healpix readcoadd_fibermap (bool): recoadd fibermap from spectra file in same dir minimal (bool): only propagate minimal subet of columns pertile (bool): input Redrock file is single tile (not healpix) counter (tuple): (i,n) log loading ith file out of n

Returns (zcat, expfibermap) where zcat is a join of the redrock REDSHIFTS catalog and the coadded FIBERMAP

desispec.scripts.zproc

One stop shopping for redshifting DESI spectra

desispec.scripts.zproc.distribute_ranks_to_blocks(nblocks, rank=None, size=None, comm=None, log=None, split_comm=False)[source]

Function to split up a set of ranks of size ‘size’ into nblock number of blocks or roughly equal size.

Parameters:
  • nblocks (int) – the number of blocks to split the ranks into

  • rank (int) – the MPI world rank

  • size (int) – the number of world MPI ranks

  • comm (object) – MPI communicator

  • log (object) – logger

  • split_comm (bool) – whether to split the world communicator into blocks and return the block communicator

Returns:

A tuple containing:

  • nblocks, int: the achievable number of block based on size

  • block_size, int: the number of ranks in the assigned block of current rank

  • block_rank. int: the rank in the assigned block of the current rank

  • block_num, int: the block number (of nblocks blocks) in which the rank was assigned

  • block_comm (optional): if split_comm is true, returns a communicator of only the ranks in the current block. Splits from the world communicator

Return type:

tuple

desispec.scripts.zproc.parse(options=None)[source]

Parses an argparser object for use with desi_zproc and returns arguments

desispec.scripts.zprocdashboard

desispec.scripts.zprocdashboard.main(args=None)[source]

Code to generate a webpage for monitoring of desi_dailyproc production status Usage: -n can be ‘all’ or series of nights separated by comma or blank like 20200101,20200102 or 20200101 20200102 Normal Mode: desi_proc_dashboard -n 3 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/ desi_proc_dashboard -n 20200101,20200102 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/

desispec.scripts.zprocdashboard.parse(options)[source]

Initialize the parser to read input

desispec.scripts.zprocdashboard.populate_night_zinfo(night, doem=True, doqso=True, dotileqa=True, check_on_disk=False, night_json_zinfo=None, skipd_tileids=None)[source]

For a given night, return the file counts and other other information for each exposure taken on that night input: night output: a dictionary containing the statistics with expid as key name FLAVOR: FLAVOR of this exposure OBSTYPE: OBSTYPE of this exposure EXPTIME: Exposure time SPECTROGRAPHS: a list of spectrographs used n_spectrographs: number of spectrographs n_psf: number of PSF files n_ff: number of fiberflat files n_frame: number of frame files n_sframe: number of sframe files n_cframe: number of cframe files n_sky: number of sky files

desispec.sky

Utility functions to compute a sky model and subtract it.

desispec.sky._model_variance(frame, cskyflux, cskyivar, skyfibers)[source]

look at chi2 per wavelength and increase sky variance to reach chi2/ndf=1

desispec.sky.calculate_throughput_corrections(frame, skymodel)[source]

Calculate the throughput corrections for each fiber based on the skymodel.

Parameters:
  • frame (Frame object) – frame containing the data that may need to be corrected

  • skymodel (SkyModel object) – skymodel object that contains the information about the sky for the given exposure/frame

Output:
corrections (1D array): 1D array where the index corresponds to the fiber % 500 and the values are the multiplicative corrections that would

be applied to the fluxes in frame.flux to correct them based on the input skymodel

desispec.sky.compute_sky(frame, nsig_clipping=4.0, max_iterations=100, model_ivar=False, add_variance=True, adjust_wavelength=False, adjust_lsf=False, only_use_skyfibers_for_adjustments=True, pcacorr=None, fit_offsets=False, fiberflat=None, skygradpca=None, min_iterations=5, tpcorrparam=None, exclude_sky_targetids=None, override_sky_targetids=None)[source]

Compute a sky model.

Sky[fiber,i] = R[fiber,i,j] Flux[j]

Input flux are expected to be flatfielded! We don’t check this in this routine.

Parameters:
  • frame – Frame object, which includes attributes - wave : 1D wavelength grid in Angstroms - flux : 2D flux[nspec, nwave] density - ivar : 2D inverse variance of flux - mask : 2D inverse mask flux (0=good) - resolution_data : 3D[nspec, ndiag, nwave] (only sky fibers)

  • nsig_clipping – [optional] sigma clipping value for outlier rejection

  • max_iterations – int, maximum number of iterations

  • model_ivar – replace ivar by a model to avoid bias due to correlated flux and ivar. this has a negligible effect on sims.

  • add_variance – evaluate calibration error and add this to the sky model variance

  • adjust_wavelength – adjust the wavelength of the sky model on sky lines to improve the sky subtraction

  • adjust_lsf – adjust the LSF width of the sky model on sky lines to improve the sky subtraction

  • only_use_skyfibers_for_adjustments – interpolate adjustments using sky fibers only

  • pcacorr – SkyCorrPCA object to interpolate the wavelength or LSF adjustment from sky fibers to all fibers

  • fit_offsets – fit offsets for regions defined in calib

  • fiberflat – desispec.FiberFlat object used for the fit of offsets

  • skygradpca – SkyGradPCA object to use to fit sky gradients, or None

  • min_iterations – int, minimum number of iterations

  • tpcorrparam – TPCorrParam object to use to fit fiber throughput variations, or None

returns SkyModel object with attributes wave, flux, ivar, mask

desispec.sky.get_sky_fibers(fibermap, override_sky_targetids=None, exclude_sky_targetids=None)[source]

Retrieve the fiber indices of sky fibers

Parameters:

fibermap – Table from frame FIBERMAP HDU (frame.fibermap)

Options:

override_sky_targetids (array of int): TARGETIDs to use, overriding fibermap exclude_sky_targetids (array of int): TARGETIDs to exclude

Returns:

array of indices of sky fibers to use

By default we rely on fibermap[‘OBJTYPE’]==’SKY’, but we can also exclude some targetids by providing a list of them through exclude_sky_targetids or by just providing all the sky targetids directly (in that case the OBJTYPE information is ignored)

desispec.sky.qa_skysub(param, frame, skymodel, quick_look=False)[source]

Calculate QA on SkySubtraction

Note: Pixels rejected in generating the SkyModel (as above), are not rejected in the stats calculated here. Would need to carry along current_ivar to do so.

Parameters:
  • param – dict of QA parameters : see qa_frame.init_skysub for example

  • frame – desispec.Frame object; Should have been flat fielded

  • skymodel – desispec.SkyModel object

  • quick_look – bool, optional If True, do QuickLook specific QA (or avoid some)

Returns:

dict of QA outputs

Need to record simple Python objects for yaml (str, float, int)

Return type:

qadict

desispec.sky.subtract_sky(frame, skymodel, apply_throughput_correction_to_lines=True, apply_throughput_correction=False, zero_ivar=True)[source]

Subtract skymodel from frame, altering frame.flux, .ivar, and .mask

Parameters:
  • frame – desispec.Frame object

  • skymodel – desispec.SkyModel object

Option:
apply_throughput_correctionif True, fit for an achromatic throughput

correction. This is to absorb variations of Focal Ratio Degradation with fiber flexure. This applies the residual throughput corrections on top of the model throughput corrections already included in the sky model.

zero_ivar : if True , set ivar=0 for masked pixels

desispec.skycorr

Class with sky model corrections.

desispec.skygradpca

Class with sky model gradient templates.

desispec.skygradpca.compute_pcs(skies, topn=6, niter=5)[source]

Computes principal components of sky spectra.

This computes the PCs used for the skygradpca product.

Parameters:
  • skies – output of gather skies; the sky spectra to do PCA on

  • topn – (optional int) number of PCA to compute

  • niter – (optional int) number of iterations to do, rejecting discrepant pixels.

Returns:

Output of np.linalg.svd on the clipped, mean-subtracted sky spectra.

desispec.skygradpca.configure_for_xyr(skygradpca, x, y, R, skyfibers=None)[source]

Add additional information to a SkyGradPCA object for fitting.

A SkyGradPCA object contains the templates used to fit out sky gradients. When these templates are actually used in fitting, it’s helpful to additionally know things like the locations of the fibers on the particular exposure and the mean resolution matrix for the exposure, etc. This function mutates the skygradpca object, adding new attributes for fitting.

Parameters:
  • skygradpca – SkyGradPCA object to extend

  • x – 1D[nfiber] array of x fiber coordinates on exposure (mm)

  • y – 1D[nfiber] array of y fiber coordinates on exposure (mm)

  • R – list of resolution matrices for each fiber (unitless)

  • skyfibers – (optional) indices of sky fibers in list of all fibers

desispec.skygradpca.doall(specprod='fuji')[source]

Perfoms and writes out analysis of sframe sky residuals.

Gathers all of the sframe residual sky spectra for each camera/petal, runs a PCA on them, makes the appropriate SkyGradPCA objects, and writes them out to DESI_SPECTRO_CALIB.

Parameters:

specprod (str, optional) – specprod to run on

desispec.skygradpca.evaluate_model(skygradpca, R, coeff, mean=None)[source]

Evaluate a skygradpca object for a frame.

The skygradpca object must first be configured for the frame with configure_for_xyr.

Parameters:
  • skygradpca – SkyGradPCA instance, configured with configure_for_xyr

  • R – resolution matrices for frame

  • coeff (array, 1D) – coefficients for eigenvectors

  • mean (optional) – mean sky spectrum

Returns:

sky spectrum (array[nfiber, nwave]) given mean spectrum, per-fiber resolution matrices, and gradient amplitudes

desispec.skygradpca.gather_skies(fn=None, camera='r', petal=0, n=numpy.inf, heliocor=True, specprod='daily')[source]

Gather sframes for performing bulk analyses; e.g., PCA.

This function gathers residual sky spectra together for many exposures, bundling it together for later analysis.

Parameters:
  • fn – (list[str], optional) sframe file names to gather If not set, all of the sframes in the specprod product in the camera camera and petal petal will be gathered.

  • camera – (str, optional) camera to gather

  • petal – (int, optional) petal to gather

  • n – (int, optional) only gather at most this many exposures worth of sky spectra

  • heliocor – (bool, optional) if True, apply correction to sframe files to bring them to earth frame

Returns:

ndarray with EXPID, FIBER, CAMERA, X, Y, WAVE, FLUX, IVAR for each sky fiber from sframe files.

desispec.skygradpca.make_all_pcs(specprod, minnight=20200101, cameras='brz', petals=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], **kw)[source]

Makes SkyGradPCA objects derived from residual skies from a specprod.

This grabs all of the sframe files passing certain criteria using gather_skies and runs a PCA on them, producing SkyGradPCA objects for each camera/petal. It uses 30 processes, one for each camera/petal combination.

Parameters:
  • specprod (str) – specprod to use

  • minnight (int, optional) – use only spectra taken after this date

  • cameras (str, optional) – string containing some of ‘brz’

  • petals (list[int], optional) – petals to use

  • **kw (dict, optional) – additional keywords passed to gather_skies.

Returns:

dictionary[camera, petal] containing the corresponding SkyGradPCA object.

desispec.skygradpca.make_all_pcs_by_filter(**kw)[source]

Wrapper for make_all_pcs, doing one filter at a time.

desispec.skygradpca.make_all_pcs_wrapper(args)[source]

Wrapper function; gathers skies for cam/petal combinations.

desispec.skygradpca.write_pcs(pcs)[source]

Writes SkyGradPCA objects to DESI_SPECTRO_CALIB.

Takes the output of make_all_pcs and writes it to DESI_SPECTRO_CALIB.

Parameters:

pcs (dict[camera, petal] of SkyGradPCA) – output of make_all_pcs

desispec.skymag

Utility function to compute the sky magnitude per arcmin2 based from the measured sky model of an exposure and a static model of the instrument throughput.

desispec.skymag._get_average_calibration(filename)[source]

Use a dictionnary referenced by a global variable to keep a copy of the calibration instead of reading it at each function call.

desispec.skymag.compute_skymag(night, expid, specprod_dir=None)[source]

Computes the sky magnitude for a given exposure. Uses the sky model and apply a fixed calibration for which the fiber aperture loss is well understood.

Parameters:
  • night – int, YYYYMMDD

  • expid – int, exposure id

  • specprod_dir – str, optional, specify the production directory. default is $DESI_SPECTRO_REDUX/$SPECPROD

Returns:

(gmag,rmag,zmag) AB magnitudes per arcsec2, tuple with 3 float values

desispec.specscore

Spectral scores routines.

desispec.specscore.compute_coadd_scores(coadd, specscores=None, update_coadd=True)[source]

Compute scores for a coadded Spectra object

Parameters:

coadd – a Spectra object from a coadd

Options:

update_coadd: if True, update coadd.scores specscores: scores Table from the uncoadded spectra including a TARGETID column

Returns tuple of dictionaries (scores, comments); see compute_frame_scores

specscores is used to update TSNR2 scores by summing inputs

desispec.specscore.compute_coadd_tsnr_scores(specscores)[source]

Compute coadded TSNR2 scores (TSNR2=Template Signal-to-Noise squared)

Parameters:

specscores – uncoadded scores with TSNR2* columns (dict or Table-like)

Returns (tsnrscores, comments) tuple of dictionaries

desispec.specscore.compute_frame_scores(frame, band=None, suffix=None, flux_per_angstrom=None)[source]

Computes scores in spectra of a frame.

The scores are sum,mean,medians in a predefined and fixed wavelength range for each DESI camera arm, or band, b, r or z. The band argument is optional because it can be automatically chosen from the wavelength range in the frame. The suffix is added to the key name in the output dictionnary, for instance ‘RAW’, ‘SKYSUB’, ‘CALIB’ … The boolean argument flux_per_angstrom is needed if there is no ‘BUNIT’ keyword in frame.meta (frame fits header)

Parameters:
  • frame (Frame or QFrame) – A Frame or a QFrame object.

  • band (str, optional) – Spectrograph band, b, r, z, autodetected by default.

  • suffix (str, optional) – Character string added to the keywords in the output dictionary, for instance suffix=’RAW’

  • flux_per_angstrom (bool, optional) – If True the spectra are assumed flux_per_angstrom, i.e. flux densities. If False, the spectra are assumed to be counts or photo-electrons per bin. None by default in which case the frame.units string is read to find out whether the flux quantity is per unit wavelenght or per bin.

Returns:

A tuple containg a dict of 1D arrays of size = number of spectra in frame and a dict of string with comments on the type of scores.

Return type:

tuple()

desispec.specstatus

Update surveyops/ops/tile-specstatus.ecsv with spectro pipeline tiles.csv.

desispec.specstatus.update_specstatus(specstatus, tiles, update_only=False, clear_qa=False)[source]

return new specstatus table, updated with tiles table

Parameters:
  • specstatus – astropy Table from surveyops/ops/tiles-specstatus.ecsv

  • tiles – astropy Table from spectro/redux/daily/tiles.csv

  • clear_qa – bool indicating whether QA data should be cleared

Returns: updated specstatus table, sorted by TILEID

New TILEID found in tiles are added to specstatus, and any TILEID already in specstatus have their non-QA columns updated from the entries in tiles.

If update_only==True, add any new tiles and update non-QA columns for tiles with new data (tiles[‘LASTNIGHT’] > specstatus[‘LASTNIGHT’]) but don’t change entries for reprocessed tiles that have no new data.

The QA-related columns that are not changed here are USER, QA, OVERRIDE, ZDONE, QANIGHT, and ARCHIVEDATE.

This does not modify either of the input tables.

desispec.spectra

Class for dealing with a group of spectra from multiple bands and the associated fibermap information.

class desispec.spectra.Spectra(bands=[], wave={}, flux={}, ivar={}, mask=None, resolution_data=None, fibermap=None, exp_fibermap=None, meta=None, extra=None, single=False, scores=None, scores_comments=None, extra_catalog=None)[source]

Represents a grouping of spectra.

This class contains an “extended” fibermap that has information about the night and exposure of each spectrum. For each band, this class has the wavelength grid, flux, ivar, mask, and resolution arrays.

Parameters:
  • bands (list) – List of strings used to identify the bands.

  • wave (dict) – Dictionary of arrays specifying the wavelength grid.

  • flux (dict) – Dictionary of arrays specifying the flux for each spectrum.

  • ivar (dict) – Dictionary of arrays specifying the inverse variance.

  • mask (dict, optional) – Dictionary of arrays specifying the bitmask.

  • resolution_data (dict, optional) – Dictionary of arrays specifying the block diagonal resolution matrix. The object for each band must be in one of the formats supported by the Resolution class constructor.

  • fibermap – Extended fibermap to use. If not specified, a fake one is created.

  • Table-like – Extended fibermap to use. If not specified, a fake one is created.

  • optional – Extended fibermap to use. If not specified, a fake one is created.

  • exp_fibermap – Exposure-specific fibermap columns, which may not apply to a coadd.

  • Table-like – Exposure-specific fibermap columns, which may not apply to a coadd.

  • optional – Exposure-specific fibermap columns, which may not apply to a coadd.

  • meta (dict, optional) – Dictionary of arbitrary properties.

  • extra (dict, optional) – Optional dictionary of dictionaries containing extra floating point arrays. The top-level is a dictionary over bands and each value is a dictionary containing string keys and values which are arrays of the same size as the flux array.

  • single (bool, optional) – If True, store flux,ivar,resolution data in memory as single precision (np.float32).

  • scores – QA scores table.

  • scores_comments – dict[column] = comment to include in output file

  • extra_catalog (numpy or astropy Table, optional) – optional table of metadata, rowmatched to fibermap, e.g. a redshift catalog for these spectra

_get_slice(index, bands=None)[source]

Slice spectra by index. :param bands: optional list of bands to select. :type bands: list

Note: This function is intended to be private,

to be used by __getitem__() and select().

property bands

the list of valid bands.

Type:

(list)

classmethod from_specutils(spectra)[source]

Convert specutils objects to a Spectra object.

Parameters:

spectra (specutils.Spectrum1D or specutils.SpectrumList) – A specutils object.

Returns:

The corresponding DESI-internal object.

Return type:

Spectra

Raises:
  • NameError – If specutils is not available in the environment.

  • ValueError – If an unknown type is found in spectra.

property ftype

the data type used for floating point numbers.

Type:

(numpy.dtype)

num_spectra()[source]

Get the number of spectra contained in this group.

Returns (int):

Number of spectra contained in this group.

num_targets()[source]

Get the number of distinct targets.

Returns (int):

Number of unique targets with spectra in this object.

select(nights=None, exposures=None, bands=None, targets=None, fibers=None, invert=False, return_index=False)[source]

Select a subset of the data.

This filters the data based on a logical AND of the different criteria, optionally inverting that selection.

Parameters:
  • nights (list) – optional list of nights to select.

  • exposures (list) – optional list of exposures to select.

  • bands (list) – optional list of bands to select.

  • targets (list) – optional list of target IDs to select.

  • fibers (list) – list/array of fiber indices to select.

  • invert (bool) – after combining all criteria, invert selection.

  • return_index (bool) – if True, also return the indices of selected spectra.

Returns:

a new Spectra object containing the selected data. indices (list, optional): indices of selected spectra. Only provided if return_index is True.

Return type:

spectra

target_ids()[source]

Return list of unique target IDs.

The target IDs are sorted by the order that they first appear.

Returns (array):

an array of integer target IDs.

to_specutils()[source]

Convert to specutils objects.

Returns:

A list with each band represented as a specutils.Spectrum1D object.

Return type:

specutils.SpectrumList

Raises:

NameError – If specutils is not available in the environment.

update(other)[source]

Overwrite or append new data.

Given another Spectra object, compare the fibermap information with the existing one. For spectra that already exist, overwrite existing data with the new values. For spectra that do not exist, append that data to the end of the spectral data.

Parameters:

other (Spectra) – the new data to add.

Returns:

nothing (object updated in place).

Note: if fibermap, scores and extra_catalog exist in the new data, they are appended to the existing tables. If those new tables have different columns, only columns with identical names will be appended. Spectra.meta is unchanged.

wavelength_grid(band)[source]

Return the wavelength grid for a band.

Parameters:

band (str) – the name of the band.

Returns (array):

an array containing the wavelength values.

desispec.spectra.stack(speclist)[source]

Stack a list of spectra, return a new spectra object

Parameters:

speclist – list of Spectra objects

returns stacked Spectra object

Note: all input spectra must have the same bands, wavelength grid, and include or not the same optional elements (mask, fibermap, extra, …). The returned Spectra have the meta from the first input Spectra.

Also see Spectra.update, which is less efficient but more flexible for handling heterogeneous inputs

desispec.tile_qa

Utility functions to compute an exposure QA scores (or is it tile?).

desispec.tile_qa.compute_tile_qa(night, tileid, specprod_dir, exposure_qa_dir=None, group='cumulative')[source]

Computes the exposure_qa

Parameters:
  • night (int) – YYYYMMDD

  • tileid (int) – tile id

  • specprod_dir (str) – specify the production directory. default is $DESI_SPECTRO_REDUX/$SPECPROD

  • exposure_qa_dir – str, optional, directory where the exposure qa are saved

  • group – str, “cumulative” or “pernight” tile group

Returns:

A tuple of astropy.table.Table:

  • fiberqa (with one row per target and at least a TARGETID column)

  • petalqa (with one row per petal and at least a PETAL_LOC column)

Return type:

tuple

desispec.tile_qa_plot

Utility functions to generate the tile QA png.

desispec.tile_qa_plot.assert_tracer(tracer)[source]

Asserts tracer in the list of allowed tracers.

Parameters:

tracer – tracer name (string, upper case)

Note

Will exit with an error if not asserted.

desispec.tile_qa_plot.deg2pix(dras, ddecs, width_deg, width_pix)[source]

Converts (dras,ddecs) to (xs,ys) in cutout img pixels.

Parameters:
  • dras – projected distance (degrees) along R.A. to the center of the cutout (np.array of floats)

  • ddecs – projected distance (degrees) along Dec. to the center of the cutout (np.array of floats)

  • width_deg – width of the cutout in degrees (np.array of floats)

  • width_pix – width of the cutout in pixels (np.array of floats)

Returns:

A tuple containing:

  • dxs: distance (pixels) along x to the center of the cutout (np.array of floats)

  • dys: distance (pixels) along y to the center of the cutout (np.array of floats)

Return type:

tuple

Notes

  • not sure at the <1 pixel level…

  • Duplicated from fiberassign.fba_launch_io.deg2pix()

desispec.tile_qa_plot.get_expid_vccdsec(expid, night, prod)[source]

Get the VCCDSEC header keyword for an exposure for each frame file.

Parameters:
  • expid – exposure id (int)

  • night – night (int)

  • prod – full path to input reduction, e.g. /global/cfs/cdirs/desi/spectro/redux/daily (string)

Returns:

dictionary with VCCDSEC value for each frame file (dict. of floats) pb_campets: list of {camera}{petal} which have VCCDSEC < threshold

Return type:

vccdsec

Notes

  • The VCCDSEC threshold is set in desispec/data/qa/qa-params.yaml

  • If VCCDSEC is not present for a given frame file, returns -99s

    (so that it gets attention)

  • If no frame files, returns an empty dictionary

desispec.tile_qa_plot.get_expids_efftimes(tileqafits, prod)[source]

Get the EFFTIME and EFFTIMEQA for the EXPIDs from the coadd.

Parameters:
  • tileqafits – path to the tile-qa-TILEID-NIGHT.fits file

  • prod – full path to input reduction, e.g. /global/cfs/cdirs/desi/spectro/redux/daily (string)

Returns:

An array with the following keys:

EXPID, NIGHT, EFFTIME_SPEC, QA_EFFTIME_SPEC

Return type:

structured array

Notes

  • We work from the spectra-*.fits files; if not present in the same folder as tileqafits, we look into the expected path using prod.

  • As this is run before desi_tsnr_afterburner, we compute here the EFFTIME_SPEC values.

  • If no GOALTYPE in tileqafits header, we default to dark.

  • TBD: we purposely do not use TSNR2 keys from qa-params.yaml, as those do not handle the TSNR2_ELG->TSNR2_LRG change from 2021 shutdown. We use:

    • dark before 20210901: TSNR2_ELG

    • dark after 20210901: TSNR2_LRG

    • bright: TSNR2_BGS

    • backup: TSNR2_BGS

    • Method assessed against all Main exposures until 20211013 in daily tsnr-exposures.fits.

desispec.tile_qa_plot.get_petalqa_props(key)[source]

For a given key, returns specific properties, to display diagnoses on the QA plot.

Parameters:

key – column name present in the PETALQA extension of qa-tile-TILEID-NIGHT.fits (string)

Returns:

A tuple containing:

  • short: shortname of the key (string)

  • precision: number of digits to report (int)

  • okmin: minimum value to be considered as ok (float)

  • okmax: maximum value to be considered as ok (float)

  • combine: None, “sum”, “mean”, i.e. how to combine the stats for all petals (string)

Return type:

tuple

desispec.tile_qa_plot.get_qa_badmsks()[source]

Returns the bitmask values for bad_qafstatus_mask and bad_petal_mask..

Parameters:

None

Returns:

The bitmask values for bad_qafstatus_mask and bad_petal_mask.

desispec.tile_qa_plot.get_qa_config()[source]

Reads the configuration file (data/qa/qa-params.yaml)

Parameters:

None.

Returns:

Content of the qa-params.yaml file

desispec.tile_qa_plot.get_quantz_cmap(name, n, cmin=0, cmax=1)[source]

Creates a quantized colormap.

Parameters:
  • name – matplotlib colormap name (e.g. “tab20”) (string)

  • n – number of colors

  • cmin (optional, defaults to 0) – first color of the original colormap to use (between 0 and 1) (float)

  • cmax (optional, defaults to 1) – last color of the original colormap to use (between 0 and 1) (float)

Returns:

A matplotlib cmap object.

Notes

https://matplotlib.org/examples/api/colorbar_only.html

desispec.tile_qa_plot.get_radec_mw(ras, decs, org)[source]

Converts R.A. and Dec. coordinates to be used in a Mollweide projection.

Parameters:
  • ras – R.A. coordinates in degrees (float array)

  • decs – Dec. coordinates in degrees (float array)

Returns:

Two float arrays with transformed R.A. and Dec.

desispec.tile_qa_plot.get_tilecov(tileid, surveys='main', programs=None, lastnight=None, indesi=True, nside=1024, outpng=None, plot_tiles=False, verbose=False)[source]

Computes the average number of observed tiles covering a given tile.

Parameters:
  • tileid (int) – tileid

  • surveys (optional, defaults to "main") – comma-separated list of surveys to consider (reads the tiles-SURVEY.ecsv file) (str)

  • programs (optional, defaults to None) – comma-separated list of programs (case-sensitive) to consider in the tiles-SURVEY.ecsv file (str)

  • lastnight (optional, defaults to today) – only consider tiles observed up to lastnight (int)

  • surveys – comma-separated list of surveys to consider (reads the tiles-SURVEY.ecsv file) (str)

  • indesi (optional, defaults to True) – restrict to IN_DESI=True tiles? (bool)

  • nside (optional, defaults to 1024) – healpix pixel nside (int)

  • outpng (optional, defaults to None) – if provided, output file with a plot (str)

  • plot_tiles (optional, defaults to False) – plot overlapping tiles? (bool)

  • verbose (optional, defaults to False) – print log.info() (bool)

Returns:

A tuple containing:

  • pixs: list of the healpix pixels covering the tile (np.array(float))

  • pix_ntiles: list of the nb of tiles covering each pixel from pixs (np.array(float))

  • nside: healpix pixel nside (int)

  • nest: healpix NESTED? (boolean)

  • outdict: a dictionary, with an entry for each observed, overlapping tile, containing the list of observed overlapping tiles (dict)

Return type:

tuple

Notes

  • The “regular” use is to provide a single PROGRAM in programs (e.g., programs=”DARK”).

  • This function relies on the following files:

    • $DESI_SURVEYOPS/ops/tiles-{SURVEY}.ecsv for SURVEY in surveys (to get the tiles to consider)

    • $DESI_ROOT/spectro/redux/daily/exposures-daily.fits (to get the existing observations up to lastnight)

  • If one wants to consider the latest observations, one should wait the 10am pacific update of exposures-daily.fits.

desispec.tile_qa_plot.get_tracer(tracer, d, fstatus_key='QAFIBERSTATUS')[source]
For a given tracer, returns the selection used for the tile QA n(z):
  • (fstatus_key & bad_qafstatus_mask) == 0

  • desi_mask or bgs_mask

  • if ELG_LOP: additional cut on (d[“DESI_TARGET”] & desi_mask[“QSO”]) == 0

Parameters:
  • tracer – “BGS_BRIGHT”, “BGS_FAINT”, “LRG”, “ELG_LOP”, or “QSO” (string)

  • d – structured array with at least FIBERSTATUS, DESI_TARGET, BGS_TARGET

  • fstatus_key (optional, defaults to QAFIBERSTATUS) – key to use as FIBERSTATUS (string)

Returns:

selected tracer sample (boolean array)

Return type:

sel

desispec.tile_qa_plot.get_tracer_zminmax(tracer)[source]

Returns some fiducial redshift range per tracer to compute basic stats.

Parameters:

tracer – tracer name (see get_tracer_names()) (string)

Returns:

A tuple containing:

  • zmin: minimum redshift (float)

  • zmax: minimum redshift (float)

Return type:

tuple

desispec.tile_qa_plot.get_tracer_zok(tracer, dchi2_min, d, fstatus_key='QAFIBERSTATUS')[source]
For a given tracer, returns the spectro. valid sample for the tile QA n(z):
  • (fstatus_key & bad_qafstatus_mask) == 0

  • DELTACHI2 > dchi2_min

  • if QSO: additional cut on SPECTYPE=”QSO”

Parameters:
  • tracer – “BGS_BRIGHT”, “BGS_FAINT”, “LRG”, “ELG_LOP”, or “QSO” (string)

  • dchi2_min – minimum DELTACHI2 value (read from YAML file)

  • d – structured array with at least DELTACHI2, SPECTYPE

  • fstatus_key (optional, defaults to QAFIBERSTATUS) – key to use as FIBERSTATUS (string)

Returns:

spectro. valid sample (boolean array)

Return type:

sel

desispec.tile_qa_plot.get_viewer_cutout(tileid, tilera, tiledec, tmpoutdir='/tmp/tmpqw_r0v8g', width_deg=4, pixscale=10, dr='dr9', timeout=15)[source]

Downloads a cutout of the tile region from legacysurvey.org/viewer.

Parameters:
  • tileid – TILEID (int)

  • tilera – tile center R.A. (float)

  • tiledec – tile center Dec. (float)

  • tmpoutdir (optional, defaults to a temporary directory) – temporary directory where

  • width_deg (optional, defaults to 4) – width of the cutout in degrees (float)

  • pixscale (optional, defaults to 10) – pixel scale of the cutout

  • dr (optional, default do "dr9") – imaging data release

  • timeout (optional, defaults to 15) – time (in seconds) after which we quit the wget call (int)

Returns:

output of mpimg.imread() reading of the cutout (np.array of floats)

Return type:

img

Notes

  • Duplicating fiberassign.fba_launch_io.get_viewer_cutout()

  • 20220109 : adding a check on img dimension..

desispec.tile_qa_plot.get_zbins()[source]

Redshift grid for the tile QA n(z)

Parameters:

None

Returns:

The redshift grid (float array)

desispec.tile_qa_plot.get_zhists(tileids, tracer, dchi2_min, d, fstatus_key='QAFIBERSTATUS', tileid_key=None)[source]

Returns the fractional, per tileid, n(z) for a given tracer.

Parameters:
  • tileids – int or list or numpy array of int

  • tracer – one of the tracers defined in get_tracer_names() (string)

  • dchi2_min – DELTACHI2 threshold for a valid redshift

  • d – structured array with at least FIBERSTATUS, DESI_TARGET, BGS_TARGET, DELTACHI2, SPECTYPE, Z

  • fstatus_key (optional, defaults to QAFIBERSTATUS) – key to use as FIBERSTATUS (string)

  • tileid_key (optional, defaults to None) – column name for TILEID (string)

Returns:

A tuple containing:

  • bins: the redshift bin grid (float array)

  • zhists: fractional, per tileid, n(z) (numpy array of shape (nbin) if tileids is int or has length=1 (nbin, len(tileids)) else.

Return type:

tuple

Notes

  • If tileid_key is not provided, assumes all spectra are from the same tile.

  • If tileid_key is provided, will identify the tileids with np.unique(d[tileid_key]).

desispec.tile_qa_plot.make_tile_qa_plot(tileqafits, prod, pngoutfile=None, dchi2_min=None, tsnr2_key=None, refdir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/desispec/checkouts/stable/py/desispec/data/qa'))[source]

Generate the tile QA png file. Will replace .fits by .png in tileqafits for the png filename.

Parameters:
  • tileqafits – path to the tile-qa-TILEID-NIGHT.fits file

  • prod – full path to input reduction, e.g. /global/cfs/cdirs/desi/spectro/redux/daily (string)

  • pngoutfile (optional) – output filename; default to tileqafits .fits -> .png

  • dchi2_min (optional, defaults to value in qa-params.yaml) – minimum DELTACHI2 for a valid zspec (float)

  • tsnr2_key (optional, defaults to value in qa-params.yaml) – TSNR2 key used for plot (string)

  • refdir (optional, defaults to "desispec","data/qa") – path to folder with reference measurements for the n(z) and the TSNR2 (string)

Note

  • If hdr[“SURVEY”] is not “main”, will not plot the n(z).

  • If hdr[“FAPRGRM”].lower() is not “bright” or “dark”, will not plot the TSNR2 plot nor the skymap.

  • 20220109 : add safety around plot_cutout() call.

desispec.tile_qa_plot.plot_cutout(ax, tileid, tilera, tiledec, width_deg, petal_c='w', ebv_c='orange')[source]

Plots a ls-dr9 cutout, with overlaying the petals and the EBV contours.

Parameters:
  • ax – pyplot object

  • tileid – TILEID (int)

  • tilera – tile center R.A. (float)

  • tiledec – tile center Dec. (float)

  • width_deg – width of the cutout in degrees (np.array of floats)

  • petal_c (optional, defaults to "w") – color used to display petals (string)

  • ebv_c (optional, default to "y") – color used to display the EBV contours (string)

Notes

Different than fiberassign.fba_launch_io.plot_cutout().

desispec.tile_qa_plot.plot_mw_skymap(fig, ax, tileid, tilera, tiledec, survey, program, org=120)[source]

Plots the target density sky map with a Mollweide projection, with highlighting the tile position.

Parameters:
  • fig – pyplot figure object

  • ax – pyplot object with projection=”mollweide”

  • tileid – TILEID (int)

  • tilera – tile R.A. in degrees (float)

  • tiledec – tile Dec. in degrees (float)

  • survey – usually “main”, “sv1”, “sv3” (string)

  • program – “dark” or “bright” (string)

  • org – R.A. at the center of the plot (default=120) (float)

Note

If survey is not “main” and program is not “bright” or “dark”, will color-code EBV, not the target density.

desispec.tile_qa_plot.print_petal_infos(ax, petalqa, fiberqa)[source]

Print some diagnoses for each petal, and for the whole tile.

Parameters:
  • ax – pyplot object

  • petalqa – the PETALQA extension data of the tile-qa-TILEID-NIGHT.fits file

desispec.tile_qa_plot.set_mwd(ax, org=120)[source]

Prepare a plot for a Mollweide map

Parameters:
  • ax – a plt.subplots() instance

  • org – R.A. at the center of the plot (default=120) (float)

desispec.tilecompleteness

Routines to determine the survey progress and tiles completion.

desispec.tilecompleteness.compute_tile_completeness_table(exposure_table, specprod_dir, auxiliary_table_filenames, min_number_of_petals=8)[source]

Computes a summary table of the observed tiles

Parameters:
  • exposure_table – astropy.table.Table with exposure summary table from a prod (output of desi_tsnr_afterburner)

  • specprod_dir – str, production directory name

  • auxiliary_table_filenames – list(str), list of auxiliary table names, optional

  • min_number_of_petals – int, minimum number of petals to declare a tile done

Returns: astropy.table.Table with one row per TILEID, with completeness and exposure time.

desispec.tilecompleteness.merge_tile_completeness_table(previous_table, new_table)[source]

Merges tile summary tables.

Parameters:
  • previous_table – astropy.table.Table

  • new_table – astropy.table.Table

Returns: astropy.table.Table with merged entries.

desispec.tpcorrparam

This module implements a model for how the throughput of a fiber varies.

It has the following components:

  1. We find that the fibers don’t follow the nighly flat field exactly, and apply a fixed mean correction from the nightly flat field. This is thought to reflect the different range of angles entering the fiber from the flat field relative to the sky.

  2. We find that fibers’ throughput depends modestly on the fibers’ location within their patrol radii. We apply a cubic polynomial in (x, y) within the patrol disk to correct for this.

  3. We find additional coherent variations in throughput from exposure to exposure. These are modeled with a PCA and fit exposure-by-exposure.

The TPCorrParam object has mean, spatial, and pca attributes describing each of these three components.

desispec.tpcorrparam.cubic(p, x, y)[source]

Returns a cubic polynomial with coefficients p in x & y.

Parameters:
  • p (list[float]) – 10 element floating point list with poly coefficients

  • x (ndarray[float]) – x coordinates

  • y (ndarray[float]) – y coordinates

Returns:

cubic polynomial in two variables with coefficients p evaluated at x, y

desispec.tpcorrparam.fit_tpcorr_per_fiber(tpcorr)[source]

Fit the spatial throughput variations for each fiber.

Parameters:

tpcorr (ndarray) – output from gather_tpcorr with tpcorr measurements and related metadata for each fiber

Returns:

X (float): central X position of each fiber used in fit (mm) Y (float): central Y position of each fiber used in fit (mm) FIBER (int): fiber number PARAM (ndarray[float], length 10): 2D cubic polynomial coefficients

Return type:

dict[camera] of ndarray including the following fields

desispec.tpcorrparam.gather_dark_tpcorr(specprod=None)[source]

Gathers TPCORR for dark exposures expected to be well behaved.

This wraps gather_tpcorr, making a sensible selection of dark exposures from the given specprod.

Parameters:

specprod (str) – specprod to use

Returns:

ndarray from gather_tpcorr with tpcorr information from selected exposures.

desispec.tpcorrparam.gather_tpcorr(recs)[source]

Gather TPCORR measurements for exposures.

Parameters:

recs (ndarray) – ndarray array containing information about exposures from which to gather TPCORR information. Includes at least the following fields: NIGHT (int): night on which exposure was observed EXPID (int): exposure id

Returns: ndarray with the following fields for each exposure

X: x coordinate of fiber (mm) Y: y coordinate of fiber (mm) TPCORR: tpcorr of fiber CAMERA: camera for exposure NIGHT: night for exposure EXPID: expid of exposure

desispec.tpcorrparam.make_tpcorrparam_files(tpcorr=None, spatial=None, dirname=None)[source]

Gather, fit, and write tpcorrparam files to output directory.

This function is intended for producing tpcorrparam files to DESI_SPECTRO_CALIB. It can gather and fit tpcorr measurements and write the results.

Parameters:
  • tpcorr (optional) – result of gather_tpcorr. If None, this is populated with the result of gather_dark_tpcorr.

  • spatial (optional) – spatial fit results. If None, this is populated with fit_tpcorr_per_fiber(tpcorr).

  • dirname (optional) – directory to which to write files. Defaults to DESI_SPECTRO_CALIB.

desispec.tpcorrparam.pca_tpcorr(tpcorr)[source]

Run a PCA on TPCORR measurements.

This uses the tpcorr gathered by gather_tpcorr and runs a PCA on them.

Parameters:

tpcorr (ndarray) – result of gather_tpcorr

Returns:

A dictionary containing:

  • dict[filter] of (tpcorrmed, pc_info, exp_info), describing the results of the fit.

  • tpcorrmed: the median TPCORR of the whole sample; i.e., how different the nightly flat usually is from the sky.

  • pc_info: ndarray with:

    • pca: the principal components

    • amplitude: their corresponding singular values

  • exp_info: ndarray with:

    • expid: exposure id

    • coeff: pc component coefficients for this exposure

Return type:

dict

desispec.tpcorrparam.tpcorrmodel(tpcorrparam, xfib, yfib, pcacoeff=None)[source]

Evaluates full TPCORR throughput model.

The full model consistents of a PCA and a spatial within-patrol-radius component. These two components are evaluated and summed.

Parameters:
  • tpcorrparam – TPCorrParam object describing the model

  • xfib – x coordinates of fibers (mm)

  • yfib – y coordinates of fibers (mm)

  • pcacoeff (optional) – PCA coefficients of throughput model If not set, no TPCORR PCA terms are fit.

Returns:

Throughput model evaluated at xfib, yfib with pca coefficients pcacoeff.

desispec.tpcorrparam.tpcorrspatialmodel(param, xfib, yfib)[source]

Evaluate spatial tpcorr throughput model at xfib, yfib.

Evaluates the throughput model at xfib, yfib. param carries the polynomial coefficients of the model as well as the central fiber location.

Parameters:
  • param – ndarray describing fit. Contains at least the following fields: PARAM: cubic polynomial coefficients X: central fiber X location (mm) Y: central fiber Y location (mm)

  • xfib – x location at which model is to be evaluated (mm)

  • yfib – y location at which model is to be evaluated (mm)

Returns:

throughput model evaluated at xfib, yfib

desispec.trace_shifts

desispec.trace_shifts.boxcar_extraction_from_filenames(image_filename, psf_filename, fibers=None, width=7)[source]

Fast boxcar extraction of spectra from a preprocessed image and a trace set

Parameters:
  • image_filename – input preprocessed fits filename

  • psf_filename – input PSF fits filename

Optional:

fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : extraction boxcar width, default is 7

Returns:

2D np.array of shape (nfibers,n0=image.shape[0]), sum of pixel values per row of length=width per fiber ivar : 2D np.array of shape (nfibers,n0), ivar[f,j] = 1/( sum_[j,b:e] (1/image.ivar) ), ivar=0 if at least 1 pixel in the row has image.ivar=0 or image.mask!=0 wave : 2D np.array of shape (nfibers,n0), determined from the traces

Return type:

flux

desispec.trace_shifts.compute_dx_dy_using_psf(psf, image, fibers, lines)[source]

Computes trace shifts along x and y from a preprocessed image, a PSF (with trace coords), and a set of emission lines, by doing a forward model of the image. Calls compute_fiber_bundle_trace_shifts_using_psf.

Parameters:
  • psf – specter psf object

  • image – DESI preprocessed image object

  • fibers – 1D array with list of fibers

  • lines – 1D array of wavelength of emission lines (in Angstrom)

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD dy : 1D array of shifts along y coordinates on CCD sx : 1D array of uncertainties on dx sy : 1D array of uncertainties on dy fiber : 1D array of fiber ID wave : 1D array of wavelength

Return type:

x

desispec.trace_shifts.compute_dx_from_cross_dispersion_profiles(xcoef, ycoef, wavemin, wavemax, image, fibers, width=7, deg=2, image_rebin=4)[source]

Measure x offsets from a preprocessed image and a trace set

Parameters:
  • xcoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to XCCD

  • ycoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to YCCD

  • wavemin – float

  • wavemax – float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1 used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])

  • image – DESI preprocessed image object

  • fibers – 1D np.array of int (default is all fibers, the first fiber is always = 0)

Optional:

width : extraction boxcar width, default is 5 deg : degree of polynomial fit as a function of y, only used to find and mask outliers image_rebin : rebinning of CCD rows to run faster (with rebin=4 loss of precision <0.01 pixel)

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD ex : 1D array of uncertainties on dx fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength

Return type:

x

desispec.trace_shifts.compute_dy_from_spectral_cross_correlation(flux, wave, refflux, ivar=None, hw=3.0, calibrate=False, prior_width_dy=None)[source]

Measure y offsets from two spectra expected to be on the same wavelength grid. refflux is the assumed well calibrated spectrum. A relative flux calibration of the two spectra is done internally.

Parameters:
  • flux – 1D array of spectral flux as a function of wavelenght

  • wave – 1D array of wavelength (in Angstrom)

  • refflux – 1D array of reference spectral flux

Optional:

ivar : 1D array of inverse variance of flux hw : half width in Angstrom of the cross-correlation chi2 scan, default=3A corresponding approximatly to 5 pixels for DESI prior_width_dy: applied sigma of the Gaussian prior on dy

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD ex : 1D array of uncertainties on dx fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength

Return type:

x

desispec.trace_shifts.compute_dy_from_spectral_cross_correlations_of_frame(flux, ivar, wave, xcoef, ycoef, wavemin, wavemax, reference_flux, n_wavelength_bins=4)[source]

Measures y offsets from a set of resampled spectra and a reference spectrum that are on the same wavelength grid. reference_flux is the assumed well calibrated spectrum. Calls compute_dy_from_spectral_cross_correlation per fiber

Parameters:
  • flux – 2D np.array of shape (nfibers,nwave)

  • ivar – 2D np.array of shape (nfibers,nwave) , inverse variance of flux

  • wave – 1D array of wavelength (in Angstrom) of size nwave

  • refflux – 1D array of reference spectral flux of size nwave

Optional:

n_wavelength_bins : number of bins along wavelength

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dy : 1D array of shifts along y coordinates on CCD ey : 1D array of uncertainties on dy fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength

Return type:

x

desispec.trace_shifts.compute_dy_using_boxcar_extraction(xytraceset, image, fibers, width=7, degyy=2)[source]

Measures y offsets (internal wavelength calibration) from a preprocessed image and a trace set using a cross-correlation of boxcar extracted spectra. Uses boxcar_extraction , resample_boxcar_frame , compute_dy_from_spectral_cross_correlations_of_frame

Parameters:
  • xytraceset – XYTraceset object

  • image – DESI preprocessed image object

Optional:

fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : int, extraction boxcar width, default is 7 degyy : int, degree of polynomial fit of shifts as a function of y, used to reject outliers.

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dy : 1D array of shifts along y coordinates on CCD ey : 1D array of uncertainties on dy fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength

Return type:

x

desispec.trace_shifts.compute_fiber_bundle_trace_shifts_using_psf(fibers, line, psf, image, maxshift=2.0)[source]

Computes trace shifts along x and y from a preprocessed image, a PSF (with trace coords), and a given emission line, by doing a forward model of the image.

Parameters:
  • fibers – 1D array with list of fibers

  • line – float, wavelength of an emission line (in Angstrom)

  • psf – specter psf object

  • image – DESI preprocessed image object

Optional:

maxshift : float maximum shift in pixels for 2D chi2 scan

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD dy : 1D array of shifts along y coordinates on CCD sx : 1D array of uncertainties on dx sy : 1D array of uncertainties on dy

Return type:

x

desispec.trace_shifts.legx(wave, wavemin, wavemax)[source]

Reduced coordinate (range [-1,1]) for calls to legval and legfit

Parameters:
  • wave – ND np.array

  • wavemin – float, min. val

  • wavemax – float, max. val

Returns:

array of same shape as wave

desispec.trace_shifts.monomials(x, y, degx, degy)[source]

Computes monomials as a function of x and y of a 2D polynomial of degrees degx and degy

Parameters:
  • x – ND array

  • y – ND array of same shape as x

  • degx – int (>=0), polynomial degree along x

  • degy – int (>=0), polynomial degree along y

Returns :

monomials : ND array of shape ( (degx+1)*(degy+1) , x shape )

desispec.trace_shifts.polynomial_fit(z, ez, xx, yy, degx, degy)[source]

Computes and 2D polynomial fit of z as a function of (x,y) of degrees degx and degy

Parameters:
  • z – ND array

  • ez – ND array of same shape as z, uncertainties on z

  • x – ND array of same shape as z

  • y – ND array of same shape as z

  • degx – int (>=0), polynomial degree along x

  • degy – int (>=0), polynomial degree along y

Returns:

1D array of size (degx+1)*(degy+1) with polynomial coefficients (as defined by routine monomials) covariance : 2D array of covariance of coeff error_floor : float , extra uncertainty needed to get chi2/ndf=1 polval : ND array of same shape as z with values of pol(x,y) mask : ND array of same shape as z indicating the masked data points in the fit

Return type:

coeff

desispec.trace_shifts.recompute_legendre_coefficients(xcoef, ycoef, wavemin, wavemax, degxx, degxy, degyx, degyy, dx_coeff, dy_coeff)[source]

Modifies legendre coefficients of an input trace set using polynomial coefficents (as defined by the routine monomials)

Parameters:
  • xcoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to XCCD

  • ycoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to YCCD

  • wavemin – float

  • wavemax – float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1 used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])

  • degxx – int, degree of polynomial for x shifts as a function of x (x is axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction)

  • degxy – int, degree of polynomial for x shifts as a function of y (y is axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis)

  • degyx – int, degree of polynomial for y shifts as a function of x

  • degyy – int, degree of polynomial for y shifts as a function of y

  • dx_coeff – 1D np.array of polynomial coefficients of size (degxx*degxy) as defined by the routine monomials.

  • dy_coeff – 1D np.array of polynomial coefficients of size (degyx*degyy) as defined by the routine monomials.

Returns:

2D np.array of shape (nfibers,ncoef) with modified Legendre coefficents ycoef : 2D np.array of shape (nfibers,ncoef) with modified Legendre coefficents

Return type:

xcoef

desispec.trace_shifts.resample_boxcar_frame(frame_flux, frame_ivar, frame_wave, oversampling=2)[source]

Resamples the spectra in a frame obtained with boxcar extraction to the same wavelength grid, with oversampling. Uses resample_flux routine.

Parameters:
  • frame_flux – 2D np.array of shape (nfibers,nwave), sum of pixel values per row of length=width per fiber

  • frame_ivar – 2D np.array of shape (nfibers,nwave), ivar[f,j] = 1/( sum_[j,b:e] (1/image.ivar) ), ivar=0 if at least 1 pixel in the row has image.ivar=0 or image.mask!=0

  • frame_wave – 2D np.array of shape (nfibers,nwave), determined from the traces

Optional:

oversampling : int , oversampling factor , default is 2

Returns:

2D np.array of shape (nfibers,nwave*oversampling) ivar : 2D np.array of shape (nfibers,nwave*oversampling) frame_wave : 1D np.array of size (nwave*oversampling)

Return type:

flux

desispec.trace_shifts.shift_ycoef_using_external_spectrum(psf, xytraceset, image, fibers, spectrum_filename, degyy=2, width=7, prior_width_dy=0.1)[source]

Measure y offsets (external wavelength calibration) from a preprocessed image , a PSF + trace set using a cross-correlation of boxcar extracted spectra and an external well-calibrated spectrum. The PSF shape is used to convolve the input spectrum. It could also be used to correct for the PSF asymetry (disabled for now). A relative flux calibration of the spectra is performed internally.

Parameters:
  • psf – specter PSF

  • xytraceset – XYTraceset object

  • image – DESI preprocessed image object

  • fibers – 1D np.array of fiber indices

  • spectrum_filename – path to input spectral file ( read with np.loadtxt , first column is wavelength (in vacuum and Angstrom) , second column in flux (arb. units)

Optional:

width : int, extraction boxcar width, default is 7 degyy : int, degree of polynomial fit of shifts as a function of y, used to reject outliers. prior_width_dy: float with of the Gaussian prior on dy

Returns:

2D np.array of same shape as input, with modified Legendre coefficents for each fiber to convert wavelenght to YCCD

Return type:

ycoef

desispec.trace_shifts.write_traces_in_psf(input_psf_filename, output_psf_filename, xytraceset)[source]

Writes traces in a PSF.

Parameters:
  • input_psf_filename – Path to input fits file which has to contain XTRACE and YTRACE HDUs

  • output_psf_filename – Path to output fits file which has to contain XTRACE and YTRACE HDUs

  • xytraceset – xytraceset

desispec.tsnr

desispec.tsnr.calc_alpha(frame, fibermap, rdnoise_sigma, npix_1d, angperpix, angperspecbin, fiberflat, skymodel)[source]

Model Var = alpha * rdnoise component + sky.

Calculate the best-fit alpha using the sky fibers available to the frame.

Parameters:
  • frame – desispec frame instance (should be uncalibrated, i.e. e/A).

  • fibermap – desispec fibermap instance.

  • rdnoise_sigma – e.g. RDNOISE value per Quadrant (float).

  • npix_1d – equivalent to 1D nea [pixels], calculated using read_nea().

  • angperpix – angstroms per pixel (float),

  • fiberflat – desispec fiberflat instance.

  • skymodel – desispec Sky instance.

  • alpha – nuisanve parameter to reweight rdnoise vs sky contribution to variance (float).

  • components – if True, return individual contributions to variance, else return total variance.

Returns:

nuisance parameter to reweight rdnoise vs sky contribution to variance (float), obtained as the best fit to the uncalibrated sky fibers VAR.

Return type:

alpha

desispec.tsnr.calc_tsnr2(frame, fiberflat, skymodel, fluxcalib, alpha_only=False, include_poisson=True, include_fiberfracs=True)[source]

Compute template SNR^2 values for a given frame

Parameters:
  • frame – uncalibrated Frame object for one camera

  • fiberflat – FiberFlat object

  • sky – SkyModel object

  • fluxcalib – FluxCalib object

Returns

(tsnr2, alpha): tsnr2 dictionary, with keys labeling tracer (bgs,elg,etc.), of values holding nfiber length array of the tsnr^2 values for this camera, and alpha, the relative weighting btwn rdnoise & sky terms to model var.

Note

Assumes DESIMODEL is set and up to date.

desispec.tsnr.calc_tsnr2_cframe(cframe)[source]

Given cframe, calc_tsnr2 guessing frame,fiberflat,skymodel,fluxcalib to use

Parameters:

cframe – input cframe Frame object

Returns:

(results, alpha) from calc_tsnr2

Return type:

tuple

desispec.tsnr.calc_tsnr_fiberfracs(fibermap, etc_fiberfracs, no_offsets=False)[source]

Nominal fiberfracs for effective depths. See: https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed

Parameters:
  • fibermap – desispec instance.

  • etc_fiberfracs – propragated etc fiberfracs per tracer, json or fits header derived (dictionary).

  • no_offsets – ignore throughput loss due to fiber offsets.

Returns:

dict of the nominal fiberloss of given type and seeing.

desispec.tsnr.fb_rdnoise(fibers, frame, tset)[source]

Approximate the readnoise for a given fiber (on a given camera) for the wavelengths present in frame. wave.

Parameters:
  • fibers – e.g. np.arange(500) to index fiber.

  • frame – frame instance for the given camera.

  • tset – xytraceset object with fiber traces coordinates.

Returns:

(nfiber x nwave) array with the estimated readnosie. Same units as OBSRDNA, e.g. ang per pix.

Return type:

rdnoise

desispec.tsnr.gen_mask(frame, skymodel, hw=5.0)[source]

Generate a mask for the alpha computation, masking out bright sky lines.

Parameters:
  • frame – uncalibrated Frame object for one camera

  • skymodel – SkyModel object

  • hw – (optional) float, half width of mask around sky lines in A

Returns:

An array of same shape as frame, here mask=1 is good, 0 is bad

desispec.tsnr.get_ensemble(dirpath=None, bands=['b', 'r', 'z'], smooth=0)[source]

Function that takes a frame object and a bitmask and returns ivar (and optionally mask) array(s) that have fibers with offending bits in fibermap[‘FIBERSTATUS’] set to 0 in ivar and optionally flips a bit in mask.

Parameters:
  • dirpath – path to the dir. with ensemble dflux files. default is $DESIMODEL/data/tsnr

  • bands (list, optional) – bands to expect, typically [BRZ] - case ignored.

  • smooth (int, optional) – Further convolve the residual ensemble flux.

Returns:

Dictionary with keys labelling each tracer (bgs, lrg, etc.) for which each value is a Spectra class instance with wave, flux for BRZ arms. Note flux is the high frequency residual for the ensemble. See doc. 4723.

class desispec.tsnr.gfa_template_ensemble[source]
desispec.tsnr.read_nea(path)[source]

Read a master noise equivalent area [sq. pixel] file.

Parameters:

path – path to a master nea file for a given camera, e.g. b0.

Returns:

A tuple containing:

  • nea: 2D split object to be evaluated at (fiber, wavelength)

  • angperpix: 2D split object to be evaluated at (fiber, wavelength), yielding angstrom per pixel.

Return type:

tuple

class desispec.tsnr.template_ensemble(tracer, config_filename=None)[source]

Generate an ensemble of templates to sample tSNR for a range of points in (z, m, OII, etc.) space.

If conditioned, uses deepfield redshifts and (currently r) magnitudes to condition simulated templates.

compute(nmodel=5, smooth=100.0, nz_table_filename=None, single_mag=True, convolve_to_nz=True)[source]

Compute a template ensemble for template S/N measurements (tSNR)

Parameters:
  • nmodel – number of template models to generate

  • smooth – smoothing scale for dF=<F - smooth(F)>

  • nz_table_filename – path to ASCII file with columns zmin,zmax,n

  • single_mag – generate all templates at same average magnitude to limit MC noise

  • convolve_to_nz – if True, each template dF^2 is convolved to match the n(z) (redshift distribution)

effmag(m1, m2)[source]

returns an effective mag which is the magnitude that gives the same average flux^2 for the mag range specified [m1,m2] assuming a flat magnitude distribution

generate_templates(nmodel, redshifts=None, mags=None, single_mag=True)[source]

Dedicated wrapper for desisim.templates.GALAXY.make_templates call, stipulating templates in a redshift range suggested by the FDR. Further, assume fluxes close to the expected (within ~0.5 mags.) in the appropriate band.

Class init will write ensemble stack to disk at outdir, for a given tracer [bgs, lrg, elg, qso], having generated nmodel templates. Optionally, provide redshifts and mags. to condition appropriately at cost of runtime.

desispec.tsnr.tsnr2_to_efftime(tsnr2, target_type)[source]

Converts TSNR2 values to effective exposure time.

Parameters:
  • tsnr2 – TSNR**2 values, float or numpy array

  • target_type – str, “ELG”,”BGS”,”LYA”, or other depending on content of data/tsnr/tsnr-efftime.yaml

Returns:

Exptime in seconds, same type and shape if applicable as input tsnr2

desispec.tsnr.var_model(rdnoise_sigma, npix_1d, angperpix, angperspecbin, fiberflat, skymodel, alpha=1.0, components=False)[source]

Evaluate a model for the 1D spectral flux variance, e.g. quadrature sum of readnoise and sky components.

Parameters:
  • rdnoise_sigma

  • npix_1d – equivalent to (1D) nea.

  • angperpix – Angstroms per pixel.

  • angperspecbin – Angstroms per bin.

  • fiberflat – fiberflat instance

  • skymodel – Sky instance.

  • alpha – empirical weighting of the rdnoise term to e.g. better fit sky fibers per exp. cam.

  • components – if True, return tuple of individual contributions to the variance. Else return variance.

Returns:

nfiber x nwave array of the expected variance.

desispec.tsnr.var_tracer(tracer, frame, angperspecbin, fiberflat, fluxcalib, exposure_seeing_fwhm=1.1)[source]
Source Poisson term to the model ivar, following conventions defined at:

https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed.

Parameters:
  • tracer – [bgs, backup] string, defines program.

  • frame – desispec.frame instance

  • angperspecbin – float, angstroms per bin in spectral reductions

  • fiberflat – desispec instance

  • fluxcalib – desispec instance

  • fiber_diameter_arcsec

Returns:

nominal flux [e/specbin] corresponding to frame.wave

desispec.util

Utility functions for desispec.

desispec.util.argmatch(a, b)[source]

Returns indices ii such that a[ii] == b

Parameters:
  • a – array-like

  • b – array-like

Returns indices ii such that a[ii] == b

Both a and b are allowed to have repeats, and a values can be a superset of b, but b cannot contain values that are not in a because then no indices ii could result in a[ii] == b.

Related: desitarget.geomask.match_to which is similar, but doesn’t allow duplicates in b.

desispec.util.combine_ivar(ivar1, ivar2)[source]

Returns the combined inverse variance of two inputs, making sure not to divide by 0 in the process.

ivar1 and ivar2 may be scalar or ndarray but must have the same dimensions

desispec.util.dateobs2night(dateobs)[source]

Convert DATE-OBS ISO8601 UTC string to YEARMMDD int night of KPNO sunset

desispec.util.header2night(header)[source]

Return YEARMMDD night from FITS header, handling common problems

desispec.util.healpix_degrade_fixed(nside, pixel)[source]

Degrade a NEST ordered healpix pixel with a fixed ratio.

This degrades the pixel to a lower nside value that is fixed to half the healpix “factor”.

Parameters:
  • nside (int) – a valid NSIDE value.

  • pixel (int) – the NESTED pixel index.

Returns (tuple):

a tuple of ints, where the first value is the new NSIDE and the second value is the degraded pixel index.

desispec.util.itemindices(a)[source]

Return dict[key] -> list of indices i where a[i] == key

Parameters:

a – array-like of hashable values

Return dict[key] -> list of indices i where a[i] == key

The dict keys are inserted in the order that they first appear in a, and the value lists of indices are sorted

e.g. itemindices([10,30,20,30]) -> {10: [0], 30: [1, 3], 20: [2]}

desispec.util.mask32(mask)[source]

Return an input mask as unsigned 32-bit

Raises ValueError if 64-bit input can’t be cast to 32-bit without losing info (i.e. if it contains values > 2**32-1)

desispec.util.mjd2night(mjd)[source]

Convert MJD to YEARMMDD int night of KPNO sunset

desispec.util.mpi_count_failures(num_cmd, num_err, comm=None)[source]

Sum num_cmd and num_err across MPI ranks

Parameters:
  • num_cmd (int) – number of commands run

  • num_err (int) – number of failures

Options:

comm: mpi4py communicator

Returns:

sum(num_cmd), sum(num_err) summed across all MPI ranks

If comm is None, returns input num_cmd, num_err

desispec.util.night2ymd(night)[source]

parse night YEARMMDD string into tuple of integers (year, month, day)

desispec.util.option_list(opts)[source]

Convert key, value pairs into command-line options.

Parameters:

opts (dict-like) – Convert a dictionary into command-line options.

Returns:

A list of command-line options.

Return type:

list

desispec.util.ordered_unique(ar, return_index=False)[source]

Find the unique elements of an array in the order they first appear

Like numpy.unique, but preserves original order instead of sorting

Parameters:

ar – array-like data to find unique elements

Options:

return_index: if True also return indices in ar where items first appear

desispec.util.parse_fibers(fiber_string, include_end=False)[source]

Short func that parses a string containing a comma separated list of integers, which can include “:” or “..” or “-” labeled ranges

Parameters:

fiber_string (str) – list of integers or integer ranges

Options:

include_end (bool): if True, include end-value in ranges

Returns (array 1-D):

1D numpy array listing all of the integers given in the list, including enumerations of ranges given.

Note: this follows python-style ranges, i,e, 1:5 or 1..5 returns 1, 2, 3, 4 unless include_end is True, which then returns 1,2,3,4,5

desispec.util.parse_int_args(arg_string, include_end=False)[source]

Short func that parses a string containing a comma separated list of integers, which can include “:” or “..” or “-” labeled ranges

Parameters:

arg_string (str) – list of integers or integer ranges

Options:

include_end (bool): if True, include end-value in ranges

Returns (array 1-D):

1D numpy array listing all of the integers given in the list, including enumerations of ranges given.

Note: this follows python-style ranges, i,e, 1:5 or 1..5 returns 1,2,3,4 unless include_end is True, which then returns 1,2,3,4,5

desispec.util.parse_keyval(keyval)[source]

Parse “key=val” string -> (key,val) tuple with int/float/str/bool val

Parameters:

keyval (str) – “key=value” string

Returns (key, value) tuple where value has been promoted from string into int/float/bool if possible.

value=”True” or “False” becomes boolean True/False, but all other forms like “T”/”F” or “true”/”false” remain strings. 0 and 1 become ints, not bool.

desispec.util.pid_exists(pid)[source]

Check whether pid exists in the current process table.

UNIX only. Should work the same as psutil.pid_exists().

Parameters:

pid (int) – A process ID.

Returns:

True if the process exists in the current process table.

Return type:

pid_exists (bool)

desispec.util.runcmd(cmd, args=None, expandargs=False, inputs=[], outputs=[], comm=None, clobber=False)[source]

Runs a command (function or script), checking for inputs and outputs

Parameters:

cmd – function object or command string to run with subprocess.call()

Options:

args : list of args to pass to the function or script expandargs: call function with cmd(*args) instead of cmd(args) inputs : list of filename inputs that must exist before running outputs : list of output filenames that should be created clobber : if True, run even if outputs already exist comm : MPI communicator to pass to cmd(…, comm=comm)

Returns:

(result, success)

Notes

  • If any inputs are missing, don’t run cmd and return (None, False).

  • If outputs exist and have timestamps after all inputs, don’t run cmd and return (None, True).

  • If spawned as a script, return (returncode, (returncode==0)).

  • If function raises an exception, return (exception, False).

  • If function returns result but outputs are missing, return (result, False).

  • If function returns result and all outputs are present, return (result, True).

desispec.util.set_backend(backend='agg')[source]

Set matplotlib to use a batch-friendly backend

This function is safe to call multiple times without tripping on a previously set backend (which remains set)

desispec.util.sprun(com, capture=False, input=None)[source]

Run a command with subprocess and handle errors.

This runs a command and returns the lines of STDOUT as a list. Any contents of STDERR are logged. If an OSError is raised by the child process, that is also logged. If another exception is raised by the child process, the traceback from the child process is printed.

Parameters:
  • com (list) – the command to run.

  • capture (bool) – if True, return the stdout contents.

  • input (str) – the string data (can include embedded newlines) to write to the STDIN of the child process.

Returns:

the return code and optionally the lines of STDOUT

from the child process.

Return type:

tuple(int, (list))

desispec.util.ymd2night(year, month, day)[source]

convert year, month, day integers into cannonical YEARMMDD night string

desispec.validredshifts

desispec.validredshifts.actually_validate(cat, fiberstatus_cut=True, ignore_emline=False)[source]

Apply redshift quality criteria

Parameters:

cat – astropy table with the necessary columns for redshift quality determination

Options:

fiberstatus_cut: bool (default True), if True, impose requirements on COADD_FIBERSTATUS and ZWARN return_target_columns: bool (default False), if True, include columns that indicate if the object belongs to each class of DESI targets extra_columns: list of str (default None), additional columns to include in the output emline_path: str (default None), specify the location of the emline file; by default the emline file is in the same directory as the redrock file ignore_emline: bool (default False), if True, ignore the emline file and do not validate the ELG redshift

Returns:

astropy table with boolean columns (e.g., GOOD_BGS) added

Return type:

cat

desispec.validredshifts.get_good_fiberstatus(cat, isqso=False)[source]

Validate the fiber status for a redrock catalog.

Parameters:

cat – redrock catalog (e.g., in astropy table format)

Options:

isqso: bool (default False), if True, use the specific QSO criteria

Returns:

boolean array

Return type:

good_fiberstatus

desispec.validredshifts.validate(redrock_path, fiberstatus_cut=True, return_target_columns=False, extra_columns=None, emline_path=None, ignore_emline=False)[source]

Validate the redshift quality with tracer-dependent criteria for redrock+afterburner results.

Parameters:

redrock_path – str, path of redrock FITS file

Options:

fiberstatus_cut: bool (default True), if True, impose requirements on COADD_FIBERSTATUS and ZWARN return_target_columns: bool (default False), if True, include columns that indicate if the object belongs to each class of DESI targets extra_columns: list of str (default None), additional columns to include in the output emline_path: str (default None), specify the location of the emline file; by default the emline file is in the same directory as the redrock file ignore_emline: bool (default False), if True, ignore the emline file and do not validate the ELG redshift

Returns:

astropy table with basic columns such as TARGETID and boolean columns (e.g., GOOD_BGS)

that indicate if each object meets the redshift quality criteria of specific tracers

Return type:

cat

desispec.workflow

Tools for workflow management and running the pipeline.

desispec.workflow.batch

Utilities for working with slurm batch queues.

desispec.workflow.batch.default_system(jobdesc=None, no_gpu=False)[source]

Guess default system to use based on environment

Parameters:
  • jobdesc (str) – Description of the job in the processing table (optional).

  • no_gpu (bool) – Don’t use GPU’s even if available. Default False.

Returns:

default system name to use

Return type:

name (str)

desispec.workflow.batch.get_config(name)[source]

Return configuration dictionary for system name

Parameters:

name (str) – e.g. cori-haswell, cori-knl, dirac, perlmutter-gpu, …

Returns dictionary with keys:
  • site: location of system, e.g. ‘NERSC’

  • cores_per_node: number of physical cores per node

  • threads_per_core: hyperthreading / SMT per core

  • memory: memory per node in GB

  • timefactor: scale time estimates by this amount on this system

  • gpus_per_node: number of GPUs per node

  • batch_opts: list of additional batch options for script header

desispec.workflow.desi_proc_funcs

desispec.workflow.desi_proc_funcs.add_desi_proc_joint_fit_terms(parser)[source]

Add parameters to the argument parser that are only used by desi_proc_joint_fit

desispec.workflow.desi_proc_funcs.add_desi_proc_singular_terms(parser)[source]

Add parameters to the argument parser that are only used by desi_proc

desispec.workflow.desi_proc_funcs.add_desi_proc_tilenight_terms(parser)[source]

Add parameters to the argument parser that are only used by desi_proc_tilenight

desispec.workflow.desi_proc_funcs.assign_mpi(do_mpi, do_batch, log)[source]

Based on whether the mpi flag is set and whether the batch flag is set, assign the appropriate MPI values for the communicator, rank, and number of ranks. Also verify that environment variables are set and provide useful information via the log

Parameters:
  • do_mpi – bool, whether to use mpi or not

  • do_batch – bool, whether the script is meant to write a batch script and exit or not.

  • log – desi log object for reporting

Returns:

A tuple containing:

  • comm: MPI communicator object

  • rank: int, the numeric number assigned to the currenct MPI rank

  • size: int, the total number of MPI ranks

Return type:

tuple

desispec.workflow.desi_proc_funcs.cameras_from_raw_data(rawdata)[source]

Takes a filepath or fitsio FITS object corresponding to a DESI raw data file and returns a list of cameras for which data exists in the file.

Parameters:
  • rawdata – it must be a fitsio.FITS object.

  • Otherwise (str or fitsio.FITS object. The input raw desi data file. str must be a full file path.) – it must be a fitsio.FITS object.

Returns:

The list of cameras that have data in the given file.

Return type:

str

desispec.workflow.desi_proc_funcs.create_desi_proc_batch_script(night, exp, cameras, jobdesc, queue, runtime=None, batch_opts=None, timingfile=None, batchdir=None, jobname=None, cmdline=None, system_name=None, use_specter=False, no_gpu=False)[source]

Generate a SLURM batch script to be submitted to the slurm scheduler to run desi_proc.

Parameters:
  • night (str or int) – The night the data was acquired

  • exp (str, int, or list of int) – The exposure id(s) for the data.

  • cameras (str or list of str) – List of cameras to include in the processing.

  • jobdesc (str) – Description of the job to be performed. Used to determine requested resources and whether to operate in a more mpi parallelism (all except poststdstar) or less (only poststdstar). Directly relate to the obstype, with science exposures being split into two (pre, post)-stdstar, and adding joint fit categories stdstarfit, psfnight, and nightlyflat. Options include: ‘prestdstar’, ‘poststdstar’, ‘stdstarfit’, ‘arc’, ‘flat’, ‘psfnight’, ‘nightlyflat’

  • queue (str) – Queue to be used.

  • runtime (str, optional) – Timeout wall clock time.

  • batch_opts (str, optional) – Other options to give to the slurm batch scheduler (written into the script).

  • timingfile (str, optional) – Specify the name of the timing file.

  • batchdir (str, optional) – Specify where the batch file will be written.

  • jobname (str, optional) – Specify the name of the slurm script written.

  • cmdline (str, optional) – Complete command as would be given in terminal to run the desi_proc. Can be used instead of reading from argv.

  • system_name (str, optional) – name of batch system, e.g. cori-haswell, cori-knl

  • use_specter (bool, optional) – Use classic specter instead of gpu_specter for extractions

  • no_gpu (bool, optional) – Do not use GPU even if available

Returns:

the full path name for the script written.

Return type:

scriptfile

Note

batchdir and jobname can be used to define an alternative pathname, but may not work with assumptions in desi_proc. These optional arguments should be used with caution and primarily for debugging.

desispec.workflow.desi_proc_funcs.create_desi_proc_tilenight_batch_script(night, exp, tileid, ncameras, queue, runtime=None, batch_opts=None, system_name=None, mpistdstars=True, use_specter=False, no_gpu=False, laststeps=None, cameras=None)[source]

Generate a SLURM batch script to be submitted to the slurm scheduler to run desi_proc.

Parameters:
  • night – str or int. The night the data was acquired.

  • exp – int, or list of ints. The exposure id(s) for the data.

  • tileid – str or int. The tile id for the data.

  • ncameras – int. The number of cameras used for joint fitting.

  • queue – str. Queue to be used.

Options:

runtime: str. Timeout wall clock time. batch_opts: str. Other options to give to the slurm batch scheduler (written into the script). system_name: name of batch system, e.g. cori-haswell, cori-knl. mpistdstars: bool. Whether to use MPI for stdstar fitting. use_specter: bool. Use classic specter instead of gpu_specter for extractions no_gpu: bool. Do not use GPU even if available laststeps: list of str. A list of laststeps to pass as the laststeps argument to tilenight cameras: str, must be camword.

Returns:

the full path name for the script written.

Return type:

scriptfile

desispec.workflow.desi_proc_funcs.determine_resources(ncameras, jobdesc, queue, nexps=1, forced_runtime=None, system_name=None)[source]

Determine the resources that should be assigned to the batch script given what desi_proc needs for the given input information.

Parameters:
  • ncameras (int) – number of cameras to be processed

  • jobdesc (str) – type of data being processed

  • queue (str) – the queue at NERSC to be submitted to. ‘realtime’ will force restrictions on number of nodes.

  • nexps (int, optional) – the number of exposures processed in this step

  • force_runtime (int, optional) – the amount of runtime in minutes to allow for the script. Should be left to default heuristics unless needed for some reason.

  • system_name (str, optional) – batch compute system, e.g. cori-haswell or perlmutter-gpu

Returns:

A tuple containing:

  • ncores: int, number of cores (actually 2xphysical cores) that should be submitted via “-n {ncores}”

  • nodes: int, number of nodes to be requested in the script. Typically (ncores-1) // cores_per_node + 1

  • runtime: int, the max time requested for the script in minutes for the processing.

Return type:

tuple

desispec.workflow.desi_proc_funcs.find_most_recent(night, file_type='psfnight', cam='r', n_nights=30)[source]

Searches back in time for either psfnight or fiberflatnight (or anything supported by desispec.calibfinder.findcalibfile. This only works on nightly-based files, so exposure id information is not used.

Parameters:
  • night – str. YYYYMMDD night to look back from

  • file_type – str. psfnight or fiberflatnight

  • cam – str. camera (b, r, or z).

  • n_nights – int. number of nights to step back before giving up

Returns:

Full pathname to calibration file of interest. If none found, None is returned.

Return type:

str

desispec.workflow.desi_proc_funcs.get_desi_proc_batch_file_name(night, exp, jobdesc, cameras)[source]

Returns the default directory location to store a batch script file given a night

Parameters:
  • night – str or int, defines the night (should be 8 digits)

  • exp – str, int, or array of ints, defines the exposure id(s) relevant to the job

  • jobdesc – str, type of data being processed

  • cameras – str or list of str. If str, must be camword, If list, must be list of cameras to include in the processing.

Returns:

str, the default script name for a desi_proc batch script file

Return type:

pathname

desispec.workflow.desi_proc_funcs.get_desi_proc_batch_file_path(night, reduxdir=None)[source]

Returns the default directory location to store a batch script file given a night

Parameters:
  • night (str or int) – defines the night (should be 8 digits)

  • reduxdir (str, optional) – define the base directory where the /run/scripts directory should or does live.

Returns:

the default location where a batch script file should be written

Return type:

str

desispec.workflow.desi_proc_funcs.get_desi_proc_batch_file_pathname(night, exp, jobdesc, cameras, reduxdir=None)[source]

Returns the default directory location to store a batch script file given a night

Parameters:
  • night – str or int, defines the night (should be 8 digits)

  • exp – str, int, or array of ints, defines the exposure id(s) relevant to the job

  • jobdesc – str, type of data being processed

  • cameras – str or list of str. If str, must be camword, If list, must be list of cameras to include in the processing.

  • reduxdir – str (optional), define the base directory where the /run/scripts directory should or does live

Returns:

str, the default location and script name for a desi_proc batch script file

Return type:

pathname

desispec.workflow.desi_proc_funcs.get_desi_proc_joint_fit_parser()[source]

Create an argparser object for use with desi_proc_joint_fit based on arguments from sys.argv

desispec.workflow.desi_proc_funcs.get_desi_proc_parser()[source]

Create an argparser object for use with desi_proc based on arguments from sys.argv

desispec.workflow.desi_proc_funcs.get_desi_proc_tilenight_batch_file_name(night, tileid)[source]

Returns the filename for a tilenight batch script file given a night and tileid

Parameters:
  • night – str or int, defines the night (should be 8 digits)

  • tileid – str or int, defines the tile id relevant to the job

Returns:

str, the default script name for a desi_proc_tilenight batch script file

Return type:

pathname

desispec.workflow.desi_proc_funcs.get_desi_proc_tilenight_batch_file_pathname(night, tileid, reduxdir=None)[source]

Returns the default directory location to store a tilenight batch script file given a night and tileid

Parameters:
  • night – str or int, defines the night (should be 8 digits)

  • tileid – str or int, defines the tile id relevant to the job

  • reduxdir – str (optional), define the base directory where the /run/scripts directory should or does live

Returns:

str, the default location and script name for a desi_proc_tilenight batch script file

Return type:

pathname

desispec.workflow.desi_proc_funcs.get_desi_proc_tilenight_parser()[source]

Create an argparser object for use with desi_proc_tilenight based on arguments from sys.argv

desispec.workflow.desi_proc_funcs.get_shared_desi_proc_parser()[source]

Create an argparser object for use with desi_proc AND desi_proc_joint_fit based on arguments from sys.argv

desispec.workflow.desi_proc_funcs.load_raw_data_header(pathname, return_filehandle=False)[source]

Open raw desi data file given at pathname and return the spectrograph header, which varied in name over time

Parameters:
  • pathname – str, the full path to the raw data file

  • return_filehandle – bool, whether to return the open file handle or to close it and only return the header. Default is False.

Returns:

A tuple containing:

  • hdr: fitsio header object from the file at pathname

  • fx: optional, fitsio file object returned only if return_filehandle is True

Return type:

tuple

desispec.workflow.desi_proc_funcs.log_timer(timer, timingfile=None, comm=None)[source]

Log timing info, optionally writing to json timingfile

Parameters:
  • timer – desiutil.timer.Timer object

  • timingfile (str) – write json output to this file

  • comm – MPI communicator

Notes

  • If comm is not None, collect timers across ranks.

  • If timingfile already exists, read and append timing then re-write.

desispec.workflow.desi_proc_funcs.update_args_with_headers(args)[source]

Update input argparse object with values from header if the argparse values are uninformative defaults (generally python None). This also returns the primary header and each camera’s header.

Parameters:

args – argparse arguments object. Parsed from the command line arguments based on the parser defined by the function get_desi_proc_parser().

Returns:

A tuple containing:

  • args: modified version of the input args where values have been updated if None using information from appropriate headers using either night+expid or an input file.

  • hdr: fitsio header object obtained using read_header() on input file or file determined from args information.

  • camhdr: dict, dictionary of fitsio header objects for each camera in the input files.

Return type:

tuple

Note

The input args is modified and returned here.

desispec.workflow.exptable

desispec.workflow.exptable.aircorr_to_airmass(aircorr)[source]

Transforms an “air correction” term of survey speed to airmass: AIRCORR = X^1.75 https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed

desispec.workflow.exptable.airfac_to_aircorr(airfac)[source]

Transforms an “AIRFAC” term of survey speed to an “air correction” term of survey speed https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed

desispec.workflow.exptable.airfac_to_airmass(airfac, k=0.114)[source]

Transforms an “AIRFAC” term of survey speed to airmass: AIRFAC = 10^[k*(X-1)/2.5] https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed

desispec.workflow.exptable.airmass_to_aircorr(airmass)[source]

Transforms an airmass to “air correction” term of survey speed: AIRCORR = X^1.75 https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed

desispec.workflow.exptable.airmass_to_airfac(airmass, k=0.114)[source]

Transforms an airmass to “AIRFAC”: AIRFAC = 10^[k*(X-1)/2.5] https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed

desispec.workflow.exptable.deconstruct_keyval_reporting(entry)[source]

Takes a reporting of the form {colname}:{oldval}->{newval} and returns colname, oldval, newval.

Parameters:

entry (str) – A string of the form {colname}:{oldval}->{newval}. colname should be an all upper case column name. oldval and newval can include any string characters except the specific combination “->”.

Returns:

A tuple containing:

  • key, str. The string that precedes the initial colon. The name of the column being reported.

  • val1, str. The string after the initial colon and preceding the ‘->’. The original value of the column.

  • val2, str. The string after the ‘->’. The value that the original was changed to.

Return type:

tuple

desispec.workflow.exptable.default_obstypes_for_exptable()[source]

Defines the exposure types to be recognized by the workflow and saved in the exposure table by default.

Returns:

A list of default obstypes to be included in an exposure table.

Return type:

list

desispec.workflow.exptable.exposure_table_column_defs()[source]

Contains the column names, data types, and default row values for a DESI Exposure table. It returns the names, datatypes, and defaults as a list of 3-tuples.

Returns:

List of tuples (name, type, default).

Return type:

list

desispec.workflow.exptable.get_exposure_flags()[source]

Defines the exposure flags that can be saved in the exposure table.

Returns:

A list of exposure flags that can be included in an exposure table.

Return type:

list

desispec.workflow.exptable.get_exposure_table_column_defaults(asdict=True)[source]

Returns the default values for each column entry of a DESI pipeline Exposure table. It returns the names as keys and defaults as values in a dictionary if asdict=True. If False, the defaults are returned as a list (in the same order as the names returned by get_exposure_table_column_names().

Parameters:

asdict (bool, optional) – Default is True. If you want the defaults as values in a dictionary with keys as the names of the columns. If False, an ordered list is returned.

Returns:

If asdict, a dict is returned with column defaults as the values and columns names as the keys. If False, a list of defaults in the same order as the names returned from get_exposure_table_column_names().

Return type:

dict or list

desispec.workflow.exptable.get_exposure_table_column_defs(return_default_values=False)[source]

Contains the column names, data types, and default row values for a DESI Exposure table. It returns the names and datatypes with the defaults being given with an optional flag. Returned as 2 (or 3) lists.

Parameters:

return_default_values (bool, optional) – True if you want the default values returned.

Returns:

A tuple containing:

  • colnames, list. List of column names for an exposure table.

  • coltypes, list. List of column datatypes for the names in colnames.

  • coldeflts, list. Optionally returned if return_default_values is True. List of default values for the corresponding colnames.

Return type:

tuple

desispec.workflow.exptable.get_exposure_table_column_names()[source]

Contains the column names, data types, and default row values for a DESI Exposure table. It returns the names as a list.

Returns:

List of column names for an exposure table.

Return type:

list

desispec.workflow.exptable.get_exposure_table_column_types(asdict=True)[source]

Returns the datatypes values for each column entry of a DESI pipeline Exposure table. It returns the names as keys and datatypes as values in a dictionary if asdict=True. If False, the datatypes are returned as a list (in the same order as the names returned by get_exposure_table_column_names().

Parameters:

asdict (bool, optional) – True if you want the types as values in a dictionary with keys as the names of the columns. If False, an ordered list is returned. Default is True.

Returns:

If asdict, a dict is returned with column datatypes as the values and columns names as the keys. If False, a list of datatypes in the same order as the names returned from get_exposure_table_column_names().

Return type:

dict or list

desispec.workflow.exptable.get_exposure_table_name(night, extension='csv')[source]

Defines the default exposure name given the night of the observations and the optional extension.

Parameters:
  • night (int or str) – The night of the observations going into the exposure table.

  • extension (str, optional) – The extension (and therefore data format) without a leading period of the saved table. Default is ‘csv’.

Returns:

The exposure table name given the input night and extension.

Return type:

str

desispec.workflow.exptable.get_exposure_table_path(night=None, usespecprod=True)[source]

Defines the default path to save an exposure table. If night is given, it saves it under a monthly directory to reduce the number of files in a large production directory.

Parameters:
  • night (int or str, optional) – The night corresponding to the exposure table. If None, no monthly subdirectory is used.

  • usespecprod (bool, optional) – Whether to use the master version in the exposure table repo or the version in a specprod.

Returns:

The full path to the directory where the exposure table should be written (or is already written). This does not including the filename.

Return type:

str

desispec.workflow.exptable.get_exposure_table_pathname(night, usespecprod=True, extension='csv')[source]

Defines the default pathname to save an exposure table.

Parameters:
  • night (int or str, optional) – The night corresponding to the exposure table.

  • usespecprod (bool, optional) – Whether to use the master version or the version in a specprod.

Returns:

The full pathname where the exposure table should be written (or is already written). This includes the filename.

Return type:

str

desispec.workflow.exptable.get_last_step_options()[source]

Defines the LASTSTEP options that can be saved in the exposure table that will be understood by the pipeline.

Returns:

A list of LASTSTEP’s that can be included in an exposure table.

Return type:

list

desispec.workflow.exptable.instantiate_exposure_table(colnames=None, coldtypes=None, rows=None)[source]

Create an empty exposure table with proper column names and datatypes. If rows is given, it inserts the rows into the table, otherwise it returns a table with no rows.

Parameters:
  • colnames (list) – List of column names for an exposure table.

  • coldtypes (list) – List of column datatypes for the names in colnames.

  • rows (list or np.array of Table.Row or dict) – An iterable set of Table.Row or dict with keys/colnames and value pairs that match the default column names and data types of the default exposure table.

Returns:

An astropy Table with the column names and data types for a DESI workflow exposure table. If the input rows was not None, it contains those rows, otherwise it has no rows.

Return type:

Table

desispec.workflow.exptable.keyval_change_reporting(keyword, original_val, replacement_val)[source]

Creates a reporting string to be saved in the HEADERERR or COMMENTS column of the exposure table. Give the keyword, the original value, and the value that it was replaced with.

Parameters:
  • keyword (str. The value typically saved for that) –

  • to. (str. Keyword in the exposure table that the values correspond) –

  • original_val

  • keyword

  • deviates. (except when it) –

  • replacement_val

  • column. (str. The value that was saved instead in that keyword) –

Returns:

Of the format ‘ keyword:original->replacement ‘

Return type:

str

desispec.workflow.exptable.night_to_month(night)[source]

Trivial function that returns the month portion of a night. Can be given a string or int.

Parameters:

night (int or str) – The night you want the month of.

Returns:

The zero-padded (length two) string representation of the month corresponding to the input month.

Return type:

str

desispec.workflow.exptable.summarize_exposure(raw_data_dir, night, exp, obstypes=None, colnames=None, coldefaults=None, verbosely=False)[source]

Given a raw data directory and exposure information, this searches for the raw DESI data files for that exposure and loads in relevant information for that flavor+obstype. It returns a dictionary if the obstype is one of interest for the exposure table, a string if the exposure signifies the end of a calibration sequence, and None if the exposure is not in the given obstypes.

Parameters:
  • raw_data_dir – nightly subdirectories reside.

  • the (str. The path to where the raw data is stored. It should be the upper level directory where) – nightly subdirectories reside.

  • night

  • in. (str or int. Used to know what nightly subdirectory to look for the given exposure) –

  • exp

  • interest. (str or int or float. The exposure number of) –

  • obstypes – information about that exposure is taken and returned for the exposure table. Otherwise None is returned (or str if it is an end-of-cal manifest). If None, the default list in default_obstypes_for_exptable() is used.

  • found (list or np.array of str's. The list of 'OBSTYPE' keywords to match to. If a match is) – information about that exposure is taken and returned for the exposure table. Otherwise None is returned (or str if it is an end-of-cal manifest). If None, the default list in default_obstypes_for_exptable() is used.

  • the – information about that exposure is taken and returned for the exposure table. Otherwise None is returned (or str if it is an end-of-cal manifest). If None, the default list in default_obstypes_for_exptable() is used.

  • colnames – get_exposure_table_column_defs().

  • None (list or np.array. List of default values for the corresponding colnames. If) – get_exposure_table_column_defs().

  • from (the defaults are taken) – get_exposure_table_column_defs().

  • coldefaults – are taken from get_exposure_table_column_defs().

  • None – are taken from get_exposure_table_column_defs().

  • defaults (the) – are taken from get_exposure_table_column_defs().

  • verbosely (True) or more succinct output (False) –

  • output (bool. Whether to print more detailed) –

Returns:

A variety of types which could be:

  • dict: Dictionary with keys corresponding to the column names of an exposure table. Values are taken from the data when found, otherwise the values are the corresponding default given in coldefaults OR;

  • str: If the exposures signifies the end of a calibration sequence, it returns a string describing the type of

    sequence that ended. Either “(short|long|arc) calib complete” OR;

  • NoneType: If the exposure obstype was not in the requested types (obstypes).

Return type:

object

desispec.workflow.proc_dashboard_funcs

desispec.workflow.proc_dashboard_funcs._initialize_page(color_profile, titlefill='Processing')[source]

Initialize the html file for showing the statistics, giving all the headers and CSS setups.

desispec.workflow.proc_dashboard_funcs._write_js_script(output_path)[source]

Return the javascript script to be added to the html file

desispec.workflow.proc_dashboard_funcs.check_running(proc_name='desi_dailyproc', suppress_outputs=False)[source]

Check if the desi_dailyproc process is running

desispec.workflow.proc_dashboard_funcs.find_new_exps(fileglob, known_exposures)[source]

Check the path given for new exposures

desispec.workflow.proc_dashboard_funcs.generate_monthly_table_html(tables, statuses, month)[source]

Add a collapsible and extendable table to the html file for a specific month Input tables: list of tables generated by ‘nightly_table’ month: string of YYYYMM, e.g. 202001 output: The string to be added to the html file

desispec.workflow.proc_dashboard_funcs.js_str()[source]

Return the javascript script to be added to the html file

desispec.workflow.proc_dashboard_funcs.what_night_is_it()[source]

Return the current night

desispec.workflow.procfuncs

desispec.workflow.procfuncs.arc_joint_fit(ptable, arcs, internal_id, queue='realtime', reservation=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]

Wrapper function for desiproc.workflow.procfuns.joint_fit specific to the psfnight joint fit.

All variables are the same except:

Arg 'arcs' is mapped to the prows argument of joint_fit.
The joint_fit argument descriptor is pre-defined as 'psfnight'.
desispec.workflow.procfuncs.assign_dependency(prow, dependency)[source]

Given input processing row and possible arcjob (processing row for psfnight) and flatjob (processing row for nightlyflat), this defines the JOBDESC keyword and assigns the dependency appropriate for the job type of prow.

Parameters:
  • prow – ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.

  • for (Table.Row or dict. Must include keyword accessible definitions for 'OBSTYPE'. A row must have column names) – ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.

  • dependency – for the job in prow. This must contain keyword accessible values for ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

  • Table.Row (NoneType or scalar/list/array of) – for the job in prow. This must contain keyword accessible values for ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

  • input (dict. Processing row corresponding to the required) – for the job in prow. This must contain keyword accessible values for ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

Returns:

The same prow type and keywords as input except with modified values updated values for ‘JOBDESC’, ‘INT_DEP_IDS’. and ‘LATEST_DEP_ID’.

Return type:

Table.Row or dict

Note

This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).

desispec.workflow.procfuncs.batch_script_name(prow)[source]

Wrapper script that takes a processing table row (or dictionary with NIGHT, EXPID, JOBDESC, PROCCAMWORD defined) and determines the script file pathname as defined by desi_proc’s helper functions.

Parameters:

prow (Table.Row or dict) – Must include keyword accessible definitions for ‘NIGHT’, ‘EXPID’, ‘JOBDESC’, and ‘PROCCAMWORD’.

Returns:

The complete pathname to the script file, as it is defined within the desi_proc ecosystem.

Return type:

str

desispec.workflow.procfuncs.check_for_outputs_on_disk(prow, resubmit_partial_complete=True)[source]
Parameters:
  • prow (Table.Row or dict) – Must include keyword accessible definitions for processing_table columns found in desispect.workflow.proctable.get_processing_table_column_defs()

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

Returns:

The same prow type and keywords as input except with modified values updated to reflect the change in job status after creating and submitting the job for processing.

Return type:

Table.Row or dict

desispec.workflow.procfuncs.checkfor_and_submit_joint_job(ptable, arcs, flats, sciences, calibjobs, lasttype, internal_id, z_submit_types=None, dry_run=0, queue='realtime', reservation=None, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]

Takes all the state-ful data from daily processing and determines whether a joint fit needs to be submitted. Places the decision criteria into a single function for easier maintainability over time. These are separate from the new standard manifest*.json method of indicating a calibration sequence is complete. That is checked independently elsewhere and doesn’t interact with this.

Parameters:
  • ptable (Table) – Processing table of all exposures that have been processed.

  • arcs (list of dict) – list of the individual arc jobs to be used for the psfnight (NOT all the arcs, if multiple sets existed). May be empty if none identified yet.

  • flats (list of dict) – list of the individual flat jobs to be used for the nightlyflat (NOT all the flats, if multiple sets existed). May be empty if none identified yet.

  • sciences (list of dict) – list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet.

  • calibjobs (dict) – Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.

  • lasttype (str or None) – the obstype of the last individual exposure row to be processed.

  • internal_id (int) – an internal identifier unique to each job. Increments with each new job. This is the smallest unassigned value.

  • z_submit_types (list of str) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.

  • dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • queue (str, optional) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.

  • reservation (str, optional) – The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • system_name (str) – batch system name, e.g. cori-haswell, cori-knl, permutter-gpu

Returns:

A tuple containing:

  • ptable, Table, Processing table of all exposures that have been processed.

  • calibjobs, dict. Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.

  • sciences, list of dicts, list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet or we just submitted them for processing.

  • internal_id, int, if no job is submitted, this is the same as the input, otherwise it is incremented upward from from the input such that it represents the smallest unused ID.

Return type:

tuple

desispec.workflow.procfuncs.create_and_submit(prow, queue='realtime', reservation=None, dry_run=0, joint=False, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None, use_specter=False, laststeps=None)[source]

Wrapper script that takes a processing table row and three modifier keywords, creates a submission script for the compute nodes, and then submits that script to the Slurm scheduler with appropriate dependencies.

Parameters:
  • prow (Table.Row or dict) – Must include keyword accessible definitions for processing_table columns found in desispect.workflow.proctable.get_processing_table_column_defs()

  • queue (str, optional) – The name of the NERSC Slurm queue to submit to. Default is the realtime queue.

  • reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • joint (bool, optional) – Whether this is a joint fitting job (the job involves multiple exposures) and therefore needs to be run with desi_proc_joint_fit. Default is False.

  • strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu

  • use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.

  • laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.

Returns:

The same prow type and keywords as input except with modified values updated to reflect the change in job status after creating and submitting the job for processing.

Return type:

Table.Row or dict

Note

This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).

desispec.workflow.procfuncs.create_batch_script(prow, queue='realtime', dry_run=0, joint=False, system_name=None, use_specter=False, laststeps=None)[source]

Wrapper script that takes a processing table row and three modifier keywords and creates a submission script for the compute nodes.

Parameters:
  • prow – desispect.workflow.proctable.get_processing_table_column_defs()

  • in (Table.Row or dict. Must include keyword accessible definitions for processing_table columns found) – desispect.workflow.proctable.get_processing_table_column_defs()

  • queue

  • queue. (str. The name of the NERSC Slurm queue to submit to. Default is the realtime) –

  • dry_run – If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • nonzero (int. If) – If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • submitted. (this is a simulated run. If dry_run=1 the scripts will be written but not) – If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • joint (the job involves multiple exposures) – run with desi_proc_joint_fit when not using tilenight. Default is False.

  • job (bool. Whether this is a joint fitting) – run with desi_proc_joint_fit when not using tilenight. Default is False.

  • system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu

  • use_specter

  • bool

  • True (optional. Default is False. If) –

  • specter (use) –

  • default. (otherwise use gpu_specter by) –

  • laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.

Returns:

The same prow type and keywords as input except with modified values updated values for scriptname.

Return type:

Table.Row or dict

Note

This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).

desispec.workflow.procfuncs.define_and_assign_dependency(prow, calibjobs, use_tilenight=False)[source]

Given input processing row and possible calibjobs, this defines the JOBDESC keyword and assigns the dependency appropriate for the job type of prow.

Parameters:
  • prow – ‘OBSTYPE’. A row must have column names for ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.

  • for (Table.Row or dict. Must include keyword accessible definitions) – ‘OBSTYPE’. A row must have column names for ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.

  • calibjobs – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

  • 'nightlybias' (dict. Dictionary containing) – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

  • 'ccdcalib' – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

  • 'psfnight' – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.

  • use_tilenight – for prestdstar, stdstar,and poststdstar steps for science exposures.

  • True (bool. Default is False. If) – for prestdstar, stdstar,and poststdstar steps for science exposures.

  • desi_proc_tilenight (use) – for prestdstar, stdstar,and poststdstar steps for science exposures.

Returns:

The same prow type and keywords as input except with modified values updated values for ‘JOBDESC’, ‘INT_DEP_IDS’. and ‘LATEST_DEP_ID’.

Return type:

Table.Row or dict

Note

This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).

desispec.workflow.procfuncs.desi_proc_command(prow, system_name, use_specter=False, queue=None)[source]

Wrapper script that takes a processing table row (or dictionary with NIGHT, EXPID, OBSTYPE, JOBDESC, PROCCAMWORD defined) and determines the proper command line call to process the data defined by the input row/dict.

Parameters:
  • prow (Table.Row or dict) – Must include keyword accessible definitions for ‘NIGHT’, ‘EXPID’, ‘JOBDESC’, and ‘PROCCAMWORD’.

  • system_name (str) – batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu

  • queue (str, optional) – The name of the NERSC Slurm queue to submit to. Default is None (which leaves it to the desi_proc default).

  • use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.

Returns:

The proper command to be submitted to desi_proc to process the job defined by the prow values.

Return type:

str

desispec.workflow.procfuncs.desi_proc_joint_fit_command(prow, queue=None)[source]

Wrapper script that takes a processing table row (or dictionary with NIGHT, EXPID, OBSTYPE, PROCCAMWORD defined) and determines the proper command line call to process the data defined by the input row/dict.

Parameters:
  • prow (Table.Row or dict) – Must include keyword accessible definitions for ‘NIGHT’, ‘EXPID’, ‘JOBDESC’, and ‘PROCCAMWORD’.

  • queue (str) – The name of the NERSC Slurm queue to submit to. Default is None (which leaves it to the desi_proc default).

Returns:

The proper command to be submitted to desi_proc_joint_fit to process the job defined by the prow values.

Return type:

str

desispec.workflow.procfuncs.flat_joint_fit(ptable, flats, internal_id, queue='realtime', reservation=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]

Wrapper function for desiproc.workflow.procfuns.joint_fit specific to the nightlyflat joint fit.

All variables are the same except:

Arg 'flats' is mapped to the prows argument of joint_fit.
The joint_fit argument descriptor is pre-defined as 'nightlyflat'.
desispec.workflow.procfuncs.get_type_and_tile(erow)[source]

Trivial function to return the OBSTYPE and the TILEID from an exposure table row

Parameters:
  • erow

  • keywords. (Table.Row or dict. Must contain 'OBSTYPE' and 'TILEID' as) –

Returns:

tuple (str, str), corresponding to the OBSTYPE and TILEID values of the input erow.

desispec.workflow.procfuncs.joint_fit(ptable, prows, internal_id, queue, reservation, descriptor, z_submit_types=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]

Given a set of prows, this generates a processing table row, creates a batch script, and submits the appropriate joint fitting job given by descriptor. If the joint fitting job is standard star fitting, the post standard star fits for all the individual exposures also created and submitted. The returned ptable has all of these rows added to the table given as input.

Parameters:
  • ptable (Table) – The processing table where each row is a processed job.

  • prows (list or array of dict) – The rows corresponding to the individual exposure jobs that are inputs to the joint fit.

  • internal_id (int) – the next internal id to be used for assignment (already incremented up from the last used id number used).

  • queue (str) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.

  • reservation (str) – The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • descriptor (str) – Description of the joint fitting job. Can either be ‘science’ or ‘stdstarfit’, ‘arc’ or ‘psfnight’, or ‘flat’ or ‘nightlyflat’.

  • z_submit_types (list of str, optional) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.

  • dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu

Returns:

A tuple containing:

  • ptable, Table. The same processing table as input except with added rows for the joint fit job and, in the case of a stdstarfit, the poststdstar science exposure jobs.

  • joint_prow, dict. Row of a processing table corresponding to the joint fit job.

  • internal_id, int, the next internal id to be used for assignment (already incremented up from the last used id number used).

Return type:

tuple

desispec.workflow.procfuncs.make_joint_prow(prows, descriptor, internal_id)[source]

Given an input list or array of processing table rows and a descriptor, this creates a joint fit processing job row. It starts by copying the first input row, overwrites relevant columns, and defines the new dependencies (based on the input prows).

Parameters:
  • prows – inputs to the joint fit.

  • are (list or array of dicts. The rows corresponding to the individual exposure jobs that) – inputs to the joint fit.

  • descriptor

  • 'stdstarfit' (str. Description of the joint fitting job. Can either be) –

  • 'psfnight'

  • 'nightlyflat'. (or) –

  • internal_id (already incremented up from the last used id number used) –

  • int (already incremented up from the last used id number used) –

  • assignment (the next internal id to be used for) –

Returns:

Row of a processing table corresponding to the joint fit job.

Return type:

dict

desispec.workflow.procfuncs.make_redshift_prow(prows, tnight, descriptor, internal_id)[source]

Given an input list or array of processing table rows and a descriptor, this creates a joint fit processing job row. It starts by copying the first input row, overwrites relevant columns, and defines the new dependencies (based on the input prows).

Parameters:
  • prows – the first steps of tilenight.

  • are (list or array of dicts. Unsumbitted rows corresponding to the individual prestdstar jobs that) – the first steps of tilenight.

  • tnight

  • depends. (Table.Row object. Row corresponding to the tilenight job on which the redshift job) –

  • internal_id (already incremented up from the last used id number used) –

  • int (already incremented up from the last used id number used) –

  • assignment (the next internal id to be used for) –

Returns:

Row of a processing table corresponding to the tilenight job.

Return type:

dict

desispec.workflow.procfuncs.make_tnight_prow(prows, calibjobs, internal_id)[source]

Given an input list or array of processing table rows and a descriptor, this creates a joint fit processing job row. It starts by copying the first input row, overwrites relevant columns, and defines the new dependencies (based on the input prows).

Parameters:
  • prows – the first steps of tilenight.

  • are (list or array of dicts. Unsumbitted rows corresponding to the individual prestdstar jobs that) – the first steps of tilenight.

  • calibjobs – None, with each table.Row() value corresponding to a calibration job on which the tilenight job depends.

  • or (dict. Dictionary containing keys that each corresponds to a Table.Row) – None, with each table.Row() value corresponding to a calibration job on which the tilenight job depends.

  • internal_id (already incremented up from the last used id number used) –

  • int (already incremented up from the last used id number used) –

  • assignment (the next internal id to be used for) –

Returns:

Row of a processing table corresponding to the tilenight job.

Return type:

dict

desispec.workflow.procfuncs.night_to_starting_iid(night=None)[source]

Creates an internal ID for a given night. The resulting integer is an 8 digit number. The digits are YYMMDDxxx where YY is the years since 2000, MM and DD are the month and day. xxx are 000, and are incremented for up to 1000 unique job ID’s for a given night.

Parameters:

night (str or int) – YYYYMMDD of the night to get the starting internal ID for.

Returns:

9 digit number consisting of YYMMDD000. YY is years after 2000, MMDD is month and day. 000 being the starting job number (0).

Return type:

int

desispec.workflow.procfuncs.parse_previous_tables(etable, ptable, night)[source]

This takes in the exposure and processing tables and regenerates all the working memory variables needed for the daily processing script.

Used by the daily processing to define most of its state-ful variables into working memory. If the processing table is empty, these are simply declared and returned for use. If the code had previously run and exited (or crashed), however, this will all the code to re-establish itself by redefining these values.

Parameters:
  • etable

  • Table

  • far. (Exposure table of all exposures that have been dealt with thus) –

  • ptable

  • Table

  • processed. (Processing table of all exposures that have been) –

  • night

  • int (str or) –

  • taken. (the night the data was) –

Returns:

A tuple containing:

  • arcs, list of dicts, list of the individual arc jobs used for the psfnight (NOT all the arcs, if multiple sets existed)

  • flats, list of dicts, list of the individual flat jobs used for the nightlyflat (NOT all the flats, if multiple sets existed)

  • sciences, list of dicts, list of the most recent individual prestdstar science exposures (if currently processing that tile)

  • calibjobs, dict. Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘badcol’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.

  • curtype, None, the obstype of the current job being run. Always None as first new job will define this.

  • lasttype, str or None, the obstype of the last individual exposure row to be processed.

  • curtile, None, the tileid of the current job (if science). Otherwise None. Always None as first new job will define this.

  • lasttile, str or None, the tileid of the last job (if science). Otherwise None.

  • internal_id, int, an internal identifier unique to each job. Increments with each new job. This is the latest unassigned value.

Return type:

tuple

desispec.workflow.procfuncs.recursive_submit_failed(rown, proc_table, submits, id_to_row_map, ptab_name=None, resubmission_states=None, reservation=None, dry_run=0)[source]

Given a row of a processing table and the full processing table, this resubmits the given job. Before submitting a job, it checks the dependencies for failures in the processing table. If a dependency needs to be resubmitted, it recursively follows dependencies until it finds the first job without a failed dependency and resubmits that. Then resubmits the other jobs with the new Slurm jobID’s for proper dependency coordination within Slurm.

Parameters:
  • rown

  • Table.Row

  • resubmit. (the row of the processing table that you want to) –

  • proc_table

  • Table

  • job. (the processing table with a row per) –

  • submits

  • int

  • scheduler. (the number of submissions made to the queue. Used for saving files and in not overloading the) –

  • id_to_row_map (INTID's) – in the processing table.

  • dict (INTID's) – in the processing table.

  • ids (lookup dictionary where the keys are internal) – in the processing table.

  • ptab_name

  • str

  • saved. (the full pathname where the processing table should be) –

  • resubmission_states – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted

  • strings (list or array of) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted

  • a (each element should be a capitalized string corresponding to) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted

  • reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • dry_run – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • int – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • nonzero (If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

Returns:

A tuple containing:

  • proc_table: Table, a table with the same rows as the input except that Slurm and jobid relevant columns have been updated for those jobs that needed to be resubmitted.

  • submits: int, the number of submissions made to the queue. This is incremented from the input submits, so it is the number of submissions made from this function call plus the input submits value.

Return type:

tuple

Note

This modifies the inputs of both proc_table and submits and returns them.

desispec.workflow.procfuncs.science_joint_fit(ptable, sciences, internal_id, queue='realtime', reservation=None, z_submit_types=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]

Wrapper function for desiproc.workflow.procfuns.joint_fit specific to the stdstarfit joint fit and redshift fitting.

All variables are the same except:

Arg 'sciences' is mapped to the prows argument of joint_fit.
The joint_fit argument descriptor is pre-defined as 'science'.
desispec.workflow.procfuncs.set_calibrator_flag(prows, ptable)[source]
Sets the “CALIBRATOR” column of a procesing table row to 1 (integer representation of True)

for all input rows. Used within joint fitting code to flag the exposures that were input to the psfnight or nightlyflat for later reference.

Parameters:
  • prows – inputs to the joint fit.

  • are (list or array of Table.Rows or dicts. The rows corresponding to the individual exposure jobs that) – inputs to the joint fit.

  • ptable

  • job. (Table. The processing table where each row is a processed) –

Returns:

The same processing table as input except with added rows for the joint fit job and, in the case of a stdstarfit, the poststdstar science exposure jobs.

Return type:

Table

desispec.workflow.procfuncs.still_a_dependency(dependency)[source]

Defines the criteria for which a dependency is deemed complete (and therefore no longer a dependency).

Args:
dependency, Table.Row or dict. Processing row corresponding to the required input for the job in prow.

This must contain keyword accessible values for ‘STATUS’, and ‘LATEST_QID’.

Returns:

False if the criteria indicate that the dependency is completed and no longer a blocking factor (ie no longer a genuine dependency). Returns True if the dependency is still a blocking factor such that the slurm scheduler needs to be aware of the pending job.

Return type:

bool

desispec.workflow.procfuncs.submit_batch_script(prow, dry_run=0, reservation=None, strictly_successful=False)[source]

Wrapper script that takes a processing table row and three modifier keywords and submits the scripts to the Slurm scheduler.

Parameters:
  • prow – desispect.workflow.proctable.get_processing_table_column_defs()

  • in (Table.Row or dict. Must include keyword accessible definitions for processing_table columns found) – desispect.workflow.proctable.get_processing_table_column_defs()

  • dry_run – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • nonzero (int. If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • strictly_successful – less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • processing (bool. Whether all jobs require all inputs to have succeeded. For daily) – less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • is (this) – less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

Returns:

The same prow type and keywords as input except with modified values updated values for scriptname.

Return type:

Table.Row or dict

Note

This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).

desispec.workflow.procfuncs.submit_redshifts(ptable, prows, tnight, internal_id, queue, reservation, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, z_submit_types=None, system_name=None)[source]

Given a set of prows, this generates a processing table row, creates a batch script, and submits the appropriate tilenight job given by descriptor. The returned ptable has all of these rows added to the table given as input.

Parameters:
  • ptable (Table) – The processing table where each row is a processed job.

  • prows (list or array of dict) – Unsubmitted prestdstar jobs that are first steps in tilenight.

  • tnight (Table.Row) – The processing table row of the tilenight job on which the redshifts depend.

  • internal_id (int) – the next internal id to be used for assignment (already incremented up from the last used id number used).

  • queue (str) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.

  • reservation (str) – The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • z_submit_types (list of str) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.

  • system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu

Returns:

A tuple containing:

  • ptable, Table. The same processing table as input except with added rows for the joint fit job.

  • internal_id, int, the next internal id to be used for assignment (already incremented up from the last used id number used).

Return type:

tuple

desispec.workflow.procfuncs.submit_tilenight(ptable, prows, calibjobs, internal_id, queue, reservation, dry_run=0, strictly_successful=False, resubmit_partial_complete=True, system_name=None, use_specter=False, laststeps=None)[source]

Given a set of prows, this generates a processing table row, creates a batch script, and submits the appropriate tilenight job given by descriptor. The returned ptable has all of these rows added to the table given as input.

Parameters:
  • ptable (Table) – The processing table where each row is a processed job.

  • prows (list or array of dict) – Unsubmitted prestdstar jobs that are first steps in tilenight.

  • calibjobs (dict) – Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.

  • internal_id (int) – the next internal id to be used for assignment (already incremented up from the last used id number used).

  • queue (str) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.

  • reservation (str) – The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu

  • use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.

  • laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.

Returns:

A tuple containing:

  • ptable, Table. The same processing table as input except with added rows for the joint fit job.

  • tnight_prow, dict. Row of a processing table corresponding to the tilenight job.

  • internal_id, int, the next internal id to be used for assignment (already incremented up from the last used id number used).

Return type:

tuple

desispec.workflow.procfuncs.submit_tilenight_and_redshifts(ptable, sciences, calibjobs, lasttype, internal_id, dry_run=0, queue='realtime', reservation=None, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, z_submit_types=None, system_name=None, use_specter=False, laststeps=None)[source]

Takes all the state-ful data from daily processing and determines whether a tilenight job needs to be submitted.

Parameters:
  • ptable (Table) – Processing table of all exposures that have been processed.

  • sciences (list of dict) – list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet.

  • lasttype (str or None) – the obstype of the last individual exposure row to be processed.

  • internal_id (int) – an internal identifier unique to each job. Increments with each new job. This is the smallest unassigned value.

  • dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • queue (str, optional) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.

  • reservation (str, optional) – The reservation to submit jobs to. If None, it is not submitted to a reservation.

  • strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.

  • check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.

  • resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.

  • z_submit_types (list of str, optional) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.

  • system_name (str) – batch system name, e.g. cori-haswell, cori-knl, permutter-gpu

  • use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.

  • laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.

Returns:

A tuple containing:

  • ptable, Table, Processing table of all exposures that have been processed.

  • sciences, list of dicts, list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet or we just submitted them for processing.

  • internal_id, int, if no job is submitted, this is the same as the input, otherwise it is incremented upward from from the input such that it represents the smallest unused ID.

Return type:

tuple

desispec.workflow.procfuncs.update_and_recurvsively_submit(proc_table, submits=0, resubmission_states=None, ptab_name=None, dry_run=0, reservation=None)[source]

Given an processing table, this loops over job rows and resubmits failed jobs (as defined by resubmission_states). Before submitting a job, it checks the dependencies for failures. If a dependency needs to be resubmitted, it recursively follows dependencies until it finds the first job without a failed dependency and resubmits that. Then resubmits the other jobs with the new Slurm jobID’s for proper dependency coordination within Slurm.

Parameters:
  • proc_table

  • Table

  • job. (the processing table with a row per) –

  • submits

  • int

  • scheduler. (the number of submissions made to the queue. Used for saving files and in not overloading the) –

  • resubmission_states – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted

  • strings (list or array of) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted

  • a (each element should be a capitalized string corresponding to) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted

  • ptab_name

  • str

  • saved. (the full pathname where the processing table should be) –

  • dry_run – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • int – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • nonzero (If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).

  • reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.

Returns:

A tuple containing:

  • proc_table: Table, a table with the same rows as the input except that Slurm and jobid relevant columns have been updated for those jobs that needed to be resubmitted.

  • submits: int, the number of submissions made to the queue. This is incremented from the input submits, so it is the number of submissions made from this function call plus the input submits value.

Return type:

tuple

Note

This modifies the inputs of both proc_table and submits and returns them.

desispec.workflow.proctable

desispec.workflow.proctable.default_obstypes_for_proctable()[source]

Defines the exposure types to be recognized by the workflow and saved in the processing table by default.

Returns:

list. A list of default obstypes to be included in a processing table.

desispec.workflow.proctable.default_prow()[source]

Creates a processing table row. The columns are filled with default values.

Parameters:

None

Returns:

prow, dict. The output processing table row.

desispec.workflow.proctable.erow_to_prow(erow)[source]

Converts an exposure table row to a processing table row. The columns unique to a processing table are filled with default values. If comments are made in COMMENTS or HEADERERR, those are ignored.

Parameters:
  • erow – If comments are made in COMMENTS or HEADERERR, those are ignored.

  • table. (Table.Row or dict. An exposure table row. The row will be converted to a row of an processing) – If comments are made in COMMENTS or HEADERERR, those are ignored.

Returns:

prow, dict. The output processing table row.

desispec.workflow.proctable.exptable_to_proctable(input_exptable, obstypes=None)[source]

Converts an exposure table to a processing table and an unprocessed table. The columns unique to a processing table are filled with default values. If comments are made in COMMENTS or HEADERERR, those will be adjusted in the values stored in the processing table.

Parameters:
  • input_exptable – comments are made in COMMENTS or HEADERERR, those will be adjusted in the values stored in the processing table.

  • If (Table. An exposure table. Each row will be converted to a row of an processing table.) – comments are made in COMMENTS or HEADERERR, those will be adjusted in the values stored in the processing table.

  • obstypes – added to the processing table).

  • therefore (list or np.array. Optional. A list of exposure OBSTYPE's that should be processed (and) – added to the processing table).

Returns:

processing_table, Table. The output processing table. Each row corresponds with an exposure that should be processed. unprocessed_table, Table. The output unprocessed table. Each row is an exposure that should not be processed.

desispec.workflow.proctable.get_processing_table_column_defs(return_default_values=False, overlap_only=False, unique_only=False)[source]

Contains the column names, data types, and default row values for a DESI processing table. It returns the names and datatypes with the defaults being given with an optional flag. Returned as 2 (or 3) lists.

Parameters:
  • return_default_values

  • returned. (bool. True if you want the default values) –

  • overlap_only

  • tables. (bool. Only return the columns that are common to both processing and exposure) –

  • unique_only

  • table. (bool. Only return columns that are not found in an exposure) –

Returns:

colnames, list. List of column names for an processing table. coldtypes, list. List of column datatypes for the names in colnames. coldeflts, list. Optionally returned if return_default_values is True. List of default values for the corresponding colnames.

desispec.workflow.proctable.get_processing_table_name(specprod=None, prodmod=None, extension='csv')[source]

Defines the default processing name given the specprod of the production and the optional extension.

Parameters:
  • specprod

  • None (str or None. The name of the production. If) –

  • variable. (it will be taken from the environment) –

  • prodmod – Used in daily workflow to add the night to the name and make it unique from other nightly tables.

  • it. (str. Additional str that can be added to the production table name to further differentiate) – Used in daily workflow to add the night to the name and make it unique from other nightly tables.

  • extension (str. The) – Default is ‘csv’.

  • extension – Default is ‘csv’.

Returns:

str. The processing table name given the input night and extension.

desispec.workflow.proctable.get_processing_table_path(specprod=None)[source]

Defines the default path to save a processing table. If specprod is not given, the environment variable ‘SPECPROD’ must exist.

Parameters:
  • specprod

  • None (str or None. The name of the production. If) –

  • variable. (it will be taken from the environment) –

Returns:

str. The full path to the directory where the processing table should be written (or is already written). This

does not including the filename.

desispec.workflow.proctable.get_processing_table_pathname(specprod=None, prodmod=None, extension='csv')[source]

Defines the default pathname to save a processing table.

Parameters:
  • specprod

  • None (str or None. The name of the production. If) –

  • variable. (it will be taken from the environment) –

  • prodmod – Used in daily workflow to add the night to the name and make it unique from other nightly tables.

  • it. (str. Additional str that can be added to the production table name to further differentiate) – Used in daily workflow to add the night to the name and make it unique from other nightly tables.

  • extension (str. The) – Default is ‘csv’.

  • extension – Default is ‘csv’.

Returns:

str. The full pathname where the processing table should be written (or is already written). This

includes the filename.

desispec.workflow.proctable.instantiate_processing_table(colnames=None, coldtypes=None, rows=None)[source]

Create an empty processing table with proper column names and datatypes. If rows is given, it inserts the rows into the table, otherwise it returns a table with no rows.

Parameters:
  • colnames

  • table. (list. List of column names for a procesing) –

  • coldtypes

  • colnames. (list. List of column datatypes for the names in) –

  • rows – pairs that match the default column names and data types of the default exposure table.

  • value (list or np.array of Table.Rows or dicts. An iterable set of Table.Row's or dicts with keys/colnames and) – pairs that match the default column names and data types of the default exposure table.

Returns:

processing_table, Table. An astropy Table with the column names and data types for a DESI workflow processing

table. If the input rows was not None, it contains those rows, otherwise it has no rows.

desispec.workflow.proctable.table_row_to_dict(table_row)[source]

Helper function to convert a table row to a dictionary, which is much easier to work with for some applications

Parameters:
  • table_row – each key is a column name and the values are the column entry.

  • where (Table.Row or dict. The row of an astropy table that you want to convert into a dictionary) – each key is a column name and the values are the column entry.

Returns:

out, dict. Dictionary where each key is a column name and the values are the column entry.

desispec.workflow.queue

desispec.workflow.queue.any_jobs_not_complete(statuses, termination_states=None)[source]

Returns True if any of the job statuses in the input column of the processing table, statuses, are not complete (as based on the list of acceptable final states, termination_states, given as an argument. These should be states that are viewed as final, as opposed to job states that require resubmission.

Parameters:
  • statuses – be a string.

  • should (Table.Column or list or np.array. The statuses in the processing table "STATUS". Each element) – be a string.

  • termination_states – by the Slurm scheduler that should be deemed terminal state.

  • returned (list or np.array. Each element should be a string signifying a state that is) – by the Slurm scheduler that should be deemed terminal state.

Returns:

bool. True if any of the statuses of the jobs given in statuses are NOT a member of the termination states.

Otherwise returns False.

desispec.workflow.queue.get_resubmission_states()[source]

Defines what Slurm job failure modes should be resubmitted in the hopes of the job succeeding the next time.

Possible values that Slurm returns are:

CA or ca or CANCELLED for cancelled jobs will only show currently running jobs in queue unless times are explicitly given
BF BOOT_FAIL   Job terminated due to launch failure
CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
CD COMPLETED Job has terminated all processes on all nodes with an exit code of zero.
DL DEADLINE Job terminated on deadline.
F FAILED Job terminated with non-zero exit code or other failure condition.
NF NODE_FAIL Job terminated due to failure of one or more allocated nodes.
OOM OUT_OF_MEMORY Job experienced out of memory error.
PD PENDING Job is awaiting resource allocation.
PR PREEMPTED Job terminated due to preemption.
R RUNNING Job currently has an allocation.
RQ REQUEUED Job was requeued.
RS RESIZING Job is about to change size.
RV REVOKED Sibling was removed from cluster due to other cluster starting the job.
S SUSPENDED Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.
TO TIMEOUT Job terminated upon reaching its time limit.
Returns:

list. A list of strings outlining the job states that should be resubmitted.

desispec.workflow.queue.get_termination_states()[source]

Defines what Slurm job states that are final and aren’t in question about needing resubmission.

Possible values that Slurm returns are:

CA or ca or CANCELLED for cancelled jobs will only show currently running jobs in queue unless times are explicitly given
BF BOOT_FAIL   Job terminated due to launch failure
CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
CD COMPLETED Job has terminated all processes on all nodes with an exit code of zero.
DL DEADLINE Job terminated on deadline.
F FAILED Job terminated with non-zero exit code or other failure condition.
NF NODE_FAIL Job terminated due to failure of one or more allocated nodes.
OOM OUT_OF_MEMORY Job experienced out of memory error.
PD PENDING Job is awaiting resource allocation.
PR PREEMPTED Job terminated due to preemption.
R RUNNING Job currently has an allocation.
RQ REQUEUED Job was requeued.
RS RESIZING Job is about to change size.
RV REVOKED Sibling was removed from cluster due to other cluster starting the job.
S SUSPENDED Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.
TO TIMEOUT Job terminated upon reaching its time limit.
Returns:

list. A list of strings outlining the job states that are considered final (without human investigation/intervention)

desispec.workflow.queue.queue_info_from_qids(qids, columns='jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode', dry_run=0)[source]

Queries the NERSC Slurm database using sacct with appropriate flags to get information within a specified time window of all jobs submitted or executed during that time.

Parameters:
  • jobids (list or array of ints) – Slurm QID’s at NERSC that you want to return information about.

  • columns (str) – Comma seperated string of valid sacct column names, in lower case. To be useful for the workflow, it should have MUST have columns “JOBID” and “STATE”. Other columns available that aren’t included in the default list are: jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode. Other options include: suspended,derivedexitcode,reason,priority,jobname.

  • dry_run (int) – Whether this is a simulated run or real run. If nonzero, it is a simulation and it returns a default table that doesn’t query the Slurm scheduler.

Returns:

Table with the columns defined by the input variable ‘columns’ and information relating to all jobs submitted by the specified user in the specified time frame.

Return type:

Table

desispec.workflow.queue.queue_info_from_time_window(start_time=None, end_time=None, user=None, columns='jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode', dry_run=0)[source]

Queries the NERSC Slurm database using sacct with appropriate flags to get information within a specified time window of all jobs submitted or executed during that time.

Parameters:
  • start_time (str) – String of the form YYYY-mm-ddTHH:MM:SS. Based on the given night and the earliest hour you want to see queue information about.

  • end_time (str) – String of the form YYYY-mm-ddTHH:MM:SS. Based on the given night and the latest hour you want to see queue information about.

  • user (str) – The username at NERSC that you want job information about. The default is an the environment name if if exists, otherwise ‘desi’.

  • columns (str) – Comma seperated string of valid sacct column names, in lower case. To be useful for the workflow, it should have MUST have columns “JOBID” and “STATE”. Other columns available that aren’t included in the default list are: jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode. Other options include: suspended,derivedexitcode,reason,priority,jobname.

  • dry_run (int) – Whether this is a simulated run or real run. If nonzero, it is a simulation and it returns a default table that doesn’t query the Slurm scheduler.

Returns:

Table with the columns defined by the input variable ‘columns’ and information relating to all jobs submitted by the specified user in the specified time frame.

Return type:

Table

desispec.workflow.queue.update_from_queue(ptable, qtable=None, dry_run=0, ignore_scriptnames=False)[source]

Given an input prcessing table (ptable) and query table from the Slurm queue (qtable) it cross matches the Slurm job ID’s and updates the ‘state’ in the table using the current state in the Slurm scheduler system.

Parameters:
  • ptable – have at least columnns ‘LATEST_QID’ and ‘STATUS’.

  • Must (Table. Processing table that contains the jobs you want updated with the most recent queue table.) – have at least columnns ‘LATEST_QID’ and ‘STATUS’.

  • qtable – to all jobs submitted by the specified user in the specified time frame.

  • relating (Table. Table with the columns defined by the input variable 'columns' and information) – to all jobs submitted by the specified user in the specified time frame.

  • ignore_scriptnames – want to check whether the scriptname matches the jobname return by the slurm scheduler.

  • not (bool. Default is False. Set to true if you do) – want to check whether the scriptname matches the jobname return by the slurm scheduler.

  • provided (The following are only used if qtable is not) –

    dry_run, int. Whether this is a simulated run or real run. If nonzero, it is a simulation and it returns a default

    table that doesn’t query the Slurm scheduler.

Returns:

ptable, Table. The same processing table as the input except that the “STATUS” column in ptable for all jobs is

updated based on the ‘STATE’ in the qtable (as matched by “LATEST_QID” in the ptable and “JOBID” in the qtable).

desispec.workflow.redshifts

desispec.workflow.redshifts.create_desi_zproc_batch_script(group, tileid=None, cameras=None, thrunight=None, nights=None, expids=None, healpix=None, survey=None, program=None, queue='regular', batch_opts=None, runtime=None, timingfile=None, batchdir=None, jobname=None, cmdline=None, system_name=None, max_gpuprocs=None, no_gpu=False, run_zmtl=False, no_afterburners=False)[source]

Generate a SLURM batch script to be submitted to the slurm scheduler to run desi_proc.

Parameters:
  • group (str) – Description of the job to be performed. zproc options include: ‘perexp’, ‘pernight’, ‘cumulative’.

  • tileid (int) – The tile id for the data.

  • cameras (str or list of str) – List of cameras to include in the processing or a camword.

  • thrunight (int) – For group=cumulative, include exposures through this night

  • nights (list of int) – The nights the data was acquired.

  • expids (list of int) – The exposure id(s) for the data.

  • healpix (list of int) – healpixels to process (group=’healpix’)

  • queue (str) – Queue to be used.

  • batch_opts (str) – Other options to give to the slurm batch scheduler (written into the script).

  • runtime (str) – Timeout wall clock time in minutes.

  • timingfile (str) – Specify the name of the timing file.

  • batchdir (str) – can define an alternative location to write the file. The default is to SPECPROD under run/scripts/tiles/GROUP/TILE/NIGHT

  • jobname (str) – name to save this batch script file as and the name of the eventual log file. Script is save within the batchdir directory.

  • cmdline (str or list of str) – Complete command as would be given in terminal to run the desi_zproc, or list of args. Can be used instead of reading from argv.

  • system_name (str) – name of batch system, e.g. cori-haswell, cori-knl

  • max_gpuprocs (int) – Number of gpu processes

  • no_gpu (bool) – Default false. If true it doesn’t use GPU’s even if available.

  • run_zmtl (bool) – Default false. If true it runs zmtl.

  • no_afterburners (bool) – Default false. If true it doesn’t run afterburners.

Returns:

the full path name for the script written.

Return type:

scriptfile

Note

batchdir and jobname can be used to define an alternative pathname, but

may not work with assumptions in the spectro pipeline.

desispec.workflow.redshifts.get_zpix_redshift_script_pathname(healpix, survey, program)[source]

Return healpix-based coadd+redshift+afterburner script pathname

Parameters:
  • healpix (int or array-like) – healpixel(s)

  • survey (str) – DESI survey, e.g. main, sv1, sv3

  • program (str) – survey program, e.g. dark, bright, backup

Returns:

zpix_script_pathname

desispec.workflow.redshifts.get_ztile_relpath(tileid, group, night=None, expid=None)[source]

Determine the relative output directory of the tile redshift batch script for spectra+coadd+redshifts for a tile

Parameters:
  • tileid (int) – Tile ID

  • group (str) – cumulative, pernight, perexp, or a custom name

  • night (int) – Night

  • expid (int) – Exposure ID

Returns:

the relative path of output directory of the batch script from the specprod/run/scripts

Return type:

outdir (str)

desispec.workflow.redshifts.get_ztile_script_pathname(tileid, group, night=None, expid=None)[source]

Generate the pathname of the tile redshift batch script for spectra+coadd+redshifts for a tile

Parameters:
  • tileid (int) – Tile ID

  • group (str) – cumulative, pernight, perexp, or a custom name

  • night (int) – Night

  • expid (int) – Exposure ID

Returns:

the pathname of the tile redshift batch script

Return type:

(str)

desispec.workflow.redshifts.get_ztile_script_suffix(tileid, group, night=None, expid=None)[source]

Generate the suffix of the tile redshift batch script for spectra+coadd+redshifts for a tile

Parameters:
  • tileid (int) – Tile ID

  • group (str) – cumulative, pernight, perexp, or a custom name

  • night (int) – Night

  • expid (int) – Exposure ID

Returns:

the suffix of the batch script

Return type:

suffix (str)

desispec.workflow.redshifts.read_minimal_exptables_columns(nights=None, tileids=None)[source]

Read exposure tables while handling evolving formats

Parameters:
  • nights (list of int) – nights to include (default all nights found)

  • tileids (list of int) – tileids to include (default all tiles found)

Returns exptable with just columns TILEID, NIGHT, EXPID, ‘CAMWORD’,

‘BADCAMWORD’, filtered by science exposures with LASTSTEP=’all’ and TILEID>=0

Note: the returned table is the full pipeline exposures table. It is trimmed

to science exposures that have LASTSTEP==’all’

desispec.workflow.schedule

Tools for scheduling MPI jobs using mpi4py.

desispec.workflow.tableio

desispec.workflow.tableio.ensure_scalar(val, joinsymb='|', comma_replacement=';')[source]

Ensures that the object in val is a scalar that can be save to a Table cell (i.e. row of a column or column of a row). If the it is an array or list, it uses joinsymb to turn them into a single string.

Parameters:
  • val (a scalar datatype, list, or array) – The value to be converted to a scalar quantity (returning the val if it is already a scalar).

  • joinsymb (str) – A string symbol other than comma that will be used to join the multiple values of a list or array.

  • comma_replacement (str) – A string symbol that should be used to replace any existing commas in the data, such that the value can be saved in a csv format.

Returns:

The output string which is a scalar quantity capable of being written to a single table cell (in a csv or fits file, for example).

Return type:

val or outstr, any scalar type or string

desispec.workflow.tableio.guess_default_by_dtype(typ)[source]

Returns a default value given a data type. To be used in filling a table if no default is given.

Parameters:

typ (DataType) – The datatype of the element you want a default value for.

Returns:

default value for that type. Can be int, float, str, list, or array. If it can’t guess, it returns the integer -99 .

Return type:

number

desispec.workflow.tableio.load_table(tablename=None, tabletype=None, joinsymb='|', verbose=False, process_mixins=True, use_specprod=True, suppress_logging=False)[source]

Workflow function to read in exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are generated from strings using the joinsymb. It reads from the file given by tablename (or the default for table of type tabletype).

Parameters:
  • tablename (str) – Full pathname of where the table should be saved, including the extension. Originally save to *.temp.{ext} and then moved to *.{ext}. If None, it looks up the default for typetable. If tabletype is None it uses this to try and identify the tabletype and uses that to get the default column names and types.

  • tabletype (str) – ‘exptable’, ‘proctable’, or ‘unproctable’. Used if tablename is None to get the default name for the type of table. Also used to get the column datatypes and defaults.

  • joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.

  • verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.

  • process_mixins (bool) – Whether to look for and try to split strings into lists/arrays. The default is True. Warning: The exposure and processing tables have default data types which are multi-value. If this is set to False, the default data types will be incorrect and issues are likely to arise.

  • use_specprod (bool) – If True and tablename not specified and tabletype is exposure table, this looks for the table in the SPECPROD rather than the exptab repository. Default is True.

  • suppress_logging (bool) – If True, the log.info() messages are skipped. This is useful in scripts looping over many tables to reduce the amount of things printed to the screen.

Returns:

table – Either exposure table or processing table that was loaded from tablename (or from default name based on tabletype). Returns None if the file doesn’t exist.

Return type:

Table

desispec.workflow.tableio.load_tables(tablenames=None, tabletypes=None, verbose=False)[source]

Workflow function to read in multiple exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are generated from strings using the joinsymb. It reads from the files given by tablenames (or the default for tables of types in tabletypes).

Parameters:
  • tablename (list/array of str) – List or array of the full pathnames of where the tables should be saved, including the extension.

  • tabletype (list/array of str) – List or array of the table types, which are used if tablenames is None to get the default name for the type of table. They are also used to get the column datatypes and defaults.

  • verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.

Returns:

tabs – Either exposure table or processing table that was loaded from tablename (or from default name based on tabletype). Returns None if the file doesn’t exist.

Return type:

list of Table

desispec.workflow.tableio.process_column(data, typ, mask=None, default=None, joinsymb='|', process_mixins=True, comma_replacement=';', verbose=False)[source]

Used with load_table to process a Table.Column after being read in. It fills in masked values with defaults, and identifies and splits mixin columns (columns that should be a list/array) back into their list/array from their string representation.

Parameters:
  • data (Table.Column or Table.MaskedColumn) – Column of data to be checked for masked rows (to be filled with default) and string-ed versions of lists/arrays that need to be expanded out.

  • typ (DataType) – The expected datatype of the data in data. May differ from the type of the input data, in which case the data will be transformed.

  • mask (np.array) – A mask array with True in row elements of the input data array that are masked and False in row elements that are not masked.

  • default (any type) – The default value to be used for masked rows.

  • joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.

  • process_mixins (bool) – Whether to look for and try to split strings into lists/arrays. The default is True. Warning: The exposure and processing tables have default data types which are multi-value. If this is set to False, the default data types will be incorrect and issues are likely to arise.

  • comma_replacement (str) – Replace instances of this symbol with commas when loading scalar columns in a table, as e.g. BADAMPS is used in the pipeline and symbols like ‘;’ are problematic on the command line.

  • verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.

Returns:

  • col, list or np.array – A new data vector similar to input ‘data’ except with masked values filled in and mixin strings expanded back into np.array’s.

  • DataType – The data type of a row element in the return col.

desispec.workflow.tableio.split_str(val, joinsymb='|', comma_replacement=';')[source]

Attempts to intelligently interpret an input scalar. If it is a string it looks to see if it was a list or array objects that was joined to be a single string using joinsymb. If it identifies that, it will split that into the original list/array. Otherwise it will return the val as-is.

Parameters:
  • val (any datatype) – The input to be checked to see if it is in fact a list/array that was joined into a string for saving in a Table.

  • joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.

  • comma_replacement (str) – Replace instances of this symbol with commas when loading ONLY scalar columns in a table, as e.g. BADAMPS is used in the pipeline and symbols like ‘;’ are problematic on the command line. Comment arrays do not need to be converted back and forth.

Return type:

val or split_list, any datatype or np.array.

desispec.workflow.tableio.standardize_tabletype(tabletype)[source]

Given the user defined type of table it returns the proper ‘tabletype’ expected by the pipeline

Parameters:

tabletype (str) – Allows for a flexible number of input options, but should refer to either the ‘exposure’, ‘processing’, or ‘unprocessed’ table types.

Returns:

tabletype – Standardized tabletype values. Either “exptable”, “proctable”, “unproctable”.

Return type:

str

desispec.workflow.tableio.translate_type_to_pathname(tabletype, use_specprod=True)[source]

Given the type of table it returns the proper file pathname

Parameters:
  • tabletype (str) – Allows for a flexible number of input options, but should refer to either the ‘exposure’, ‘processing’, or ‘unprocessed’ table types.

  • use_specprod (bool) – If True and tablename not specified and tabletype is exposure table, this looks for the table in the SPECPROD rather than the exptab repository. Default is True.

Returns:

tablename – Full pathname including extension of the table type. Uses environment variables to determine the location.

Return type:

str

desispec.workflow.tableio.write_table(origtable, tablename=None, tabletype=None, joinsymb='|', overwrite=True, verbose=False, comma_replacement=';', write_empty=False, use_specprod=True)[source]

Workflow function to write exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are reduced to strings using the joinsymb. It writes to a temp file before moving the fully written file to the name given by tablename (or the default for table of type tabletype).

Parameters:
  • origtable (Table) – Either exposure table or processing table.

  • tablename (str) – Full pathname of where the table should be saved, including the extension. Originally save to *.temp.{ext} and then moved to *.{ext}. If None, it looks up the default for typetable.

  • tabletype (str) – Used if tablename is None to get the default name for the type of table.

  • joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.

  • overwrite (bool) – Whether to overwrite the file on disk if it already exists. Default is currently True.

  • verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.

  • write_empty (bool) – Whether to write an empty table to disk. The default is False. Warning: code is less robust to column datatypes on read/write if the table is empty. May cause issues if this is set to True.

  • comma_replacement (str) – Replace instances of this symbol with commas when loading scalar columns in a table, as e.g. BADAMPS is used in the pipeline and symbols like ‘;’ are problematic on the command line.

  • use_specprod (bool) – If True and tablename not specified and tabletype is exposure table, this looks for the table in the SPECPROD rather than the exptab repository. Default is True.

desispec.workflow.tableio.write_tables(tables, tablenames=None, tabletypes=None, write_empty=False, verbose=False, overwrite=True)[source]

Workflow function to write multiple exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are reduced to strings. It writes to a temp file before moving the fully written file to the name given by tablenames (or the default for table of types tabletypes).

Parameters:
  • tables (list/array of Table) – List or array of exposure tables, unprocessed tables, and/or processing table.

  • tablenames (list/array of str) – List or array of the full pathnames to where the tables should be saved, including the extension. If None, it looks up the default for each of tabletypes.

  • tabletype (list/array of str) – List or array of table types to be used if tablenames is None to get the default name for each type of table.

  • write_empty (bool) – Whether to write an empty table to disk. The default is False. Warning: code is less robust to column datatypes on read/write if the table is empty. May cause issues if this is set to True.

  • overwrite (bool) – Whether to overwrite the file on disk if it already exists. Default is currently True.

  • verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.

desispec.workflow.timing

desispec.workflow.timing.during_operating_hours(dry_run=False, starthour=None, endhour=None)[source]

Determines if the desi_daily_proc_manager should be running or not based on the time of day. Can be overwridden with dry_run for testing purposes.

Parameters:
  • dry_run – bool. If true, this is a simulation so return True so that the simulation can proceed.

  • starthour – str or int. The number of hours (between 0 and 24) after midnight.

  • endhour – str or int. The number (between 0 and 24) of hours after midnight. Assumes an endhour smaller than starthour implies the following day.

Returns:

bool. True if dry_run is true OR if the current time is between the starthour and endhour.

desispec.workflow.timing.ensure_tucson_time()[source]

Define the start and end of a ‘night’ based on times at the mountain. So ensure that the times are with respect to Arizona.

desispec.workflow.timing.get_nightly_end_time()[source]

Defines when a night ends for desi_daily_proc_manager in local Tucson time. Once this time is exceeded, the manager will enter into queue cleanup mode and then exit when the jobs have finished. End of night is altered slightly for summer vs. winter.

Returns:

int. The number of hours after midnight that signifies the end of a night of observing.

Return type:

end_night

desispec.workflow.timing.get_nightly_start_time()[source]

Defines the time of day that the desi_daily_proc_manager should start (in Tucson local time). Before this time, the manager being woken by a cron job will exit immediately. Selected to give plenty of time between end of night and start of the next, but early enough to catch and begin running afternoon calibrations.

Returns:

int. The number of hours after midnight that signifies the start of a new night of observing.

Return type:

14

desispec.workflow.timing.nersc_end_time(night=None, endhour=None)[source]

Transforms a night and time into a YYYY-MM-DD[THH:MM[:SS]] time string Slurm can interpret. Correctly accounts for the fact that the night is defined starting at Noon on a given day.

Parameters:
  • night – str or int. In the form YYYMMDD, the night the jobs are being run.

  • endhour – str or int. The number (between 0 and 24) of hours after midnight where you stop submitting jobs to the queue.

Returns:

MM:SS. Based on the given night and endhour

Return type:

str. String of the form YYYY-mm-ddTHH

desispec.workflow.timing.nersc_format_datetime(timetup=None)[source]

Given a time tuple from the time module, this will return a string in the proper format to be properly interpreted by the NERSC Slurm queue scheduler.

Parameters:

timetup – tuple. A time.time() tuple object representing the time you want to trasnform into a Slurm readable string.

Returns:

MM:SS.

Return type:

str. String of the form YYYY-mm-ddTHH

desispec.workflow.timing.nersc_start_time(night=None, starthour=None)[source]

Transforms a night and time into a YYYY-MM-DD[THH:MM[:SS]] time string Slurm can interpret

Parameters:
  • night – str or int. In the form YYYMMDD, the night the jobs are being run.

  • starthour – str or int. The number of hours (between 0 and 24) after midnight where you began submitting jobs to the queue.

Returns:

MM:SS. Based on the given night and starthour

Return type:

str. String of the form YYYY-mm-ddTHH

desispec.workflow.timing.what_night_is_it()[source]

Return the current night

desispec.workflow.utils

desispec.workflow.utils.check_running(proc_name='desi_daily_proc_manager', suppress_outputs=False)[source]

Check if the given process name is running. Default is desi_daily_proc_manager.

Parameters:
  • proc_name

  • tables. (str. The name of the process as it would appear in the os's process) –

  • suppress_outputs

  • printed. (bool. True if you don't want to anything to be) –

Returns:

running, bool. True if the process name was found in the list of processes and has a pid different that

the current process (signifying a second instance of that program).

desispec.workflow.utils.define_variable_from_environment(env_name, var_descr)[source]

Returns the environment variable if it exists, otherwise raises an error telling the user that a variable must be specified either directly or by defining the environment variable. It exits with exit code 1 if no environment variable exists.

Parameters:
  • env_name

  • variable. (str. The name of the environment) –

  • var_descr

  • env_name. (str. A description of the variable you are trying to define with environment variable) –

Returns:

str or Nothing. If the environment variable exists, it returns the value of that environment variable.

Otherwise it raises an exit code with status 1.

desispec.workflow.utils.get_json_dict(reqpath)[source]

Return a dictionary representation of the json file at the specified location. If it doesn’t exist, return an empty dictionary.

Parameters:
  • reqpath – as it is accurate based on the current working directory.

  • given (str. The full pathname including file name of the json file. A relative path can be) – as it is accurate based on the current working directory.

  • long (so) – as it is accurate based on the current working directory.

Retrun:

req_dict, dict. A dictionary with keys and values defined by the json file.

desispec.workflow.utils.get_printable_banner(input_str=None)[source]

Returns a string that when printed shows a banner with the given string in the center.

Parameters:

input_str – str. Any string of moderate length roughly less than 30 that you want placed inside a pound/hashtag (#) banner.

Returns:

str. A banner comprised of ascii pound symbols (#) and the given string in the center.

Return type:

banner

desispec.workflow.utils.globpath(*args)[source]

Helper function that takes the same arguments as os.path.join (i.e. a comma separated set of directory names, paths, and file names that can be joined to created a single unified file path), and returns the glob’ed list of files in that location that matched the specified parameters.

Parameters:
  • args (i.e. pathjoin) – for which you want a list of files in that location. Any of the strings can be wildcards used by glob.glob, so long as they don’t confuse os.path.join().

  • os.path.join (undetermined number of input args that are each a string. The values to be given to) – for which you want a list of files in that location. Any of the strings can be wildcards used by glob.glob, so long as they don’t confuse os.path.join().

Returns:

list. Sorted list of files in the location defined by the input args that are consistent with all wildcards (if

any). Ignores Mac file .DS_STORE. If location doesn’t exist, it returns an empty list.

desispec.workflow.utils.listpath(*args)[source]

Helper function that takes the same arguments as os.path.join (i.e. a comma separated set of directory names, paths, and file names that can be joined to created a single unified file path), and returns the list of files in that location.

Parameters:
  • args (i.e. pathjoin) – for which you want a list of files in that location.

  • os.path.join (undetermined number of input args that are each a string. The values to be given to) – for which you want a list of files in that location.

Returns:

list. Sorted list of files in the location defined by the input args. Ignores Mac file .DS_STORE. If location

doesn’t exist, it returns an empty list.

desispec.workflow.utils.sleep_and_report(sleep_duration, message_suffix='', logfunc=<built-in function print>, dry_run=False)[source]

Sleeps for the given number of seconds, printing a message before and afterward

Parameters:
  • sleep_duration – int or float. Number of seconds for the process to sleep.

  • message_suffix – str. The message to be printed after “Sleeping XXs (suffix)” prior to sleeping.

  • logfunc – func. Default is print. The function used to serve the message to the user.

  • dry_run – bool. Default is False. Whether to perform the action (dry_run=False) or pretend (True).

desispec.workflow.utils.verify_variable_with_environment(var, var_name, env_name)[source]

Helper function that assigns a variable based on the inputs and gives relevant outputs to understand what is being done. If the variable is defined, it will make sure that the environment variable (used by some lower-level code) is consistent before returning the user specified value. If it is not specified, then the environment variable is used. If the environment variable is also undefined, then it gives a useful output and then exits.

Parameters:
  • var

  • variable. (str. The name of the environment variable that would hold the value relevant to the var) –

  • var_name (used exclusively for outputting useful messages) –

  • variable (str. The name of the) –

  • env_name

  • variable.

Returns:

var, any type. Either the input var if that is not NoneType. Otherwise the value from the environment variable. If neither is defined it exits with status 1 rather than returning anything.

desispec.xytraceset

Lightweight wrapper class for trace coordinates and wavelength solution, to be returned by read_xytraceset().

desispec.zcatalog

The zcatalog.py script contains the following utility functions for redshift catalogs:

  1. find_primary_spectra:

Given an input table with possibly multiple spectra per TARGETID, this function returns arrays for a primary flag (whether a spectrum is the primary [“best”] spectrum based on the ZWARN value and on a sort column [default=’TSNR2_LRG’]) and for the number of spectra per TARGETID.

Usage:

nspec, spec_primary = find_primary_spectra(table, sort_column = 'TSNR2_LRG')
  1. create_summary_catalog:

This function combines individual redshift catalogs from a given release into a single compilation catalog. The output catalog is saved in a FITS file in a user-specified location and filename.

This function can be used for ‘fuji’ (EDR) or ‘guadalupe’ (Main) by setting the keyword specprod. By default, this function aggregates all the columns for the redshift catalogs, and adds columns to quantify the number of coadded spectra listed in the catalogs for each TARGETID and to identify the primary (“best”) spectrum out of them using the find_primary_spectra function. Optionally, the script can return a list of pre-selected “summary” column, or a list of user-specified columns via the columns_list keyword.

Usage:

create_summary_catalog(specprod, specgroup = 'zpix', all_columns = True,                           columns_list = None, output_filename = './zcat-all.fits')

Ragadeepika Pucha, Stephanie Juneau, and DESI data team Version: 2022, March 31st

desispec.zcatalog._get_survey_program_from_filename(filename)[source]

Return SURVEY,PROGRAM parsed from zpix/ztile filename; fragile!

desispec.zcatalog.create_summary_catalog(specgroup, indir=None, specprod=None, all_columns=True, columns_list=None, output_filename=None)[source]

This function combines all the individual redshift catalogs for either ‘zpix’ or ‘ztile’ with the desired columns (all columns, or a pre-selected list, or a user-given list). It further adds ‘NSPEC’ and ‘PRIMARY’ columns, two for SV(or MAIN), and two for the entire combined redshift catalog.

Parameters:
  • specgroup (str) – The option to run the code on ztile* files or zpix* files. It can either be ‘zpix’ or ‘ztile’

  • indir (str) – Input directory to look for zpix/ztile files.

  • specprod (str) – Internal Release Name for the DESI Spectral Release. Used to derive input directory if indir is not provided.

  • all_columns (bool) – Whether or not to include all the columns into the final table. Default is True.

  • columns_list (list) – If all_columns = False, list of columns to include in the final table. If None, a list of pre-decided summary columns will be used. Default is None. The ‘SV/MAIN’ primary flag columns as well as the primary flag columns for the entire catalog witll be included.

  • output_filename (str) – Path+Filename for the output summary redshift catalog. The output FITS file will be saved at this path. If not specified, the output filename will be derived from specgroup and $SPECPROD

Returns:

The function saves a FITS file at the location and file name specified by output_filename with the summary redshift catalog in HDU1.

Return type:

None

desispec.zcatalog.find_primary_spectra(table, sort_column='TSNR2_LRG')[source]

Function to select the best “primary” spectrum for objects with multiple (coadded) spectra.

The best spectrum is selected based on the following steps:

  1. Spectra with ZWARN=0 are given top preference.

  2. Among multiple entries with ZWARN=0, spectra with the highest value of ‘sort_column’ are given the preference.

  3. If there are no spectra with ZWARN=0, spectra with the highest value of ‘sort_column’ are given the preference.

Parameters:
  • table (Numpy array or Astropy Table) – The input table should be a redshift catalog, with at least the following columns: ‘TARGETID’, ‘ZWARN’ and the sort_column (Default:’TSNR2_LRG’) Additional columns will be ignored.

  • sort_column (str) – Column name that will be used to select the best spectrum. Default = ‘TSNR2_LRG’ The higher values of the column will be considered as better (descending order).

Returns:

  • nspec (Numpy int array) – Array of number of entries available per target

  • spec_primary (Numpy bool array) – Array of spec_primary (= TRUE for the best spectrum)

desispec.zcatalog.update_table_columns(table, specgroup='zpix', all_columns=True, columns_list=None)[source]

This function fills the *TARGET masked columns and returns the final table with the required columns.

Parameters:
  • table (Astropy Table) – A table.

  • specgroup (str) – The option to run the code on ztile* files or zpix* files. It can either be ‘zpix’ or ‘ztile’. Default is ‘zpix’

  • all_columns (bool) – Whether or not to include all the columns into the final table. Default is True.

  • columns_list (list) – If all_columns = False, list of columns to include in the final table. If None, a list of pre-decided summary columns will be used. Default is None. The ‘SV/MAIN’ primary flag columns as well as the primary flag columns for the entire catalog witll be included.

Returns:

t_final – Final table with non-masked columns with required columns.

Return type:

Astropy Table

desispec.zmtl

Post-redrock ML processing for LyA Quasar object identification for creating zmtl files to be used by the merged target list (MTL) updates.

desispec.zmtl.add_abs_data(zmtl, coaddname)[source]

Add the MgII absorption line finder data to the input zmtl array.

Parameters:
  • zmtl (:class:'~numpy.array`) – The structured array that was created by make_new_zmtl()

  • coaddname (class:str) – The name of the coadd file corresponding to the redrock file used in make_new_zmtl()

Returns:

The zmtl array with MgII Absorption data included in the columns:

  • Z_ABS - The highest redshift of MgII absorption

  • Z_ABS_CONF - The confidence value for this redshift.

Return type:

array

Notes

  • The original function was written by Lucas Napolitano (LGN) and modified for this script by Eleanor Lyke (EBL).

desispec.zmtl.add_qn_data(zmtl, coaddname, qnp_model, qnp_lines, qnp_lines_bal)[source]

Apply the QuasarNP model to the input zmtl and add data to columns.

Parameters:
  • zmtl (array) – The structured array that was created by make_new_zmtl()

  • coaddname (str) – The name of the coadd file corresponding to the redrock file used in make_new_zmtl()

  • qnp_model (h5.array) – The array containing the pre-trained QuasarNP model.

  • qnp_lines (list) – A list containing the names of the emission lines that quasarnp.process_preds() should use.

  • qnp_lines_bal (list) – A list containing the names of the emission lines to check for BAL troughs.

Returns:

The zmtl array with QuasarNP data included in the columns:

  • Z_QN - The best QuasarNP redshift for the object

  • Z_QN_CONF - The confidence of Z_QN

  • IS_QSO_QN - A binary flag indicated object is a quasar

Return type:

array

desispec.zmtl.add_sq_data(zmtl, coaddname, squeze_model)[source]

Apply the SQUEzE model to the input zmtl and add data to columns.

Parameters:
  • zmtl (array) – The structured array that was created by make_new_zmtl()

  • coaddname (class:str) – The name of the coadd file corresponding to the redrock file used in make_new_zmtl()

  • squeze_model (numpy.array) – The loaded SQUEzE model file

Returns:

The zmtl array with SQUEzE data included in the columns:

  • Z_SQ - The best redshift from SQUEzE for each object.

  • Z_SQ_CONF - The confidence value of this redshift.

Return type:

array

desispec.zmtl.add_tileqa_data(zmtl, tileqafile)[source]

Modifies zmtl[‘ZWARN’] in-place to add tile QA flags

Parameters:
  • zmtl (array) – The structured array that was created by make_new_zmtl()

  • tileqafile (str) – The full filepath of the tile-qa file

  • not (Returns boolean array for whether targets were flagged as bad (True) or) –

desispec.zmtl.create_zmtl(zmtldir, outputdir, tile=None, night=None, petal_num=None, qn_flag=False, qnp_model=None, qnp_model_file=None, qnp_lines=None, qnp_lines_bal=None, sq_flag=False, squeze_model=None, squeze_model_file=None, abs_flag=False, zcomb_flag=False)[source]

This will create a single zmtl file from a set of user inputs.

Parameters:
  • zmtldir (str) –

    If any of tile, night or petal_num are None:

    The name of a redrock redrock file.

    If none of tile, night and petal_num are None:

    The root directory from which to read redrock and coadd spectro files. The full directory is constructed as zmtldir + tile + night, with files redrock-/coadd-petal_num`*`night.fits.

  • outputdir (str) –

    If any of tile, night or petal_num are None:

    The name of an output file.

    If none of tile, night and petal_num are None:

    The output directory to which to write the output file. The full directory is constructed as outputdir + tile + night, with file zmtl-petal_num`*`night.fits.

  • tile (int) – The TILEID of the tile to process.

  • night (int) – The date associated with the observation of the ‘tile’ used. * Must be in YYYYMMDD format

  • petal_num (int) – If ‘all_petals’ isn’t used, the single petal to create a zmtl for.

  • qn_flag (bool, optional) – Flag to add QuasarNP data (or not) to the zmtl file.

  • qnp_model (h5 array, optional) – The QuasarNP model file to be used for line predictions.

  • qnp_model_file (str, optional) – File from which to load the QuasarNP model (qnp_model), qnp_lines and qnp_lines_bal if qnp_model is None. Also written to the output header of the zmtl file.

  • qnp_lines (list, optional) – The list of lines to use in the QuasarNP model to test against.

  • qnp_lines_bal (list, optional) – The list of BAL lines to use for QuasarNP to identify BALs.

  • sq_flag (bool, optional) – Flag to add SQUEzE data (or not) to the zmtl file.

  • squeze_model (numpy.array, optional) – The numpy array for the SQUEzE model file.

  • squeze_model_file (str, optional) – File from which to load the SQUEzE model if squeze_model is None. Also written to the output header of the zmtl file.

  • abs_flag (bool, optional) – Flag to add MgII Absorption data (or not) to the zmtl file.

  • zcomb_flag (bool, optional) – Flag if a combined redshift (or not) was added to the zmtl file.

Notes

  • Writes a FITS catalog that incorporates redrock, and a range of afterburner redshifts and confidence values. This will write to the same directory of the redrock and coadd files unless a different output directory is passed.

desispec.zmtl.get_qn_model_fname(qnmodel_fname=None)[source]

Convenience function to grab the $QN_MODEL_FILE environment variable.

Parameters:

qnmodel_fname (str, optional, defaults to $QN_MODEL_FILE) – If qnmodel_fname is passed, it is returned from this function. If it’s not passed, the $QN_MODEL_FILE variable is returned.

Returns:

not passed, the directory stored in the $QN_MODEL_FILE environment variable is returned prepended to the default filename.

Return type:

str

desispec.zmtl.get_sq_model_fname(sqmodel_fname=None)[source]

Convenience function to grab the $SQ_MODEL_FILE environment variable.

Parameters:

sqmodel_fname (str, optional, defaults to $SQ_MODEL_FILE) – If sqmodel_fname is passed, it is returned from this function. If it’s not passed, the $SQ_MODEL_FILE environment variable is returned.

Returns:

If sqmodel_fname is passed, it is returned from this function. If it’s not passed, the directory stored in the $SQ_MODEL_FILE environment variable is returned.

Return type:

str

desispec.zmtl.load_qn_model(model_filename)[source]

Convenience function to load the QuasarNP model and line lists.

Parameters:

model_filename (str) – The filename and path of the QuasarNP model. Either input by user or defaults to get_qn_model_fname().

Returns:

  • array – The QuasarNP model file loaded as an array.

  • array – An array of the emission line names to be used for quasarnp.process_preds().

  • array – An array of the BAL emission line names to be used by quasarnp.process_preds().

desispec.zmtl.load_sq_model(model_filename)[source]

Convenience function for loading the SQUEzE model file.

Parameters:

model_filename (str) – The filename and path of the SQUEzE model file. Either input by user or defaults to get_sq_model_fname().

Returns:

A numpy array of the SQUEzE model.

Return type:

array

Notes

  • The input model file needs to be in the json file format.

desispec.zmtl.make_new_zmtl(redrockname, qn_flag=False, sq_flag=False, abs_flag=False, zcomb_flag=False)[source]

Make the initial zmtl array with redrock data.

Parameters:
  • redrockname (str) – Full filename and path for the redrock file to process.

  • qn_flag (bool' optional) – Flag to add QuasarNP data (or not) to the zmtl file.

  • sq_flag (bool, optional) – Flag to add SQUEzE data (or not) to the zmtl file.

  • abs_flag (bool, optional) – Flag to add MgII Absorption data (or not) to the zmtl file.

  • zcomb_flag (bool, optional) – Flag if a combined redshift (or not) was added to the zmtl file.

Returns:

A zmtl in the official format (zmtldatamodel) compiled from the tile’, ‘night’, and ‘petal_num’, in `zmtldir. If the redrock file for that petal doesn’t exist, returns False.

Return type:

array or bool

desispec.zmtl.tmark(istring)[source]

A function to mark the time an operation starts or ends.

Parameters:

istring (:class:'str') – The input string to print to the terminal.

Notes

  • A string with the date and time in ISO 8061 standard followed by the ‘istring’.

desispec.zmtl.write_zmtl(zmtl, outputname, qn_flag=False, sq_flag=False, abs_flag=False, zcomb_flag=False, qnp_model_file=None, squeze_model_file=None)[source]

Writes the zmtl structured array out as a FITS file.

Parameters:
  • zmtl (array) – The structured array that was created by make_new_zmtl()

  • outputname (str) – The full filepathname of the zmtl output file.

  • qn_flag (bool) – Flag if QuasarNP data (or not) was added to the zmtl file.

  • sq_flag (bool) – Flag if SQUEzE data (or not) was added to the zmtl file.

  • abs_flag (bool) – Flag if MgII Absorption data (or not) was added to the zmtl file.

  • zcomb_flag (bool) – Flag if a combined redshift (or not) was added to the zmtl file.

  • qnp_model_file (str, optional) – File from which the QuasarNP model was loaded. Written to the output header.

  • squeze_model_file (str, optional) – File from which the SQUEzE model was loaded. Written to the output header.

Returns:

The filename, with path, of the FITS file written out.

Return type:

str

desispec.zmtl.zcomb_selector(zmtl, proc_flag=False)[source]

Compare results from redrock, QuasarNP, SQUEzE, and MgII data.

Parameters:
  • zmtl (array) – The structured array that was created by make_new_zmtl()

  • proc_flag (bool) – Turn on extra comparison procedure.

Returns:

The zmtl array with SQUEzE data included in the columns:

  • Z_COMB - The best models-combined redshift for each object.

  • Z_COMB_PROB - The combined probability value of that redshift.

Return type:

array