Full desispec API Reference

desispec

Tools for DESI spectroscopic processing.

desispec.bootcalib

Utility functions to perform a quick calibration of DESI data

TODO: 1. Expand to r, i cameras 2. QA plots 3. Test with CR data

desispec.bootcalib.bootcalib(deg, flatimage, arcimage)

Parameters:

• deg – Legendre polynomial degree to use to fit
• flatimage – desispec.image.Image object of flatfield
• arcimage – desispec.image.Image object of arc

Mostly inherited from desispec/bin/desi_bootcalib directly as needed

Returns:xfit, fdicts, gauss, all_wave_soln

TODO: document what those return objects are

desispec.bootcalib.extract_sngfibers_gaussianpsf(img, img_ivar, xtrc, sigma, box_radius=2, verbose=True)

Extract spectrum for fibers one-by-one using a Gaussian PSF

Parameters:

• img (ndarray) – Image
• img_ivar (ndarray) – Image inverse variance
• xtrc (ndarray) – fiber trace
• sigma (float) – Gaussian sigma for PSF
• box_radius (int, optional) – Radius for extraction (+/-)

Returns:

spec – Extracted spectrum

Return type:

ndarray

desispec.bootcalib.fiber_gauss_new(flat, xtrc, xerr, box_radius=2, max_iter=5, debug=False, verbose=False)

Find the PSF sigma for each fiber This serves as an initial guess to what follows

Parameters:

• flat (ndarray of fiber flat image) –
• xtrc (ndarray of fiber traces) –
• xerr (ndarray of error in fiber traces) –
• box_radius (int, optinal) – Radius of boxcar extraction in pixels
• max_iter (int, optional) – Maximum number of iterations for rejection

Returns:

list of Gaussian sigma

Return type:

gauss

desispec.bootcalib.fiber_gauss_old(flat, xtrc, xerr, box_radius=2, max_iter=5, debug=False, verbose=False)

Find the PSF sigma for each fiber This serves as an initial guess to what follows

Parameters:

• flat (ndarray of fiber flat image) –
• xtrc (ndarray of fiber traces) –
• xerr (ndarray of error in fiber traces) –
• box_radius (int, optinal) – Radius of boxcar extraction in pixels
• max_iter (int, optional) – Maximum number of iterations for rejection

Returns:

list of Gaussian sigma

Return type:

gauss

desispec.bootcalib.find_arc_lines(spec, rms_thresh=7.0, nwidth=5)

Find and centroid arc lines in an input spectrum

Parameters:

• spec (ndarray) – Arc line spectrum
• rms_thresh (float) – RMS threshold scale
• nwidth (int) – Line width to test over

desispec.bootcalib.find_fiber_peaks(flat, ypos=None, nwidth=5, debug=False, thresh=None)

Find the peaks of the fiber flat spectra Preforms book-keeping error checking

Parameters:

• flat – ndarray of fiber flat image
• ypos – int [optional] Row for finding peaks Default is half-way up the image
• nwidth – int [optional] Width of peak (end-to-end)
• debug – bool, optional

Returns:

xpk, ypos, cut

list of xpk (nearest pixel) at ypos ndarray of cut through the image

desispec.bootcalib.fit_traces(xset, xerr, func='legendre', order=6, sigrej=20.0, RMS_TOLER=0.03, verbose=False)

Fit the traces Default is 6th order Legendre polynomials

Parameters:

• xset (ndarray) – traces
• xerr (ndarray) – Error in the trace values (999.=Bad)
• RMS_TOLER (float, optional [0.02]) – Tolerance on size of RMS in fit

Returns:

• xnew, fits
• xnew (ndarray) – New fit values (without error)
• fits (list) – List of the fit dicts

desispec.bootcalib.fix_ycoeff_outliers(xcoeff, ycoeff, deg=5, tolerance=2)

Fix outliers in coefficients for wavelength solution, assuming a continuous function of CCD coordinates

Parameters:

• ncoeff] (ycoeff[nfiber,) – 2D array of Legendre coefficients for X(wavelength)
• ncoeff] – 2D array of Legendre coefficients for Y(wavelength)

Options:

deg : integer degree of polynomial to fit tolerance : replace fibers with difference of wavelength solution larger than this number of pixels after interpolation

Returns:new_ycoeff[nfiber, ncoeff] with outliers replaced by interpolations

For each coefficient, fit a polynomial vs. fiber number with one pass of sigma clipping. Remaining outliers are than replaced with the interpolated fit value.

desispec.bootcalib.id_arc_lines_using_triplets(id_dict, w, dwdy_prior, d2wdy2_prior=1.5e-05, toler=0.2, ntrack=50, nmax=40)

Match (as best possible), a set of the input list of expected arc lines to the detected list

Parameters:

• id_dict (dictionnary with Pixel locations of detected arc lines in "pixpk" and fluxes in "flux") –
• w (ndarray) – array of expected arc lines to be detected and identified
• dwdy (float) – Average dispersion in the spectrum
• d2wdy2_prior (float) – Prior on second derivative
• toler (float, optional) – Tolerance for matching (20%)
• ntrack (max. number of solutions to be tracked) –

Returns:

id_dict – dict of identified lines

Return type:

dict

desispec.bootcalib.load_arcline_list(camera, vacuum=True, lamps=None)

Loads arc line list from NIST files Parses and rejects

Taken from PYPIT

Parameters:

• lines (list) – List of ions to load
• vacuum (bool, optional) – Use vacuum wavelengths
• lamps (optional numpy array of ions, ex np.array(["HgI","CdI","ArI","NeI"])) –

Returns:

alist – Table of arc lines

Return type:

Table

desispec.bootcalib.load_gdarc_lines(camera, llist, vacuum=True, lamps=None, good_lines_filename=None)

Loads a select set of arc lines for initial calibrating

Parameters:

• camera (str) – Camera (‘b’, ‘g’, ‘r’)
• llist (table of lines to use, with columns Ion, wave) –
• vacuum (bool, optional) – Use vacuum wavelengths
• lamps (optional numpy array of ions, ex np.array(["HgI","CdI","ArI","NeI"])) –

Returns:

• dlamb (float) – Dispersion for input camera
• wmark (float) – wavelength to key off of [???]
• gd_lines (ndarray) – Array of lines expected to be recorded and good for ID
• line_guess (int or None) – Guess at the line index corresponding to wmark (default is to guess the 1/2 way point)

desispec.bootcalib.load_parse_dict()

Dicts for parsing Arc line lists from NIST

Rejected lines are in the rejected_lines.yaml file

desispec.bootcalib.parse_nist(ion, vacuum=True)

Parse a NIST ASCII table.

Note that the long —- should have been commented out and also the few lines at the start.

Taken from PYPIT

Parameters:

• ion (str) – Name of ion
• vaccuum (bool, optional) – Use vacuum wavelengths

desispec.bootcalib.parse_nist_tbl(tbl, parse_dict)

Parses a NIST table using various criteria

Parameters:

• tbl (Table) – Read previously from NIST ASCII file
• parse_dict (dict) – Dict of parsing criteria. Read from load_parse_dict

Returns:

tbl – Rows meeting the criteria

Return type:

Table

desispec.bootcalib.qa_arc_spec(all_spec, all_soln, pp, figsz=None)

Generate QA plots of the arc spectra with IDs

Parameters:

• all_spec – Arc 1D fiber spectra
• all_soln – Wavelength solutions
• pp – PDF file pointer
• figsz – figure size, optional

desispec.bootcalib.qa_fiber_Dx(xfit, fdicts, pp=None, figsz=None)

Show the spread in the trace per fiber

Used to diagnose the traces

Parameters:

• xfit – traces
• fdicts – dict of the traces
• pp – PDF file pointer
• figsz – figure size, optional

desispec.bootcalib.qa_fiber_arcrms(all_soln, pp, figsz=None)

Show the RMS of the wavelength solutions vs. fiber

Parameters:

• all_soln – Wavelength solutions
• pp – PDF file pointer
• figsz – figure size, optional

desispec.bootcalib.qa_fiber_dlamb(all_spec, all_soln, pp, figsz=None)

Show the Dlamb of the wavelength solutions vs. fiber

Parameters:

• all_soln – Wavelength solutions
• pp – PDF file pointer
• figsz – figure size, optional

desispec.bootcalib.qa_fiber_gauss(gauss, pp=None, figsz=None)

Show the Gaussian (sigma) fits to each fiber

Parameters:

• gauss – Gaussian of each fiber
• pp – PDF file pointer
• figsz – figure size, optional

desispec.bootcalib.qa_fiber_peaks(xpk, cut, pp=None, figsz=None, nper=100)

Generate a QA plot for the fiber peaks

Parameters:

• xpk – x positions on the CCD of the fiber peaks at a ypos
• cut – Spatial cut through the detector
• pp – PDF file pointer
• figsz – figure size, optional
• nper – number of fibers per row in the plot, optional

desispec.bootcalib.qa_fiber_trace(flat, xtrc, outfil=None, Nfiber=25, isclmin=0.5)

Generate a QA plot for the fiber traces

Parameters:

• flat (ndarray) – image
• xtrc (ndarray) – Trace array
• isclmin (float, optional [0.5]) – Fraction of 90 percentile flux to scale image by
• outfil (str, optional) – Output file
• normalize (bool, optional) – Normalize the flat? If not, use zscale for output

desispec.bootcalib.reject_lines(tbl, rej_dict, rej_tol=0.1)

Rejects lines from a NIST table

Taken from PYPIT

Parameters:

• tbl (Table) – Read previously from NIST ASCII file
• rej_dict (dict) – Dict of rejected lines
• rej_tol (float, optional) – Tolerance for matching a line to reject to linelist (Angstroms)

Returns:

tbl – Rows not rejected

Return type:

Table

desispec.bootcalib.script_bootcalib(arc_idx, flat_idx, cameras=None, channels=None, nproc=10)

Runs desi_bootcalib on a series of preproc files

Returns:script_bootcalib([0,1,2,3,4,5,6,7,8,9], [10,11,12,13,14])

desispec.bootcalib.trace_crude_init(image, xinit0, ypass, invvar=None, radius=2.0, maxshift0=0.5, maxshift=0.15, maxerr=0.2)

Python port of trace_crude_idl.pro from IDLUTILS

Modified for initial guess

Parameters:

• image (2D ndarray) – Image for tracing
• xinit (ndarray) – Initial guesses for trace peak at ypass
• ypass (int) – Row for initial guesses

Returns:

• xset (Trace for each fiber)
• xerr (Estimated error in that trace)

desispec.bootcalib.trace_fweight(fimage, xinit, ycen=None, invvar=None, radius=2.0, debug=False)

Python port of trace_fweight.pro from IDLUTILS

Parameters:

• fimage (2D ndarray) – Image for tracing
• xinit (ndarray) – Initial guesses for x-trace
• invvar (ndarray, optional) – Inverse variance array for the image
• radius (float, optional) – Radius for centroiding; default to 3.0

desispec.bootcalib.write_psf(outfile, xfit, fdicts, gauss, wv_solns, legendre_deg=5, without_arc=False, XCOEFF=None, fiberflat_header=None, arc_header=None, fix_ycoeff=True)

Write the output to a Base PSF format

Parameters:

• outfile (str) – Output file
• xfit (ndarray) – Traces
• gauss (list) – List of gaussian sigmas
• fdicts (list) – List of trace fits
• wv_solns (list) – List of wavelength calibrations
• ncoeff (int) – Number of Legendre coefficients in fits

desispec.calibfinder

Reading and selecting calibration data from $DESI_SPECTRO_CALIB using content of image headers

desispec.calibfinder._load_smsp()

Loads $DESI_SPECTRO_CALIB/spec/smsp.txt into global _sp2sm and _sm2sp dicts

desispec.calibfinder.badfibers(headers, keys=['BROKENFIBERS', 'BADCOLUMNFIBERS', 'LOWTRANSMISSIONFIBERS'], yaml_file=None)

find list of bad fibers from $DESI_SPECTRO_CALIB using the keywords found in the headers

Parameters:headers – list of fits headers, or list of dictionnaries

Optional:

keys: list of keywords, among [“BROKENFIBERS”,”BADCOLUMNFIBERS”,”LOWTRANSMISSIONFIBERS”]. Default is all of them. yaml_file: path to a specific yaml file. By default, the code will automatically find the yaml file from the environment variable DESI_SPECTRO_CALIB and the CAMERA keyword in the headers

Returns List of bad fibers as a 1D array of intergers

desispec.calibfinder.findcalibfile(headers, key, yaml_file=None)

read and select calibration data file from $DESI_SPECTRO_CALIB using the keywords found in the headers

Parameters:

• headers – list of fits headers, or list of dictionnaries
• key – type of calib file, e.g. ‘PSF’ or ‘FIBERFLAT’

Optional:

yaml_file: path to a specific yaml file. By default, the code will automatically find the yaml file from the environment variable DESI_SPECTRO_CALIB and the CAMERA keyword in the headers

Returns path to calibration file

desispec.calibfinder.parse_date_obs(value)

converts DATE-OBS keywork to int with for instance DATE-OBS=2016-12-21T18:06:21.268371-05:00

desispec.calibfinder.sm2sp(sm, night=None)

Converts spectrograph sm hardware number to sp logical number

Parameters:sm – spectrograph sm number 1-10 or str sm[1-10]

Returns “spP” if input is str “smM”, or int P if input is int M

Note: uses $DESI_SPECTRO_CALIB/spec/smsp.txt

TODO: add support for different mappings based on night

desispec.calibfinder.sp2sm(sp)

Converts spectrograph sp logical number to sm hardware number

Parameters:sp – spectrograph int 0-9 or str sp[0-9]

Returns “smM” if input is str “spP”, or int M if input is int P

Note: uses $DESI_SPECTRO_CALIB/spec/smsp.txt

TODO: add support for different mappings based on night

Coadd spectra

desispec.coaddition.coadd(spectra, cosmics_nsig=0.0, onetile=False)

Coadd spectra for each target and each camera, modifying input spectra obj.

Parameters:spectra – desispec.spectra.Spectra object

Options:

cosmics_nsig: float, nsigma clipping threshold for cosmics rays onetile: bool, if True, inputs are from a single tile

Notes: if onetile is True, additional tile-specific columns

like LOCATION and FIBER are included the FIBERMAP; otherwise these are only in the EXP_FIBERMAP since for the same target they could be different on different tiles.

desispec.coaddition.coadd_cameras(spectra, cosmics_nsig=0.0, onetile=False)

Return coadd across both exposures and cameras

Parameters:spectra – desispec.spectra.Spectra object

Options:

cosmics_nsig: float, nsigma clipping threshold for cosmics rays onetile: bool, if True, inputs are from a single tile

If onetile is True, additional tile-specific columns like LOCATION and FIBER are included the FIBERMAP; otherwise these are only in the EXP_FIBERMAP since for the same target they could be different on different tiles.

Note: unlike coadd, this does not modify the input spectra object

desispec.coaddition.coadd_fibermap(fibermap, onetile=False)

Coadds fibermap

Parameters:fibermap (Table or ndarray) – fibermap of individual exposures

Options:

onetile (bool): this is a coadd of a single tile, not across tiles

Returns: (coadded_fibermap, exp_fibermap) Tables

coadded_fibermap contains the coadded_fibermap for the columns that can be coadded, while exp_fibermap is the subset of columns of the original fibermap that can’t be meaningfully coadded because they are per-exposure quantities like FIBER_X.

If onetile is True, the coadded_fibermap includes additional columns like MEAN_FIBER_X that are meaningful if coadding a single tile, but not if coadding across tiles.

desispec.coaddition.decorrelate_divide_and_conquer(Cinv, Cinvf, wavebin, flux, ivar, rdata)

Decorrelate an inverse covariance using the matrix square root.

Implements the decorrelation part of the spectroperfectionism algorithm described in Bolton & Schlegel 2009 (BS) http://arxiv.org/abs/0911.2689.

with the divide and conquer approach, i.e. per diagonal block of the matrix, with an overlapping ‘skin’ from one block to another.

Parameters:

• Cinv – Square 2D array: input inverse covariance matrix
• Cinvf – 1D array: input
• wavebin – minimal size of wavelength bin in A, used to define the core and skin size
• flux – 1D array: output flux (has to be allocated)
• ivar – 1D array: output flux inverse variance (has to be allocated)
• rdata – 2D array: output resolution matrix per diagonal (has to be allocated)

desispec.coaddition.fast_resample_spectra(spectra, wave)

Fast resampling of spectra file. The output resolution = Id. The neighboring flux bins are correlated.

Parameters:

• spectra – desispec.spectra.Spectra object
• wave – 1D numy array with new wavelenght grid

Returns:

desispec.spectra.Spectra object, resolution data=Id

desispec.coaddition.get_resampling_matrix(global_grid, local_grid, sparse=False)

Build the rectangular matrix that linearly resamples from the global grid to a local grid.

The local grid range must be contained within the global grid range.

Parameters:

• global_grid (numpy.ndarray) – Sorted array of n global grid wavelengths.
• local_grid (numpy.ndarray) – Sorted array of m local grid wavelengths.

Returns:

Array of (m,n) matrix elements that perform the linear resampling.

Return type:

numpy.ndarray

desispec.coaddition.resample_spectra_lin_or_log(spectra, linear_step=0, log10_step=0, fast=False, wave_min=None, wave_max=None, nproc=1)

Resampling of spectra file.

Parameters:

• spectra – desispec.spectra.Spectra object
• linear_step – if not null the ouput wavelenght grid will be linear with this step
• log10_step – if not null the ouput wavelenght grid will be logarthmic with this step

Options:

fast: simple resampling. fast but at the price of correlated output flux bins and no information on resolution wave_min: if set, use this min wavelength wave_max: if set, use this max wavelength

Returns:desispec.spectra.Spectra object

desispec.coaddition.spectroperf_resample_spectra(spectra, wave, nproc=1)

Resampling of spectra file using the spectrophotometic approach

Parameters:

• spectra – desispec.spectra.Spectra object
• wave – 1D numy array with new wavelenght grid

Returns:

desispec.spectra.Spectra object

desispec.cosmics

Utility functions to find cosmic rays

desispec.cosmics._reject_cosmic_rays_ala_sdss_single(pix, ivar, selection, psf_gradients, nsig, cfudge, c2fudge)

Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)

This routine is a single call, similar to IDL routine reject_cr_single Called by reject_cosmic_rays_ala_sdss with an iteration

Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image

A faster version of this routine using numba in implemented in _reject_cosmic_rays_ala_sdss_single_numba

Parameters:

• pix – input desispec.Image.pix (counts in pixels, 2D image)
• ivar – inverse variance of pix
• selection – array of booleans
• psf_gradients – 1D array of size 4, for 4 axes: horizontal,vertical and 2 diagonals
• nsig – number of sigma above background required
• cfudge – number of sigma inconsistent with PSF required
• c2fudge – fudge factor applied to PSF

desispec.cosmics._reject_cosmic_rays_ala_sdss_single_numba(pix, ivar, selection, psf_gradients, nsig, cfudge, c2fudge)

Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)

This routine is a single call, similar to IDL routine reject_cr_single Called by reject_cosmic_rays_ala_sdss with an iteration

Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image

This routine is much faster than _reject_cosmic_rays_ala_sdss_single (if you have numba installed, otherwise it is catastrophically slower)

Parameters:

• pix – input desispec.Image.pix (counts in pixels, 2D image)
• ivar – inverse variance of pix
• selection – array of booleans
• psf_gradients – 1D array of size 4, for 4 axes: horizontal,vertical and 2 diagonals
• nsig – number of sigma above background required
• cfudge – number of sigma inconsistent with PSF required
• c2fudge – fudge factor applied to PSF

desispec.cosmics.reject_cosmic_rays(img, nsig=5.0, cfudge=3.0, c2fudge=0.9, niter=100, dilate=True)

Cosmic ray rejection Input is a pre-processed image : desispec.Image The image mask is modified

Parameters:img – input desispec.Image

desispec.cosmics.reject_cosmic_rays_1d(frame, nsig=3, psferr=0.05)

Use resolution matrix in frame to detect spikes in the spectra that are narrower than the PSF, and mask them

desispec.cosmics.reject_cosmic_rays_ala_sdss(img, nsig=6.0, cfudge=3.0, c2fudge=0.5, niter=6, dilate=True)

Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)

This routine is calling several times reject_cosmic_rays_ala_sdss_single, similar to IDL routine reject_cr. There is an optionnal dilatation of the mask by one pixel, as done in sdssproc.pro for SDSS

Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image

Parameters:

• img – input desispec.Image
• nsig – number of sigma above background required
• cfudge – number of sigma inconsistent with PSF required
• c2fudge – fudge factor applied to PSF
• niter – number of iterations on neighboring pixels of rejected pixels
• dilate – force +1 pixel dilation of rejection mask

desispec.database

Tools for loading DESI data into databases.

At the present time, this covers:

1. Raw metadata database.
2. Redshift results database (a.k.a “SpectraDB”).

It does not include:

1. Pipeline processing status database.
2. Imaging and targeting databases.

desispec.database.metadata

Code for interacting with the file metadatabase.

class desispec.database.metadata.Brick(**kwargs)

Representation of a region of the sky.

class desispec.database.metadata.BrickStatus(**kwargs)

Representation of the status of a particular Brick.

class desispec.database.metadata.ExposureFlavor(**kwargs)

List of exposure flavors. Used to constrain the possible values.

class desispec.database.metadata.Frame(**kwargs)

Representation of a particular pointing, exposure, spectrograph and band (a.k.a. ‘channel’ or ‘arm’).

class desispec.database.metadata.FrameStatus(**kwargs)

Representation of the status of a particular Frame.

class desispec.database.metadata.Night(**kwargs)

List of observation nights. Used to constrain the possible values.

class desispec.database.metadata.Status(**kwargs)

List of possible processing statuses.

class desispec.database.metadata.Tile(*args, **kwargs)

Representation of a particular pointing of the telescope.

_coarse_overlapping_bricks(session)

Get the bricks that may overlap a tile.

Parameters:session (sqlalchemy.orm.session.Session) – Database connection. Returns:A list of Brick objects. Return type:list

_constants()

Define mathematical constants associated with a tile.

area

Area of the tile in steradians.

brick_offset(brick)

Offset a brick in the same way as a tile.

Parameters:brick (Brick) – A brick. Returns:A tuple containing the shifted ra1 and ra2. Return type:tuple()

circum_square

Defines a square-like region on the sphere which circumscribes the tile.

cos_radius

Cosine of the radius, precomputed for speed.

offset(shift=10.0)

Provide an offset to move RA away from wrap-around.

Parameters:shift (float, optional) – Amount to offset in degrees. Returns:An amount to offset in degrees. Return type:float

overlapping_bricks(session, map_petals=False)

Perform a geometric calculation to find bricks that overlap a tile.

Parameters:

• session (sqlalchemy.orm.session.Session) – Database connection.
• map_petals (bool, optional) – If True a map of petal number to a list of overlapping bricks is returned.

Returns:

If map_petals is False, a list of Polygon objects. Otherwise, a dict mapping petal number to the Brick objects that overlap that petal.

Return type:

list

petals(Npetals=10)

Convert a tile into a set of Wedge objects.

Parameters:Npetals (int, optional) – Number of petals (default 10). Returns:A list of Wedge objects. Return type:list

radius

Radius of tile in degrees.

simulate_frame(session, band, spectrograph, flavor='science', exptime=1000.0)

Simulate a DESI frame given a Tile object.

Parameters:

• session (sqlalchemy.orm.session.Session) – Database connection.
• band (str) – ‘b’, ‘r’, ‘z’
• spectrograph (int) – Spectrograph number [0-9].
• flavor (str, optional) – Exposure flavor (default ‘science’).
• exptime (float, optional) – Exposure time in seconds (default 1000).

Returns:

A tuple containing a Frame object ready for loading, and a list of bricks that overlap.

Return type:

tuple

desispec.database.metadata.get_all_tiles(session, obs_pass=0, limit=0)

Get all tiles from the database.

Parameters:

• session (sqlalchemy.orm.session.Session) – Database connection.
• obs_pass (int, optional) – Select only tiles from this pass.
• limit (int, optional) – Limit the number of tiles returned

Returns:

A list of Tiles.

Return type:

list

desispec.database.metadata.load_data(session, datapath)

Load a night or multiple nights into the frame table.

Parameters:

• session (sqlalchemy.orm.session.Session) – Database connection.
• datapath (str) – Name of a data directory.

Returns:

A list of the exposure numbers found.

Return type:

list

desispec.database.metadata.load_simulated_data(session, obs_pass=0)

Load simulated frame and brick data.

Parameters:

• session (sqlalchemy.orm.session.Session) – Database connection.
• obs_pass (int, optional) – If set, only simulate one pass.

desispec.database.metadata.main()

Entry point for command-line script.

Returns:An integer suitable for passing to sys.exit(). Return type:int

desispec.database.redshift

Code for loading spectroscopic pipeline results (specifically redshifts) into a database.

class desispec.database.redshift.FiberAssign(**kwargs)

Representation of the fiberassign table.

class desispec.database.redshift.ObsList(**kwargs)

Representation of the obslist table.

class desispec.database.redshift.SchemaMixin

Mixin class to allow schema name to be changed at runtime. Also automatically sets the table name.

class desispec.database.redshift.Target(**kwargs)

Representation of the target table.

class desispec.database.redshift.Truth(**kwargs)

Representation of the truth table.

class desispec.database.redshift.ZCat(**kwargs)

Representation of the zcat table.

desispec.database.redshift.get_options(*args)

Parse command-line options.

Parameters:args (iterable) – If arguments are passed, use them instead of sys.argv. Returns:The parsed options. Return type:argparse.Namespace

desispec.database.redshift.load_fiberassign(datapath, maxpass=4, hdu='FIBERASSIGN', q3c=False, latest_epoch=False, last_column='NUMOBS_MORE')

Load fiber assignment files into the fiberassign table.

Tile files can appear in multiple epochs, so for a given tileid, load the tile file with the largest value of epoch. In the “real world”, a tile file appears in each epoch until it is observed, therefore the tile file corresponding to the actual observation is the one with the largest epoch.

Parameters:

• datapath (str) – Full path to the directory containing tile files.
• maxpass (int, optional) – Search for pass numbers up to this value (default 4).
• hdu (int or str, optional) – Read a data table from this HDU (default ‘FIBERASSIGN’).
• q3c (bool, optional) – If set, create q3c index on the table.
• latest_epoch (bool, optional) – If set, search for the latest tile file among several epochs.
• last_column (str, optional) – Do not load columns past this name (default ‘NUMOBS_MORE’).

desispec.database.redshift.load_file(filepath, tcls, hdu=1, expand=None, convert=None, index=None, rowfilter=None, q3c=False, chunksize=50000, maxrows=0)

Load a data file into the database, assuming that column names map to database column names with no surprises.

Parameters:

• filepath (str) – Full path to the data file.
• tcls (sqlalchemy.ext.declarative.api.DeclarativeMeta) – The table to load, represented by its class.
• hdu (int or str, optional) – Read a data table from this HDU (default 1).
• expand (dict, optional) – If set, map FITS column names to one or more alternative column names.
• convert (dict, optional) – If set, convert the data for a named (database) column using the supplied function.
• index (str, optional) – If set, add a column that just counts the number of rows.
• rowfilter (callable, optional) – If set, apply this filter to the rows to be loaded. The function should return bool, with True meaning a good row.
• q3c (bool, optional) – If set, create q3c index on the table.
• chunksize (int, optional) – If set, load database chunksize rows at a time (default 50000).
• maxrows (int, optional) – If set, stop loading after maxrows are loaded. Alteratively, set maxrows to zero (0) to load all rows.

desispec.database.redshift.load_redrock(datapath=None, hdu='REDSHIFTS', q3c=False)

Load redrock files into the zcat table.

This function is deprecated since there should now be a single redshift catalog file.

Parameters:

• datapath (str) – Full path to the directory containing redrock files.
• hdu (int or str, optional) – Read a data table from this HDU (default ‘REDSHIFTS’).
• q3c (bool, optional) – If set, create q3c index on the table.

desispec.database.redshift.main()

Entry point for command-line script.

Returns:An integer suitable for passing to sys.exit(). Return type:int

desispec.database.redshift.q3c_index(table, ra='ra')

Create a q3c index on a table.

Parameters:

• table (str) – Name of the table to index.
• ra (str, optional) – If the RA, Dec columns are called something besides “ra” and “dec”, set its name. For example, ra='target_ra'.

desispec.database.redshift.setup_db(options=None, **kwargs)

Initialize the database connection.

Parameters:

• options (argpare.Namespace) – Parsed command-line options.
• kwargs (keywords) – If present, use these instead of options. This is more user-friendly than setting up a Namespace object in, e.g. a Jupyter Notebook.

Returns:

True if the configured database is a PostgreSQL database.

Return type:

bool

desispec.database.redshift.update_truth(filepath, hdu=2, chunksize=50000, skip=('SLOPES', 'EMLINES'))

Add data from columns in other HDUs of the Truth table.

Parameters:

• filepath (str) – Full path to the data file.
• hdu (int or str, optional) – Read a data table from this HDU (default 2).
• chunksize (int, optional) – If set, update database chunksize rows at a time (default 50000).
• skip (tuple(), optional) – Do not load columns with these names (default, ('SLOPES', 'EMLINES'))

desispec.database.util

Classes and functions for use by all database code.

desispec.database.util.convert_dateobs(timestamp, tzinfo=None)

Convert a string timestamp into a datetime.datetime object.

Parameters:

• timestamp (str) – Timestamp in string format.
• tzinfo (datetime.tzinfo, optional) – If set, add time zone to the timestamp.

Returns:

The converted timestamp.

Return type:

datetime.datetime

desispec.database.util.parse_pgpass(hostname='nerscdb03.nersc.gov', username='desidev_admin')

Read a ~/.pgpass file.

Parameters:

• hostname (str, optional) – Database hostname.
• username (str, optional) – Database username.

Returns:

A string suitable for creating a SQLAlchemy database engine, or None if no matching data was found.

Return type:

str

desispec.fiberflat

Utility functions to compute a fiber flat correction and apply it We try to keep all the (fits) io separated.

desispec.fiberflat.apply_fiberflat(frame, fiberflat)

Apply fiberflat to frame. Modifies frame.flux and frame.ivar.

Checks whether an heliocentric correction has been applied to the frame wavelength in which case also apply it to the flat field array.

Parameters:

• frame – desispec.Frame object
• fiberflat – desispec.FiberFlat object

The frame is divided by the fiberflat, except where the fiberflat=0.

frame.mask gets bit specmask.BADFIBERFLAT set where

• fiberflat.fiberflat == 0
• fiberflat.ivar == 0
• fiberflat.mask != 0

desispec.fiberflat.autocalib_fiberflat(fiberflats)

Combine fiberflats of all spectrographs from different lamps to maximize uniformity :param fiberflats: list of desispec.FiberFlat object

returns a dictionary of desispec.FiberFlat objects , one per spectrograph

desispec.fiberflat.average_fiberflat(fiberflats)

Average several fiberflats :param fiberflats: list of desispec.FiberFlat object

returns a desispec.FiberFlat object

desispec.fiberflat.compute_fiberflat(frame, nsig_clipping=10.0, accuracy=0.0005, minval=0.1, maxval=10.0, max_iterations=15, smoothing_res=5.0, max_bad=100, max_rej_it=5, min_sn=0, diag_epsilon=0.001)

Compute fiber flat by deriving an average spectrum and dividing all fiber data by this average. Input data are expected to be on the same wavelength grid, with uncorrelated noise. They however do not have exactly the same resolution.

Parameters:

• frame (desispec.Frame) – input Frame object with attributes wave, flux, ivar, resolution_data
• nsig_clipping – [optional] sigma clipping value for outlier rejection
• accuracy – [optional] accuracy of fiberflat (end test for the iterative loop)
• minval – [optional] mask pixels with flux < minval * median fiberflat.
• maxval – [optional] mask pixels with flux > maxval * median fiberflat.
• max_iterations – [optional] maximum number of iterations
• smoothing_res – [optional] spacing between spline fit nodes for smoothing the fiberflat
• max_bad – [optional] mask entire fiber if more than max_bad-1 initially unmasked pixels are masked during the iterations
• max_rej_it – [optional] reject at most the max_rej_it worst pixels in each iteration
• min_sn – [optional] mask portions with signal to noise less than min_sn
• diag_epsilon – [optional] size of the regularization term in the deconvolution

Returns:

desispec.FiberFlat object with attributes

wave, fiberflat, ivar, mask, meanspec

Notes: - we first iteratively :

• compute a deconvolved mean spectrum
• compute a fiber flat using the resolution convolved mean spectrum for each fiber
• smooth the fiber flat along wavelength
• clip outliers

• then we compute a fiberflat at the native fiber resolution (not smoothed)
• the routine returns the fiberflat, its inverse variance , mask, and the deconvolved mean spectrum
• the fiberflat is the ratio data/mean , so this flat should be divided to the data

NOTE THAT THIS CODE HAS NOT BEEN TESTED WITH ACTUAL FIBER TRANSMISSION VARIATIONS, OUTLIER PIXELS, DEAD COLUMNS …

desispec.fiberflat.qa_fiberflat(param, frame, fiberflat)

Calculate QA on FiberFlat object

Parameters:

• param – dict of QA parameters
• frame – Frame
• fiberflat – FiberFlat

Returns:

dict of QA outputs

Need to record simple Python objects for yaml (str, float, int)

Return type:

qadict

desispec.fluxcalibration

Flux calibration routines.

desispec.fluxcalibration.ZP_from_calib(exptime, wave, calib)

Calculate the ZP in AB magnitudes given the calibration and the wavelength arrays :param exptime: float; exposure time in seconds :param wave: 1D array (A) :param calib: 1D array (converts erg/s/A to photons/s/A)

Returns:1D array of ZP values in AB magnitudes Return type:ZP_AB

desispec.fluxcalibration._compute_coef(coord, node_coords)

Function used by interpolate_on_parameter_grid2

Parameters:

• coord – 1D array of coordinates of size n_axis
• node_coords – 2D array of coordinates of nodes, shape = (n_nodes,n_axis)

Returns:

1D array of linear coefficients for each node, size = n_nodes

Return type:

coef

desispec.fluxcalibration._func(arg)

Used for multiprocessing.Pool

desispec.fluxcalibration._func2(arg)

Used for multiprocessing.Pool

desispec.fluxcalibration._smooth_template(template_id, camera_index, template_flux)

Used for multiprocessing.Pool

desispec.fluxcalibration.applySmoothingFilter(flux, width=200)

Return a smoothed version of the input flux array using a median filter

Parameters:

• flux – 1D array of flux
• width – size of the median filter box

Returns:

median filtered flux of same size as input

Return type:

smooth_flux

desispec.fluxcalibration.apply_flux_calibration(frame, fluxcalib)

Applies flux calibration to input flux and ivar

Parameters:

• frame – Spectra object with attributes wave, flux, ivar, resolution_data
• fluxcalib – FluxCalib object with wave, calib, …

Modifies frame.flux and frame.ivar

desispec.fluxcalibration.compute_flux_calibration(frame, input_model_wave, input_model_flux, input_model_fibers, nsig_clipping=10.0, deg=2, debug=False, highest_throughput_nstars=0, exposure_seeing_fwhm=1.1, stdcheck=True, nsig_flux_scale=3)

Compute average frame throughput based on data frame.(wave,flux,ivar,resolution_data) and spectro-photometrically calibrated stellar models (model_wave,model_flux). Wave and model_wave are not necessarily on the same grid

Parameters:

• frame – Frame object with attributes wave, flux, ivar, resolution_data
• input_model_wave – 1D[nwave] array of model wavelengths
• input_model_flux – 2D[nstd, nwave] array of model fluxes
• input_model_fibers – 1D[nstd] array of model fibers
• nsig_clipping – (optional) sigma clipping level
• exposure_seeing_fwhm – (optional) seeing FWHM in arcsec of the exposure
• stdcheck – check if the model stars are actually standards according to the fibermap and only rely on those
• nsig_flux_scale – n sigma cutoff on the flux scale among standard stars

Returns:

desispec.FluxCalib object calibration: mean calibration (without resolution)

Notes

• we first resample the model on the input flux wave grid
• then convolve it to the data resolution (the input wave grid is supposed finer than the spectral resolution)
• then iteratively - fit the mean throughput (deconvolved, this is needed because of sharp atmospheric absorption lines) - compute a scale factor to fibers (to correct for small mis-alignement for instance) - perform outlier rejection

There is one subtelty with the relation between calibration and resolution.

• The input frame flux is on average flux^frame_fiber = R_fiber*C*flux^true where C is the true calibration (or throughput) which is a function of wavelength. This is the system we solve.
• But we want to return a calibration vector per fiber C_fiber defined by flux^cframe_fiber = flux^frame_fiber/C_fiber, such that flux^cframe can be compared with a convolved model of the truth, flux^cframe_fiber = R_fiber*flux^true, i.e. (R_fiber*C*flux^true)/C_fiber = R_fiber*true_flux, giving C_fiber = (R_fiber*C*flux^true)/(R_fiber*flux^true)
• There is no solution for this for all possible input specta. The solution for a flat spectrum is returned, which is very close to C_fiber = R_fiber*C (but not exactly).

desispec.fluxcalibration.interpolate_on_parameter_grid(data_wave, data_flux, data_ivar, template_flux, teff, logg, feh, template_chi2)

3D Interpolation routine among templates based on a grid of parameters teff, logg, feh.

The tricky part is to define a cube on the parameter grid populated with templates, and it is not always possible. The routine never extrapolates, so that we stay in the range of input parameters.

Parameters:

• data_wave – 1D[nwave] array of wavelength (concatenated list of input wavelength of different cameras and exposures)
• data_flux – 1D[nwave] array of normalized flux = (input flux)/median_filter(input flux) (concatenated list)
• data_ivar – 1D[nwave] array of inverse variance of normalized flux
• template_flux – 2D[ntemplates,nwave] array of normalized flux of templates (after resample, convolution and division by median_filter)
• teff – 1D[ntemplates]
• logg – 1D[ntemplates]
• feh – 1D[ntemplates]
• template_chi2 – 1D[ntemplatess] array of precomputed chi2 = sum(data_ivar*(data_flux-template_flux)**2)

Returns:

best fit coefficient of linear combination of templates chi2 : chi2 of the linear combination

Return type:

coefficients

desispec.fluxcalibration.isStdStar(fibermap, bright=None)

Determines if target(s) are standard stars

Parameters:fibermap – table including DESI_TARGET or SV1_DESI_TARGET bit mask(s)

Optional:

bright: if True, only bright time standards; if False, only darktime, otherwise both

Returns bool or array of bool

desispec.fluxcalibration.match_templates(wave, flux, ivar, resolution_data, stdwave, stdflux, teff, logg, feh, ncpu=1, z_max=0.005, z_res=2e-05, template_error=0, comm=None)

For each input spectrum, identify which standard star template is the closest match, factoring out broadband throughput/calibration differences.

Parameters:

• wave – A dictionary of 1D array of vacuum wavelengths [Angstroms]. Example below.
• flux – A dictionary of 1D observed flux for the star
• ivar – A dictionary 1D inverse variance of flux
• resolution_data – resolution corresponding to the star’s fiber
• stdwave – 1D standard star template wavelengths [Angstroms]
• stdflux – 2D[nstd, nwave] template flux
• teff – 1D[nstd] effective model temperature
• logg – 1D[nstd] model surface gravity
• feh – 1D[nstd] model metallicity
• ncpu – number of cpu for multiprocessing
• comm – MPI communicator; if given, ncpu will be ignored and only rank 0 will return results that are not None

Returns:

numpy.array of linear coefficient of standard stars redshift : redshift of standard star chipdf : reduced chi2

Return type:

coef

Notes

• wave and stdwave can be on different grids that don’t necessarily overlap
• wave does not have to be uniform or monotonic. Multiple cameras can be supported by concatenating their wave and flux arrays

desispec.fluxcalibration.normalize_templates(stdwave, stdflux, mag, band, photsys)

Returns spectra normalized to input magnitudes.

Parameters:

• stdwave – 1D array of standard star wavelengths [Angstroms]
• stdflux – 1D observed flux
• mag – float desired magnitude
• band – G,R,Z,W1 or W2
• photsys – N or S (for Legacy Survey North or South)

Returns:

same as input normflux : normalized flux array

Return type:

stdwave

Only SDSS_r band is assumed to be used for normalization for now.

desispec.fluxcalibration.qa_fluxcalib(param, frame, fluxcalib)

Parameters:

• param – dict of QA parameters
• frame – Frame
• fluxcalib – FluxCalib

Returns:

dict of QA outputs

Need to record simple Python objects for yaml (str, float, int)

Return type:

qadict

desispec.fluxcalibration.redshift_fit(wave, flux, ivar, resolution_data, stdwave, stdflux, z_max=0.005, z_res=5e-05, template_error=0.0)

Redshift fit of a single template

Parameters:

• wave – A dictionary of 1D array of vacuum wavelengths [Angstroms]. Example below.
• flux – A dictionary of 1D observed flux for the star
• ivar – A dictionary 1D inverse variance of flux
• resolution_data – resolution corresponding to the star’s fiber
• stdwave – 1D standard star template wavelengths [Angstroms]
• stdflux – 1D[nwave] template flux
• z_max – float, maximum blueshift and redshift in scan, has to be positive
• z_res – float, step of of redshift scan between [-z_max,+z_max]
• template_error – float, assumed template flux relative error

Returns:

redshift of standard star

Return type:

redshift

Notes

• wave and stdwave can be on different grids that don’t necessarily overlap
• wave does not have to be uniform or monotonic. Multiple cameras can be supported by concatenating their wave and flux arrays

desispec.fluxcalibration.resample_template(data_wave_per_camera, resolution_data_per_camera, template_wave, template_flux, template_id)

Resample a spectral template on the data wavelength grid. Then convolve the spectra by the resolution for each camera. Also returns the result of applySmoothingFilter. This routine is used internally in a call to multiprocessing.Pool.

Parameters:

• data_wave_per_camera – A dictionary of 1D array of vacuum wavelengths [Angstroms], one entry per camera and exposure.
• resolution_data_per_camera – A dictionary of resolution corresponding for the fiber, one entry per camera and exposure.
• template_wave – 1D array, input spectral template wavelength [Angstroms] (arbitrary spacing).
• template_flux – 1D array, input spectral template flux density.
• template_id – int, template identification index, used to ensure matching of input/output after a multiprocessing run.

Returns:

int, template identification index, same as input. output_wave : A dictionary of 1D array of vacuum wavelengths output_flux : A dictionary of 1D array of output template flux output_norm : A dictionary of 1D array of output template smoothed flux

Return type:

template_id

desispec.frame

Lightweight wrapper class for spectra, to be returned by io.read_frame

Lightweight wrapper class for preprocessed image data

desispec.interpolation

Utility functions for interpolation of spectra over different wavelength grids.

desispec.interpolation._unweighted_resample(output_x, input_x, input_flux_density, extrapolate=False)

Returns a flux conserving resampling of an input flux density. The total integrated flux is conserved.

Parameters:

• output_x – SORTED vector, not necessarily linearly spaced
• input_x – SORTED vector, not necessarily linearly spaced
• input_flux_density – input flux density dflux/dx sampled at x

both must represent the same quantity with the same unit input_flux_density = dflux/dx sampled at input_x

Options:

extrapolate: extrapolate using edge values of input array, default is False,

in which case values outside of input array are set to zero

Returns:returns output_flux

This interpolation conserves flux such that, on average, output_flux_density = input_flux_density

The input flux density outside of the range defined by the edges of the first and last bins is considered null. The bin size of bin ‘i’ is given by (x[i+1]-x[i-1])/2 except for the first and last bin where it is (x[1]-x[0]) and (x[-1]-x[-2]) so flux density is zero for x<x[0]-(x[1]-x[0])/2 and x>x[-1]+(x[-1]-x[-2])/2

The input is interpreted as the nodes positions and node values of a piece-wise linear function:

y(x) = sum_i y_i * f_i(x)

with::

f_i(x) = (x_{i-1}<x<=x_{i})*(x-x_{i-1})/(x_{i}-x_{i-1})

• (x_{i}<x<=x_{i+1})*(x-x_{i+1})/(x_{i}-x_{i+1})

the output value is the average flux density in a bin flux_out(j) = int_{x>(x_{j-1}+x_j)/2}^{x<(x_j+x_{j+1})/2} y(x) dx / 0.5*(x_{j+1}+x_{j-1})

desispec.interpolation.resample_flux(xout, x, flux, ivar=None, extrapolate=False)

Returns a flux conserving resampling of an input flux density. The total integrated flux is conserved.

Parameters:

• xout (-) – output SORTED vector, not necessarily linearly spaced
• x (-) – input SORTED vector, not necessarily linearly spaced
• flux (-) – input flux density dflux/dx sampled at x

both x and xout must represent the same quantity with the same unit

Options:

• ivar: weights for flux; default is unweighted resampling
• extrapolate: extrapolate using edge values of input array, default is False, in which case values outside of input array are set to zero.

Setting both ivar and extrapolate raises a ValueError because one cannot assign an ivar outside of the input data range.

Returns:if ivar is None, returns outflux if ivar is not None, returns outflux, outivar

This interpolation conserves flux such that, on average, output_flux_density = input_flux_density

The input flux density outside of the range defined by the edges of the first and last bins is considered null. The bin size of bin ‘i’ is given by (x[i+1]-x[i-1])/2 except for the first and last bin where it is (x[1]-x[0]) and (x[-1]-x[-2]) so flux density is zero for x<x[0]-(x[1]-x[0])/2 and x>x[-1]-(x[-1]-x[-2])/2

The input is interpreted as the nodes positions and node values of a piece-wise linear function:

y(x) = sum_i y_i * f_i(x)

with:

f_i(x) =    (x_{i-1}<x<=x_{i})*(x-x_{i-1})/(x_{i}-x_{i-1})
          + (x_{i}<x<=x_{i+1})*(x-x_{i+1})/(x_{i}-x_{i+1})

the output value is the average flux density in a bin:

flux_out(j) = int_{x>(x_{j-1}+x_j)/2}^{x<(x_j+x_{j+1})/2} y(x) dx /  0.5*(x_{j+1}+x_{j-1})

desispec.io

Tools for data and metadata I/O.

desispec.io.download

Download files from DESI repository.

desispec.io.download._auth(machine='data.desi.lbl.gov')

Get authentication credentials.

desispec.io.download._map_download(map_tuple)

Wrapper function to pass to multiprocess.Pool.map().

desispec.io.download.download(filenames, single_thread=False, workers=None, baseurl='https://data.desi.lbl.gov/desi')

Download files from the DESI repository.

This function will try to create any directories that don’t already exist, using the exact paths specified in filenames.

Parameters:

• filenames – string or list-like object containing filenames.
• single_thread – (optional) if True, do not use multiprocessing to download files.
• workers – (optional) integer indicating the number of worker processes to create.
• baseurl – (optional) string containing the URL of the top-level DESI directory.

Returns:

Full, local path to the file(s) downloaded.

desispec.io.fiberflat

IO routines for fiberflat.

desispec.io.fiberflat.read_fiberflat(filename)

Read fiberflat from filename

Parameters:filename (str) – Name of fiberflat file, or (night, expid, camera) tuple Returns:

FiberFlat object with attributes

fiberflat, ivar, mask, meanspec, wave, header

Notes

fiberflat, ivar, mask are 2D [nspec, nwave] meanspec and wave are 1D [nwave]

desispec.io.fiberflat.write_fiberflat(outfile, fiberflat, header=None, fibermap=None)

Write fiberflat object to outfile

Parameters:

• outfile – filepath string or (night, expid, camera) tuple
• fiberflat – FiberFlat object

Optional:

header: dict or fits.Header object to use as HDU 0 header fibermap: table to store as FIBERMAP HDU

Returns:filepath of file that was written

desispec.io.fibermap

IO routines for fibermap.

desispec.io.fibermap._set_fibermap_columns()

Prepare survey-specific list of columns.

Returns:As a convenience, return the full set of survey-specific columns. Return type:dict

desispec.io.fibermap.assemble_fibermap(night, expid, badamps=None, badfibers_filename=None, force=False, allow_svn_override=True)

Create a fibermap for a given night and expid.

Parameters:

• night (int) – YEARMMDD night of sunset.
• expid (int) – Exposure ID.
• badamps (str, optional) – Comma separated list of "{camera}{petal}{amp}", i.e. "[brz][0-9][ABCD]". Example: 'b7D,z8A'.
• badfibers_filename (str, optional) – Filename with table of bad fibers with at least two columns: FIBER and FIBERSTATUS
• force (bool, optional) – Create fibermap even if missing coordinates/guide files.
• allow_svn_override (bool, optional) – If True (default), allow fiberassign SVN to override raw data.

Returns:

A representation of a fibermap FITS file.

Return type:

astropy.io.fits.HDUList

desispec.io.fibermap.compare_fiberassign(fa1, fa2, compare_all=False)

Check whether two fiberassign tables agree for cols used by ICS/platemaker

Parameters:fa2 (fa1,) – fiberassign astropy Tables or numpy structured arrays

Options:

compare_all: if True, compare all columns, not just those used by ops

Returns list of columns with mismatches; empty list if all agree

Note: if both are NaN, it is considered a match

desispec.io.fibermap.empty_fibermap(nspec, specmin=0, survey='main')

Return an empty fibermap Table to be filled in.

Parameters:

• nspec (int) – Number of fibers (spectra) to include.
• specmin (int, optional) – Staring spectrum index.
• survey (string, optional) – Define columns for this survey; default ‘main’.

Returns:

An empty Table.

Return type:

Table

desispec.io.fibermap.fibermap_new2old(fibermap)

Converts new format fibermap into old format fibermap

Parameters:fibermap – new-format fibermap table (e.g. with FLUX_G column) Returns:old format fibermap (e.g. with MAG column)

Note: this is a transitional convenience function to allow us to simulate new format fibermaps while still running code that expects the old format. After all code has been converted to use the new format, this will be removed.

desispec.io.fibermap.find_fiberassign_file(night, expid, tileid=None, nightdir=None)

Walk backwards in exposures to find matching fiberassign file

Parameters:

• night (int) – YEARMMDD night of observations
• expid (int) – spectroscopic exposure ID

Options:

tileid (int): tileid to look for nightdir (str): base directory for raw data on that night

Returns first fiberassign file found on or before expid on night.

Raises FileNotFoundError if no fiberassign file is found

desispec.io.fibermap.read_fibermap(filename)

Reads a fibermap file and returns its data as an astropy Table

Parameters:filename – input file name

desispec.io.fibermap.write_fibermap(outfile, fibermap, header=None, clobber=True, extname='FIBERMAP')

Write fibermap binary table to outfile.

Parameters:

• outfile (str) – output filename
• fibermap – astropy Table of fibermap data
• header – header data to include in same HDU as fibermap
• clobber (bool, optional) – overwrite outfile if it exists
• extname (str, optional) – set the extension name.

Returns:

full path to filename of fibermap file written.

Return type:

write_fibermap (str)

desispec.io.filters.load_filter(given_filter)

Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object

Parameters:

• given_filter – given filter for which the qe is to be loaded. Desi templates/
• have them in uppercase, so it should be in upper case like SDSS, DECAM or (files) –
• Speclite has lower case so are mapped here. (WISE.) –

desispec.io.filters.load_gaia_filter(band, dr=2)

Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object

Parameters:

• band – filter pass-band in “G”,”BP”,”RP”
• dr – 2 or 3

desispec.io.filters.load_legacy_survey_filter(band, photsys)

Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object

Parameters:

• band – filter pass-band in “G”,”R”,”Z”,”W1”,”W2”
• photsys – “N” or “S” for North (BASS+MzLS) or South (CTIO/DECam)

desispec.io.fluxcalibration

IO routines for flux calibration.

desispec.io.fluxcalibration.read_average_flux_calibration(filename)

Read average flux calibration file; returns an AverageFluxCalib object

desispec.io.fluxcalibration.read_flux_calibration(filename)

Read flux calibration file; returns a FluxCalib object

desispec.io.fluxcalibration.read_stdstar_models(filename)

Read stdstar models from filename.

Parameters:filename (str) – File containing standard star models. Returns:flux[nspec, nwave], wave[nwave], fibers[nspec] Return type:read_stdstar_models (tuple)

desispec.io.fluxcalibration.read_stdstar_templates(stellarmodelfile)

Reads an input stellar model file

Parameters:stellarmodelfile – input filename

Returns (wave, flux, templateid, teff, logg, feh) tuple:

wave : 1D[nwave] array of wavelengths [Angstroms] flux : 2D[nmodel, nwave] array of model fluxes templateid : 1D[nmodel] array of template IDs for each spectrum teff : 1D[nmodel] array of effective temperature for each model logg : 1D[nmodel] array of surface gravity for each model feh : 1D[nmodel] array of metallicity for each model

desispec.io.fluxcalibration.write_average_flux_calibration(outfile, averagefluxcalib)

Writes average flux calibration.

Parameters:

• outfile – output file name
• averagefluxcalib – AverageFluxCalib object

Options:

header : dict-like object of key/value pairs to include in header

desispec.io.fluxcalibration.write_flux_calibration(outfile, fluxcalib, header=None)

Writes flux calibration.

Parameters:

• outfile – output file name
• fluxcalib – FluxCalib object

Options:

header : dict-like object of key/value pairs to include in header

desispec.io.fluxcalibration.write_stdstar_models(norm_modelfile, normalizedFlux, wave, fibers, data, fibermap, input_frames, header=None)

Writes the normalized flux for the best models.

Parameters:

• norm_modelfile – output file path
• normalizedFlux – 2D array of flux[nstdstars, nwave]
• wave – 1D array of wavelengths[nwave] in Angstroms
• fibers – 1D array of fiberids for these spectra
• data – meta data table about which templates best fit
• fibermap – fibermaps rows for the input standard stars
• input_frames – Table with NIGHT, EXPID, CAMERA of input frames used

desispec.io.frame

I/O routines for Frame objects

desispec.io.frame.read_frame(filename, nspec=None, skip_resolution=False)

Reads a frame fits file and returns its data.

Parameters:

• filename – path to a file, or (night, expid, camera) tuple where night = string YEARMMDD expid = integer exposure ID camera = b0, r1, .. z9
• skip_resolution – bool, option Speed up read time (>5x) by avoiding the Resolution matrix

Returns:

desispec.Frame object with attributes wave, flux, ivar, etc.

desispec.io.frame.read_meta_frame(filename, extname=0)

Load the meta information of a Frame :param filename: path to a file :param extname: int, optional; Extension for grabbing header info

Returns:dict or astropy.fits.header Return type:meta

desispec.io.frame.search_for_framefile(frame_file, specprod_dir=None)

Search for an input frame_file in the desispec redux hierarchy :param frame_file: str :param specprod_dir: str, optional

Returns:str, full path to frame_file if found else raise error Return type:mfile

desispec.io.frame.write_frame(outfile, frame, header=None, fibermap=None, units=None)

Write a frame fits file and returns path to file written.

Parameters:

• outfile – full path to output file, or tuple (night, expid, channel)
• frame – desispec.frame.Frame object with wave, flux, ivar…

Optional:

header: astropy.io.fits.Header or dict to override frame.header fibermap: table to store as FIBERMAP HDU

Returns:full filepath of output file that was written

Note

to create a Frame object to pass into write_frame, frame = Frame(wave, flux, ivar, resolution_data)

desispec.io.image

I/O routines for Image objects

desispec.io.image.read_image(filename)

Returns desispec.image.Image object from input file

desispec.io.image.write_image(outfile, image, meta=None)

Writes image object to outfile

Parameters:

• outfile – output file string
• image – desispec.image.Image object (or any object with 2D array attributes image, ivar, mask)

Optional:

meta : dict-like object with metadata key/values (e.g. FITS header)

desispec.io.meta

IO metadata functions.

desispec.io.meta.faflavor2program(faflavor)

Map FAFLAVOR keywords to what we wish we had set for FAPRGRM

Parameters:faflavor (str or array of str) – FAFLAVOR keywords from fiberassign Returns:what FAPRGM would be if we had set it (dark, bright, backup, other) Return type:faprgm (str or array of str)

Note: this was standardized by sv3 and main, but evolved during sv1 and sv2

desispec.io.meta.find_exposure_night(expid, specprod_dir=None)

Find the night that has the exposure :param expid: int :param specprod_dir: str, optional

Returns:str Return type:night

desispec.io.meta.findfile(filetype, night=None, expid=None, camera=None, tile=None, groupname=None, nside=64, band=None, spectrograph=None, survey=None, faprogram=None, rawdata_dir=None, specprod_dir=None, download=False, outdir=None, qaprod_dir=None)

Returns location where file should be

Parameters:filetype – file type, typically the prefix, e.g. “frame” or “psf”

Args depending upon filetype:

night : YEARMMDD string expid : integer exposure id camera : ‘b0’ ‘r1’ .. ‘z9’ tile : integer tile (pointing) number groupname : spectral grouping name (healpix pixel, tile “cumulative” or “pernight”) nside : healpix nside band : one of ‘b’,’r’,’z’ identifying the camera band spectrograph : integer spectrograph number, 0-9 survey : e.g. sv1, sv3, main, special faprogram : fiberassign program, e.g. dark, bright

Options:

rawdata_dir : overrides $DESI_SPECTRO_DATA specprod_dir : overrides $DESI_SPECTRO_REDUX/$SPECPROD/ qaprod_dir : defaults to $DESI_SPECTRO_REDUX/$SPECPROD/QA/ if not provided download : if not found locally, try to fetch remotely outdir : use this directory for output instead of canonical location

Raises:

• ValueError – for invalid file types, and other invalid input
• KeyError – for missing environment variables

desispec.io.meta.get_exposures(night, raw=False, rawdata_dir=None, specprod_dir=None)

Get a list of available exposures for the specified night.

Exposures are identified as correctly formatted subdirectory names within the night directory, but no checks for valid contents of these exposure subdirectories are performed.

Parameters:

• night (str) – Date string for the requested night in the format YYYYMMDD.
• raw (bool) – Returns raw exposures if set, otherwise returns processed exposures.
• rawdata_dir (str) – [optional] overrides $DESI_SPECTRO_DATA
• specprod_dir (str) – Path containing the exposures/ directory to use. If the value is None, then the value of specprod_root() is used instead. Ignored when raw is True.

Returns:

List of integer exposure numbers available for the specified night. The

list will be empty if no the night directory exists but does not contain any exposures.

Return type:

list

Raises:

• ValueError – Badly formatted night date string
• IOError – non-existent night.

desispec.io.meta.get_files(filetype, night, expid, specprod_dir=None, qaprod_dir=None, **kwargs)

Get files for a specified exposure.

Uses findfile() to determine the valid file names for the specified type. Any camera identifiers not matching the regular expression [brz][0-9] will be silently ignored.

Parameters:

• filetype (str) – Type of files to get. Valid choices are ‘frame’, ‘cframe’, ‘psf’, etc.
• night (str) – Date string for the requested night in the format YYYYMMDD.
• expid (int) – Exposure number to get files for.
• specprod_dir (str) – Path containing the exposures/ directory to use. If the value is None, then the value of specprod_root() is used instead. Ignored when raw is True.

Returns:

Dictionary of found file names using camera id strings as keys,

which are guaranteed to match the regular expression [brz][0-9].

Return type:

dict

desispec.io.meta.get_nights(strip_path=True, specprod_dir=None, sub_folder='exposures')

Generate a list of nights in a given folder (default is exposures/) Demands an 8 digit name beginning with 20

Parameters:

• strip_path – bool, optional; Strip the path to the nights folders
• rawdata_dir –
• specprod_dir –
• sub_root – str, optional; ‘exposures’, ‘calib2d’

Returns:

list of nights (without or with paths)

Return type:

nights

desispec.io.meta.get_pipe_database()

Get the production database location based on the environment.

desispec.io.meta.get_pipe_logdir()

Return the name of the subdirectory containing pipeline logs.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_pipe_nightdir()

Return the name of the subdirectory containing per-night files.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_pipe_pixeldir()

Return the name of the subdirectory containing per-pixel files.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_pipe_rundir(specprod_dir=None)

Return the directory path for pipeline runtime files.

Parameters:specprod_dir (str) – Optional path to production directory. If None, the this is obtained from specprod_root().

Returns (str):

the directory path for pipeline runtime files.

desispec.io.meta.get_pipe_scriptdir()

Return the name of the subdirectory containing pipeline scripts.

Returns (str):

The name of the subdirectory.

desispec.io.meta.get_raw_files(filetype, night, expid, rawdata_dir=None)

Get files for a specified exposure.

Uses findfile() to determine the valid file names for the specified type. Any camera identifiers not matching the regular expression [brz][0-9] will be silently ignored.

Parameters:

• filetype (str) – Type of files to get. Valid choices are ‘raw’, ‘preproc’, ‘fibermap’.
• night (str) – Date string for the requested night in the format YYYYMMDD.
• expid (int) – Exposure number to get files for.
• rawdata_dir (str) – [optional] overrides $DESI_SPECTRO_DATA

Returns:

Dictionary of found file names using camera id strings as keys,

which are guaranteed to match the regular expression [brz][0-9].

Return type:

dict

desispec.io.meta.get_reduced_frames(channels=['b', 'r', 'z'], nights=None, ftype='cframe', **kwargs)

Loops through a production to find all reduced frames (default is cframes) One can choose a subset of reduced frames by argument :param channels: list, optional :param nights: list, optional :param ftype: str, optional :param kwargs: passed to get_files()

Returns:list for frame filenames Return type:all_frames

desispec.io.meta.qaprod_root(specprod_dir=None)

Return directory root for spectro production QA, i.e. $DESI_SPECTRO_REDUX/$SPECPROD/QA.

Raises:KeyError – if these environment variables aren’t set.

desispec.io.meta.rawdata_root()

Returns directory root for raw data, i.e. $DESI_SPECTRO_DATA

Raises:KeyError – if these environment variables aren’t set.

desispec.io.meta.shorten_filename(filename)

Attempt to shorten filename to fit in FITS header without CONTINUE

Parameters:filename (str) – input filename

Returns potentially shortened filename

Replaces prefixes from environment variables:

• $DESI_SPECTRO_CALIB -> SPCALIB
• $DESI_SPECTRO_REDUX/$SPECPROD -> SPECPROD

desispec.io.meta.specprod_root(specprod=None)

Return directory root for spectro production, i.e. $DESI_SPECTRO_REDUX/$SPECPROD.

Options:

specprod (str): overrides $SPECPROD

Raises:KeyError – if these environment variables aren’t set.

desispec.io.meta.validate_night(night)

Validates a night string and converts to a date.

Parameters:night (str) – Date string for the requested night in the format YYYYMMDD. Returns:Date object representing this night. Return type:datetime.date Raises:ValueError – Badly formatted night string.

desispec.io.params

IO routines for parameter values

desispec.io.params.read_params(filename=None, reload=False)

Read parameter data from file

desispec.io.qa

IO routines for QA

desispec.io.qa.load_qa_brick(filename)

Load an existing QA_Brick or generate one, as needed :param filename: str

Returns: qa_brick: QA_Brick object

desispec.io.qa.load_qa_frame(filename, frame_meta=None, flavor=None)

Load an existing QA_Frame or generate one, as needed

Parameters:

• filename – str
• frame_meta – dict like, optional
• flavor – str, optional Type of QA_Frame

Returns:

QA_Frame object

Return type:

qa_frame

desispec.io.qa.load_qa_multiexp(inroot)

Load QA for a given production

Parameters:inroot – str base filename without format extension Returns:dict Return type:odict

desispec.io.qa.qafile_from_framefile(frame_file, qaprod_dir=None, output_dir=None)

Derive the QA filename from an input frame file :param frame_file: str :param output_dir: str, optional Over-ride default output path :param qa_dir: str, optional Over-ride default QA

Returns:

desispec.io.qa.read_qa_brick(filename)

Generate a QA_Brick object from a data file

desispec.io.qa.read_qa_data(filename)

Read data from a QA file

desispec.io.qa.read_qa_frame(filename)

Generate a QA_Frame object from a data file

desispec.io.qa.write_qa_brick(outfile, qabrick)

Write QA for a given exposure

Parameters:

• outfile – filename
• qabrick – QA_Brick object _data: dict of QA info

desispec.io.qa.write_qa_exposure(outroot, qaexp, ret_dict=False)

Write QA for a given exposure

Parameters:

• outroot – str filename without format extension
• qa_exp – QA_Exposure object
• ret_dict – bool, optional Return dict only? [for qa_prod, mainly]

Returns:

str or dict

Return type:

outfile or odict

desispec.io.qa.write_qa_frame(outfile, qaframe, verbose=False)

Write QA for a given frame

Parameters:

• outfile – str filename
• qa_exp – QA_Frame object, with the following attributes qa_data: dict of QA info

desispec.io.qa.write_qa_multiexp(outroot, mdict, indent=True)

Write QA for a given production

Parameters:

• outroot – str filename without format extension
• mdict – dict

Returns:

str

output filename

Return type:

outfile

desispec.io.qa.write_qa_ql(outfile, qaresult)

Write QL output files

Parameters:

• outfile – str filename to be written (yaml)
• qaresult – dict QAresults from run_qa()

Returns:

str

Return type:

outfile

I/O for DESI raw data files

See DESI-1229 for format details TODO: move into datamodel after we have verified the format

desispec.io.raw.read_raw(filename, camera, fibermapfile=None, fill_header=None, **kwargs)

Returns preprocessed raw data from camera extension of filename.

Parameters:

• filename (str) – Input FITS filename with DESI raw data.
• camera (str) – Camera name (B0, R1, … Z9) or FITS extension name.
• fibermapfile (str, optional) – Read fibermap from this file; if None create blank fibermap.
• fill_header (list, optional) – A list of HDU names or numbers. The header cards from these HDUs will be added to the header of the camera HDU read from filename.

Returns:

Image object with member variables pix, ivar, mask, readnoise.

Return type:

desispec.image.Image

Raises:

• IOError – If camera is not a HDU in filename.
• KeyError – If EXPTIME is not present in any header in filename, or if both NIGHT and DATE-OBS are missing from input headers.
• ValueError – If NIGHT in the primary header does not match NIGHT in the camera header, or if fill_header is not a list.

Notes

Other keyword arguments are passed to desispec.preproc.preproc(), e.g. bias, pixflat, mask. See preproc() documentation for details.

desispec.io.raw.write_raw(filename, rawdata, header, camera=None, primary_header=None)

Write raw pixel data to a DESI raw data file

Parameters:

• filename – file name to write data; if this exists, append a new HDU
• rawdata – 2D ndarray of raw pixel data including overscans
• header – dict-like object or fits.Header with keywords CCDSECx, BIASSECx, DATASECx where x=A,B,C,D

Options:

camera : b0, r1 .. z9 - override value in header primary_header : header to write in HDU0 if filename doesn’t yet exist

The primary utility of this function over raw fits calls is to ensure that all necessary keywords are present before writing the file. CCDSECx, BIASSECx, DATASECx where x=A,B,C,D DATE-OBS will generate a non-fatal warning if missing

desispec.io.sky

IO routines for sky.

desispec.io.sky.read_sky(filename)

Read sky model and return SkyModel object with attributes wave, flux, ivar, mask, header.

skymodel.wave is 1D common wavelength grid, the others are 2D[nspec, nwave]

desispec.io.sky.write_sky(outfile, skymodel, header=None)

Write sky model.

Parameters:

• outfile – filename or (night, expid, camera) tuple
• skymodel – SkyModel object, with the following attributes wave : 1D wavelength in vacuum Angstroms flux : 2D[nspec, nwave] sky flux ivar : 2D inverse variance of sky flux mask : 2D mask for sky flux stat_ivar : 2D inverse variance of sky flux (statistical only)
• header – optional fits header data (fits.Header, dict, or list)

desispec.io.spectra

I/O routines for working with spectral grouping files.

desispec.io.spectra.read_frame_as_spectra(filename, night=None, expid=None, band=None, single=False)

Read a FITS file containing a Frame and return a Spectra.

A Frame file is very close to a Spectra object (by design), and only differs by missing the NIGHT and EXPID in the fibermap, as well as containing only one band of data.

Parameters:infile (str) – path to read

Options:

night (int): the night value to use for all rows of the fibermap. expid (int): the expid value to use for all rows of the fibermap. band (str): the name of this band. single (bool): if True, keep spectra as single precision in memory.

Returns (Spectra):

The object containing the data read from disk.

desispec.io.spectra.read_spectra(infile, single=False)

Read Spectra object from FITS file.

This reads data written by the write_spectra function. A new Spectra object is instantiated and returned.

Parameters:

• infile (str) – path to read
• single (bool) – if True, keep spectra as single precision in memory.

Returns (Spectra):

The object containing the data read from disk.

desispec.io.spectra.read_tile_spectra(tileid, night, specprod=None, reduxdir=None, coadd=False, single=False, targets=None, fibers=None, redrock=True, group=None)

Read and return combined spectra for a tile/night

Parameters:

• tileid (int) – Tile ID
• night (int or str) – YEARMMDD night or tile group, e.g. ‘deep’ or ‘all’

Options:

specprod (str) : overrides $SPECPROD reduxdir (str) : overrides $DESI_SPECTRO_REDUX/$SPECPROD coadd (bool) : if True, read coadds instead of per-exp spectra single (bool) : if True, use float32 instead of double precision targets (array-like) : filter by TARGETID fibers (array-like) : filter by FIBER redrock (bool) : if True, also return row-matched redrock redshift catalog group (str) : reads spectra in group (pernight, cumulative, …)

Returns: spectra or (spectra, redrock)

combined Spectra obj for all matching targets/fibers filter row-matched redrock catalog (if redrock=True)

Raises:ValueError if no files or matching spectra are found

Note: the returned spectra are not necessarily in the same order as the targets or fibers input filters

desispec.io.spectra.write_spectra(outfile, spec, units=None)

Write Spectra object to FITS file.

This places the metadata into the header of the (empty) primary HDU. The first extension contains the fibermap, and then HDUs are created for the different data arrays for each band.

Floating point data is converted to 32 bits before writing.

Parameters:

• outfile (str) – path to write
• spec (Spectra) – the object containing the data
• units (str) – optional string to use for the BUNIT key of the flux HDUs for each band.

Returns:

The absolute path to the file that was written.

desispec.io.util

Utility functions for desispec IO.

desispec.io.util._dict2ndarray(data, columns=None)

Convert a dictionary of ndarrays into a structured ndarray

Also works if DATA is an AstroPy Table.

Parameters:

• data – input dictionary, each value is an ndarray
• columns – optional list of column names

Returns:

structured numpy.ndarray with named columns from input data dictionary

Notes

data[key].shape[0] must be the same for every key every entry in columns must be a key of data

Example

d = dict(x=np.arange(10), y=np.arange(10)/2) nddata = _dict2ndarray(d, columns=[‘x’, ‘y’])

desispec.io.util._supports_memmap(filename)

Returns True if the filesystem containing filename supports opening memory-mapped files in update mode.

desispec.io.util.add_columns(data, colnames, colvals)

Adds extra columns to a data table

Parameters:

• data – astropy Table or numpy structured array
• colnames – list of new column names to add
• colvals – list of column values to add; each element can be scalar or vector

Returns:

new table with extra columns added

Example

fibermap = add_columns(fibermap,

[‘NIGHT’, ‘EXPID’], [20102020, np.arange(len(fibermap))])

Notes

This is similar to numpy.lib.recfunctions.append_fields, but it also accepts astropy Tables as the data input, and accepts scalar values to expand as entries in colvals.

desispec.io.util.addkeys(hdr1, hdr2, skipkeys=None)

Add new header keys from hdr2 to hdr1, skipping skipkeys

Parameters:

• hdr1 (dict-like) – destination header for keywords
• hdr2 (dict-like) – source header for keywords

Modifies hdr1 in place

desispec.io.util.camword_to_spectros(camword, full_spectros_only=False)

Takes a camword as input and returns any spectrograph represented within that camword. By default this includes partial spectrographs (with one or two cameras represented). But if full_spectros_only is set to True, only spectrographs with all cameras represented are given.

Parameters:

• str. The camword of all cameras. (camword,) –
• bool. Default is False. Flag to specify if you want all spectrographs with any cameras existing (full_spectros_only,) – in the camword (the default) or if you only want fully populated spectrographs.

Returns:

spectros, list. A list of integer spectrograph numbers represented in the camword input.

desispec.io.util.camword_union(camwords, full_spectros_only=False)

Returns the union of a list of camwords. Optionally can return only those spectros with complete b, r, and z cameras. Note this intentionally does the union before truncating spectrographs, so two partial camwords can lead to an entire spectrograph,

e.g. [a0b1z1, a3r1z2] -> [a013z2] if full_spectros_only=False

[a0b1z1, a3r1z2] -> [a013] if full_spectros_only=True

even through no camword has a complete set of camera 1, a complete set is represented in the union.

Parameters:

• list or array of strings. List of camwords. (camwords,) –
• bool. True if only complete spectrographs with (full_spectros_only,) – b, r, and z cameras in the funal union should be returned.

Returns:

final_camword, str. The final union of all input camwords, where

truncation of incomplete spectrographs may or may not be performed based on full_spectros_only.

desispec.io.util.checkgzip(filename)

Check for existence of filename, with or without .gz extension

Parameters:filename (str) – filename to check for

Returns path of existing file without or without .gz, or raises FileNotFoundError if neither exists

desispec.io.util.create_camword(cameras)

Function that takes in a list of cameras and creates a succinct listing of all spectrographs in the list with cameras. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.

Parameters:cameras (1-d array or list) – iterable containing strings of cameras, e.g. ‘b0’,’r1’,…

Returns (str):

A string representing all information about the spectrographs/cameras given in the input iterable, e.g. a01234678b59z9

desispec.io.util.decode_camword(camword)

Function that takes in a succinct listing of all spectrographs and outputs a 1-d numpy array with a list of all spectrograph/camera pairs. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.

Parameters:camword (str) – A string representing all information about the spectrographs/cameras e.g. a01234678b59z9

Returns (np.ndarray, 1d): an array containing strings of

cameras, e.g. ‘b0’,’r1’,…

desispec.io.util.difference_camwords(fullcamword, badcamword)

Returns the difference of two camwords. The second argument cameras are removed from the first argument and the remainer is returned. Smart enough to ignore bad cameras if they don’t exist in full camword list.

Parameters:

• str. The camword of all cameras (fullcamword,) –
• str. A camword defining the bad cameras you don't want to include in the final camword that is output (badcamword,) –

Returns:

str. A camword of cameras in fullcamword that are not in badcamword.

desispec.io.util.fitsheader(header)

Convert header into astropy.io.fits.Header object.

header can be:

• None: return a blank Header
• list of (key, value) or (key, (value,comment)) entries
• dict d[key] -> value or (value, comment)
• Header: just return it unchanged

Returns fits.Header object

desispec.io.util.get_speclog(nights, rawdir=None)

Scans raw data headers to return speclog of observations. Slow.

Parameters:nights – list of YEARMMDD nights to scan

Options:

rawdir (str): overrides $DESI_SPECTRO_DATA

Returns:Table with columns NIGHT,EXPID,MJD,FLAVOR,OBSTYPE,EXPTIME

Scans headers of rawdir/NIGHT/EXPID/desi-EXPID.fits.fz

desispec.io.util.get_tempfilename(filename)

Returns unique tempfile in same directory as filename with same extension

Parameters:filename (str) – input filename

Returns unique filename in same directory

Example intended usage:

tmpfile = get_tempfile(filename)
table.write(tmpfile)
os.rename(tmpfile, filename)

By keeping the same extension as the input file, this preserves the ability of table.write to derive the format to use, and if something goes wrong with the I/O it doesn’t leave a corrupted partially written file with the final name. The tempfile includes the PID to provide some race condition protection (the last one do do os.rename wins, but at least different processes won’t corrupt each other’s files).

desispec.io.util.header2wave(header)

Converts header keywords into a wavelength grid.

returns wave = CRVAL1 + range(NAXIS1)*CDELT1

if LOGLAM keyword is present and true/non-zero, returns 10**wave

desispec.io.util.healpix_subdirectory(nside, pixel)

Return a fixed directory path for healpix grouped files.

Given an NSIDE and NESTED pixel index, return a directory named after a degraded NSIDE and pixel followed by the original nside and pixel. This function is just to ensure that this subdirectory path is always created by the same code.

Parameters:

• nside (int) – a valid NSIDE value.
• pixel (int) – the NESTED pixel index.

Returns (str):

a path containing the low and high resolution directories.

desispec.io.util.is_svn_current(dirname)

Return True/False for if svn checkout dirname is up-to-date with server

Raises ValueError if unable to determine (e.g. dirname isn’t svn checkout)

desispec.io.util.iterfiles(root, prefix, suffix=None)

Returns iterator over files starting with prefix found under root dir Optionally also check if filename ends with suffix

desispec.io.util.makepath(outfile, filetype=None)

Create path to outfile if needed.

If outfile isn’t a string and filetype is set, interpret outfile as a tuple of parameters to locate the file via findfile().

Returns /path/to/outfile

TODO: maybe a different name?

desispec.io.util.native_endian(data)

Convert numpy array data to native endianness if needed.

Returns new array if endianness is swapped, otherwise returns input data

Context: By default, FITS data from astropy.io.fits.getdata() are not Intel native endianness and scipy 0.14 sparse matrices have a bug with non-native endian data.

desispec.io.util.parse_badamps(badamps, joinsymb=', ')

Parses badamps string from an exposure or processing table into the (camera,petal,amplifier) sets, with appropriate checking of those values to make sure they’re valid. Returns empty list if badamps is None.

Parameters:

• str. A string of {camera}{petal}{amp} entries separated by symbol given with joinsymb (comma (badamps,) – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.
• str. The symbol separating entries in the str list given by badamps. (joinsymb,) –

Returns:

cam_petal_amps, list. A list where each entry is a length 3 tuple of (camera,petal,amplifier).

Camera is a lowercase string in [b, r, z]. Petal is an int from 0 to 9. Amplifier is an upper case string in [A, B, C, D].

desispec.io.util.parse_cameras(cameras, loglevel='INFO')

Function that takes in a representation of all spectrographs and outputs a string that succinctly lists all spectrograph/camera pairs. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.

Parameters:str. 1-d array, list (cameras,) – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().

Options:

loglevel, str: use e.g. “WARNING” to avoid INFO-level log messages for just this call

Returns (str):

camword, str. A string representing all information about the spectrographs/cameras

given in the input iterable, e.g. a01234678b59z9

desispec.io.util.replace_prefix(filepath, oldprefix, newprefix)

Replace filename prefix even if prefix is elsewhere in path or filename

Parameters:

• filepath – filename, optionally including path
• oldprefix – original prefix to filename part of filepath
• newprefix – new prefix to use for filename

Returns:

new filepath with replaced prefix

e.g. replace_prefix(‘/blat/foo/blat-bar-blat.fits’, ‘blat’, ‘quat’) returns ‘/blat/foo/quat-bar-blat.fits’

desispec.io.util.validate_badamps(badamps, joinsymb=', ')

Checks (and transforms) badamps string for consistency with the for need in an exposure or processing table for use in the Pipeline. Specifically ensure they come in (camera,petal,amplifier) sets, with appropriate checking of those values to make sure they’re valid. Returns the input string except removing whitespace and replacing potential character separaters with joinsymb (default ‘,’). Returns None if None is given.

Parameters:

• str. A string of {camera}{petal}{amp} entries separated by symbol given with joinsymb (comma (badamps,) – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.
• str. The symbol separating entries in the str list given by badamps. (joinsymb,) –

Returns:

newbadamps, str. Input badamps string of {camera}{petal}{amp} entries separated by symbol given with

joinsymb (comma by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’. Differs from input in that other symbols used to separate terms are replaced by joinsymb and whitespace is removed.

desispec.io.util.write_bintable(filename, data, header=None, comments=None, units=None, extname=None, clobber=False, primary_extname='PRIMARY')

Utility function to write a fits binary table complete with comments and units in the FITS header too. DATA can either be dictionary, an Astropy Table, a numpy.recarray or a numpy.ndarray.

desispec.io.xytraceset

I/O routines for XYTraceSet objects

desispec.io.xytraceset.read_xytraceset(filename)

Reads traces in PSF fits file

Parameters:filename – Path to input fits file which has to contain XTRACE and YTRACE HDUs Returns:XYTraceSet object

desispec.io.xytraceset.write_xytraceset(outfile, xytraceset)

Write a traceset fits file and returns path to file written.

Parameters:

• outfile – full path to output file
• xytraceset – desispec.xytraceset.XYTraceSet object

Returns:

full filepath of output file that was written

desispec.linalg

Some linear algebra functions.

desispec.linalg.cholesky_invert(A)

returns the inverse of a positive definite matrix

Args :

A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray)

Returns:2D positive definite matrix, inverse of A (numpy.ndarray) Return type:cov

desispec.linalg.cholesky_solve(A, B, overwrite=False, lower=False)

Returns the solution X of the linear system A.X=B assuming A is a positive definite matrix

Args :

A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray) B : 1D vector, must have dimension n (numpy.ndarray)

Options :

overwrite: replace A data by cholesky decomposition (faster) lower: cholesky decomposition triangular matrix is lower instead of upper

Returns :

X : 1D vector, same dimension as B (numpy.ndarray)

desispec.linalg.cholesky_solve_and_invert(A, B, overwrite=False, lower=False)

returns the solution X of the linear system A.X=B assuming A is a positive definite matrix

Args :

A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray) B : 1D vector, must have dimension n (numpy.ndarray)

Options :

overwrite: replace A data by cholesky decomposition (faster) lower: cholesky decomposition triangular matrix is lower instead of upper

Returns:X,cov, where X : 1D vector, same dimension n as B (numpy.ndarray) cov : 2D positive definite matrix, inverse of A (numpy.ndarray)

desispec.linalg.spline_fit(output_wave, input_wave, input_flux, required_resolution, input_ivar=None, order=3, max_resolution=None)

Performs spline fit of input_flux vs. input_wave and resamples at output_wave

Parameters:

• output_wave – 1D array of output wavelength samples
• input_wave – 1D array of input wavelengths
• input_flux – 1D array of input flux density
• required_resolution (float) – resolution for spline knot placement (same unit as wavelength)

Options:

input_ivar : 1D array of weights for input_flux order (int) : spline order max_resolution (float) : if not None and first fit fails, try once this resolution

Returns:1D array of flux sampled at output_wave Return type:output_flux

desispec.log

This is a transitional dummy wrapper on desiutil.log.

desispec.log.get_logger(*args, **kwargs)

Transitional dummy wrapper on desiutil.log.get_logger().

desispec.maskbits

Mask bits for the spectro pipeline.

Stephen Bailey, LBNL, January 2015

Example:

from desispec.maskbits import ccdmask

#- bit operations
mask |= ccdmask.COSMIC     #- set ccdmask.COSMIC in integer/array `mask`
mask & ccdmask.COSMIC      #- get ccdmask.COSMIC from integer/array `mask`
(mask & ccdmask.COSMIC) != 0  #- test boolean status of ccdmask.COSMIC in integer/array `mask`
ccdmask.COSMIC | specmask.SATURATED  #- Combine two bitmasks.

#- bit attributes
ccdmask.mask('COSMIC')     #- 2**0, same as ccdmask.COSMIC
ccdmask.mask(0)            #- 2**0, same as ccdmask.COSMIC
ccdmask.COSMIC             #- 2**0, same as ccdmask.mask('COSMIC')
ccdmask.bitnum('COSMIC')   #- 0
ccdmask.bitname(0)         #- 'COSMIC'
ccdmask.names()            #- ['COSMIC', 'HOT', 'DEAD', 'SATURATED', ...]
ccdmask.names(3)           #- ['COSMIC', 'HOT']
ccdmask.comment(0)         #- "Cosmic ray"
ccdmask.comment('BADPIX')  #- "Cosmic ray"

desispec.parallel

Helper functions and classes for dealing with parallelization and related topics.

desispec.parallel.default_nproc = 1

Default number of multiprocessing processes. Set globally on first import.

desispec.parallel.dist_balanced(nwork, maxworkers)

Distribute items between a flexible number of workers.

This assumes that each item has equal weight, and that they should be divided into contiguous blocks of items and assigned to workers in rank order.

If the number of workers is less than roughly sqrt(nwork), then we do not reduce the number of workers and the result is the same as the dist_uniform function. If there are more workers than this, then the number of workers is reduced until all workers have close to the same number of tasks.

Parameters:

• nwork (int) – The number of work items.
• maxworkers (int) – The maximum number of workers. The actual number may be less than this.

Returns:

A list of tuples, one for each worker. The first element of the tuple is the first item assigned to the worker, and the second element is the number of items assigned to the worker.

desispec.parallel.dist_discrete(worksizes, nworkers, workerid, power=1.0)

Distribute indivisible blocks of items between groups.

Given some contiguous blocks of items which cannot be subdivided, distribute these blocks to the specified number of groups in a way which minimizes the maximum total items given to any group. Optionally weight the blocks by a power of their size when computing the distribution.

This is effectively the “Painter”s Partition Problem”.

Parameters:

• worksizes (list) – The sizes of the indivisible blocks.
• nworkers (int) – The number of workers.
• workerid (int) – The worker ID whose range should be returned.
• power (float) – The power to use for weighting

Returns:

A tuple. The first element of the tuple is the first block assigned to the worker ID, and the second element is the number of blocks assigned to the worker.

desispec.parallel.dist_discrete_all(worksizes, nworkers, power=1.0)

Distribute indivisible blocks of items between groups.

Given some contiguous blocks of items which cannot be subdivided, distribute these blocks to the specified number of groups in a way which minimizes the maximum total items given to any group. Optionally weight the blocks by a power of their size when computing the distribution.

This is effectively the “Painter”s Partition Problem”.

Parameters:

• worksizes (list) – The sizes of the indivisible blocks.
• nworkers (int) – The number of workers.
• pow (float) – The power to use for weighting

Returns:

list of length nworkers; each element is a list of indices of

worksizes assigned to that worker.

desispec.parallel.dist_uniform(nwork, nworkers, id=None)

Statically distribute some number of items among workers.

This assumes that each item has equal weight, and that they should be divided into contiguous blocks of items and assigned to workers in rank order.

This function returns the index of the first item and the number of items for the specified worker ID, or the information for all workers.

Parameters:

• nwork (int) – the number of things to distribute.
• nworkers (int) – the number of workers.
• id (int) – optionally return just the tuple associated with this worker

Returns (tuple):

A tuple of ints, containing the first item and number of items. If id=None, then return a list containing the tuple for all workers.

desispec.parallel.stdouterr_redirected(to=None, comm=None)

Redirect stdout and stderr to a file.

The general technique is based on:

http://stackoverflow.com/questions/5081657 http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/

One difference here is that each process in the communicator redirects to a different temporary file, and the upon exit from the context the rank zero process concatenates these in order to the file result.

Parameters:

• to (str) – The output file name.
• comm (mpi4py.MPI.Comm) – The optional MPI communicator.

desispec.parallel.take_turns(comm, at_a_time, func, *args, **kwargs)

Processes call a function in groups.

Any extra positional and keyword arguments are passed to the function.

Parameters:

• comm – mpi4py.MPI.Comm or None.
• at_a_time (int) – the maximum number of processes to run at a time.
• func – the function to call.

Returns:

The return value on each process is the return value of the function.

desispec.parallel.use_mpi()

Return whether we can use MPI.

desispec.parallel.weighted_partition(weights, n, groups_per_node=None)

Partition weights into n groups with approximately same sum(weights)

Parameters:

• weights – array-like weights
• n – number of groups

Returns list of lists of indices of weights for each group

Notes

compared to dist_discrete_all, this function allows non-contiguous items to be grouped together which allows better balancing.

desispec.pipeline

Tools for pipeline creation and running.

desispec.pipeline.control

Tools for controling pipeline production.

desispec.pipeline.control.chain(tasktypes, nightstr=None, states=None, expid=None, spec=None, pack=False, nosubmitted=False, depjobs=None, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, dryrun=False)

Run a chain of jobs for multiple pipeline steps.

For the list of task types, get all ready tasks meeting the selection criteria. Then either pack all tasks into one job or submit each task type as its own job. Input job dependencies can be specified, and dependencies are tracked between jobs in the chain.

Parameters:

• tasktypes (list) – list of valid task types.
• nightstr (str) – Comma separated (YYYYMMDD) or regex pattern. Only nights matching these patterns will be considered.
• states (list) – list of task states to select.
• nights (list) – list of nights to select.
• expid (int) – exposure ID to select.
• pack (bool) – if True, pack all tasks into a single job.
• nosubmitted (bool) – if True, do not run jobs that have already been submitted.
• depjobs (list) – list of job ID dependencies.
• nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
• nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
• nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
• nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
• nersc_shifter (str) – the name of the shifter image to use.
• mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
• mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
• procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
• nodb (bool) – if True, do not use the production DB.
• out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
• debug (bool) – if True, enable DEBUG log level in generated scripts.
• dryrun (bool) – if True, do not submit the jobs.

Returns:

the job IDs from the final step in the chain.

Return type:

list

desispec.pipeline.control.check_tasks(tasks, db=None)

Check the state of pipeline tasks.

If the database handle is given, use the DB for checking. Otherwise use the filesystem.

Parameters:

• tasks (list) – list of tasks to check.
• db (DataBase) – the database to use.

Returns:

dictionary of the state of each task.

Return type:

OrderedDict

desispec.pipeline.control.cleanup(db, tasktypes, failed=False, submitted=False, expid=None)

Clean up stale tasks in the DB.

Parameters:

• db (DataBase) – the production DB.
• tasktypes (list) – list of valid task types.
• failed (bool) – also clear failed states.
• submitted (bool) – also clear submitted flag.
• expid (int) – only clean this exposure ID.

desispec.pipeline.control.create(root=None, data=None, redux=None, prod=None, force=False, basis=None, calib=None, db_sqlite=False, db_sqlite_path=None, db_postgres=False, db_postgres_host='nerscdb03.nersc.gov', db_postgres_port=5432, db_postgres_name='desidev', db_postgres_user='desidev_admin', db_postgres_authorized='desidev_ro', nside=64)

Create (or re-create) a production.

Parameters:

• root (str) – value to use for DESI_ROOT.
• data (str) – value to use for DESI_SPECTRO_DATA.
• redux (str) – value to use for DESI_SPECTRO_REDUX.
• prod (str) – value to use for SPECPROD.
• force (bool) – if True, overwrite existing production DB.
• basis (str) – value to use for DESI_BASIS_TEMPLATES.
• calib (str) – value to use for DESI_SPECTRO_CALIB.
• db_sqlite (bool) – if True, use SQLite for the DB.
• db_sqlite_path (str) – override path to SQLite DB.
• db_postgres (bool) – if True, use PostgreSQL for the DB.
• db_postgres_host (str) – PostgreSQL hostname.
• db_postgres_port (int) – PostgreSQL connection port number.
• db_postgres_name (str) – PostgreSQL DB name.
• db_postgres_user (str) – PostgreSQL user name.
• db_postgres_authorized (str) – Additional PostgreSQL users to authorize.
• nside (int) – HEALPix nside value used for spectral grouping.

desispec.pipeline.control.dryrun(tasks, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, db_postgres_user='desidev_ro', force=False)

Print equivalent command line jobs.

For the specified tasks, print the equivalent stand-alone commands that would be run on each task. A pipeline job calls the internal desispec.scripts entrypoints directly.

Parameters:

• tasks (list) – list of tasks to run.
• nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
• nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
• nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
• nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
• nersc_shifter (str) – the name of the shifter image to use.
• mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
• mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
• procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
• nodb (bool) – if True, do not use the production DB.
• db_postgres_user (str) – If using postgres, connect as this user for read-only access”
• force (bool) – if True, print commands for all tasks, not just the ones in a ready state.

desispec.pipeline.control.gen_scripts(tasks_by_type, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, db_postgres_user='desidev_ro')

Generate scripts to run tasks of one or more types.

If multiple task type keys are contained in the dictionary, they will be packed into a single batch job.

Parameters:

• tasks_by_type (dict) – each key is the task type and the value is a list of tasks.
• nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
• nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
• nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
• nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
• nersc_shifter (str) – the name of the shifter image to use.
• mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
• mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
• procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
• nodb (bool) – if True, do not use the production DB.
• out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
• debug (bool) – if True, enable DEBUG log level in generated scripts.
• db_postgres_user (str) – If using postgres, connect as this user for read-only access”

Returns:

the generated script files

Return type:

list

desispec.pipeline.control.get_tasks(db, tasktypes, nights, states=None, expid=None, spec=None, nosubmitted=False, taskfile=None)

Get tasks of multiple types that match certain criteria.

Parameters:

• db (DataBase) – the production DB.
• tasktypes (list) – list of valid task types.
• states (list) – list of task states to select.
• nights (list) – list of nights to select.
• expid (int) – exposure ID to select.
• spec (int) – spectrograph to select.
• nosubmitted (bool) – if True, ignore tasks that were already submitted.

Returns:

all tasks of all types.

Return type:

list

desispec.pipeline.control.get_tasks_type(db, tasktype, states, nights, expid=None, spec=None)

Get tasks of one type that match certain criteria.

Parameters:

• db (DataBase) – the production DB.
• tasktype (str) – a valid task type.
• states (list) – list of task states to select.
• nights (list) – list of nights to select.
• expid (int) – exposure ID to select.
• spec (int) – spectrograph to select.

Returns:

list of tasks meeting the criteria.

Return type:

(list)

desispec.pipeline.control.getready(db, nightstr=None)

Update forward dependencies in the database.

Update database for one or more nights to ensure that forward dependencies know that they are ready to run.

Parameters:

• db (DataBase) – the production DB.
• nightstr (list) – comma separated (YYYYMMDD) or regex pattern.

desispec.pipeline.control.run(taskfile, nosubmitted=False, depjobs=None, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False)

Create job scripts and run them.

This gets tasks from the taskfile and sorts them by type. Then it generates the scripts. Finally, it runs or submits those scripts to the scheduler.

Parameters:

• taskfile (str) – read tasks from this file (if not specified, read from STDIN).
• nosubmitted (bool) – if True, do not run jobs that have already been submitted.
• depjobs (list) – list of job ID dependencies.
• nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
• nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
• nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
• nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
• nersc_shifter (str) – the name of the shifter image to use.
• mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
• mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
• procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
• nodb (bool) – if True, do not use the production DB.
• out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
• debug (bool) – if True, enable DEBUG log level in generated scripts.

Returns:

the job IDs returned by the scheduler.

Return type:

list

desispec.pipeline.control.run_scripts(scripts, deps=None, slurm=False)

Run job scripts with optional dependecies.

This either submits the jobs to the scheduler or simply runs them in order with subprocess.

Parameters:

• scripts (list) – list of pathnames of the scripts to run.
• deps (list) – optional list of job IDs which are dependencies for these scripts.
• slurm (bool) – if True use slurm to submit the jobs.

Returns:

the job IDs returned by the scheduler.

Return type:

list

desispec.pipeline.control.script(taskfile, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, db_postgres_user='desidev_ro')

Generate pipeline scripts for a taskfile.

This gets tasks from the taskfile and sorts them by type. Then it generates the scripts.

Parameters:

• taskfile (str) – read tasks from this file (if not specified, read from STDIN).
• nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
• nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
• nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
• nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
• nersc_shifter (str) – the name of the shifter image to use.
• mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
• mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
• procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
• nodb (bool) – if True, do not use the production DB.
• out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
• debug (bool) – if True, enable DEBUG log level in generated scripts.
• db_postgres_user (str) – If using postgres, connect as this user for read-only access”

Returns:

the generated script files

Return type:

list

desispec.pipeline.control.status(task=None, tasktypes=None, nightstr=None, states=None, expid=None, spec=None, db_postgres_user='desidev_ro')

Check the status of pipeline tasks.

Args:

Returns:None

desispec.pipeline.control.sync(db, nightstr=None, specdone=False)

Synchronize DB state based on the filesystem.

This scans the filesystem for all tasks for the specified nights, and updates the states accordingly.

Parameters:

• db (DataBase) – the production DB.
• nightstr (list) – comma separated (YYYYMMDD) or regex pattern.
• specdone – If true, set spectra to done if files exist.

desispec.pipeline.control.tasks(tasktypes, nightstr=None, states=None, expid=None, spec=None, nosubmitted=False, db_postgres_user='desidev_ro', taskfile=None)

Get tasks of multiple types that match certain criteria.

Parameters:

• tasktypes (list) – list of valid task types.
• nightstr (list) – comma separated (YYYYMMDD) or regex pattern.
• states (list) – list of task states to select.
• expid (int) – exposure ID to select.
• spec (int) – spectrograph to select.
• nosubmitted (bool) – if True, ignore tasks that were already submitted.
• db_postgres_user (str) – If using postgres, connect as this user for read-only access”
• taskfile (str) – if set write to this file, else write to STDOUT.

desispec.pipeline.control.update(nightstr=None, nside=64, expid=None)

Update a production.

Parameters:

• nightstr (str) – Comma separated (YYYYMMDD) or regex pattern. Only nights matching these patterns will be considered.
• nside (int) – HEALPix nside value used for spectral grouping.
• expid (int) – Only update the production for a single exposure ID.

desispec.pipeline.db

Pipeline processing database

class desispec.pipeline.db.DataBase

Class for tracking pipeline processing objects and state.

cleanup(tasktypes=None, expid=None, cleanfailed=False, cleansubmitted=False)

Reset states of tasks.

Any tasks that are marked as “running” will have their state reset to “ready”. This can be called if a job dies before completing all tasks.

Parameters:

• tasktypes (list) – if not None, clean up only tasks of these types.
• expid (int) – if not None, only clean tasks related to this exposure ID. Note that tasks which are independent of an expid (psfnight, fiberflatnight, spectra, redshift) will be ignored if this option is given.
• cleanfailed (bool) – if True, also reset failed tasks to ready.
• cleansubmitted (bool) – if True, set submitted flag to False.

count_task_states(tasktype)

Return a dictionary of how many tasks are in each state

Parameters:tasktype (str) – the type of these tasks. Returns:keyed by state, values are number of tasks in that state0 Return type:dict

get_states(tasks)

Efficiently get the state of many tasks at once.

Parameters:tasks (list) – list of task names. Returns:the state of each task. Return type:dict

get_states_type(tasktype, tasks)

Efficiently get the state of many tasks of a single type.

Parameters:

• tasktype (str) – the type of these tasks.
• tasks (list) – list of task names.

Returns:

the state of each task.

Return type:

dict

get_submitted(tasks)

Return the submitted flag for the list of tasks.

Parameters:tasks (list) – list of task names. Returns:

the boolean submitted state of each task (True means that

the task has been submitted).

Return type:(dict)

getready(night=None)

Update DB, changing waiting to ready depending on status of dependencies .

Parameters:night (str) – The night to process.

set_states(tasks)

Efficiently set the state of many tasks at once.

Parameters:tasks (list) – list of tuples containing the task name and the state to set. Returns:Nothing.

set_states_type(tasktype, tasks, postprocessing=True)

Efficiently get the state of many tasks of a single type.

Parameters:

• tasktype (str) – the type of these tasks.
• tasks (list) – list of tuples containing the task name and the state to set.

Returns:

Nothing.

set_submitted(tasks, unset=False)

Flag a list of tasks as submitted.

Parameters:

• tasks (list) – list of task names.
• unset (bool) – if True, invert the behavior and unset the submitted flag for these tasks.

Returns:

Nothing.

set_submitted_type(tasktype, tasks, unset=False)

Flag a list of tasks of a single type as submitted.

Parameters:

• tasktype (str) – the type of these tasks.
• tasks (list) – list of task names.
• unset (bool) – if True, invert the behavior and unset the submitted flag for these tasks.

Returns:

Nothing.

sync(night, specdone=False)

Update states of tasks based on filesystem.

Go through all tasks in the DB for the given night and determine their state on the filesystem. Then update the DB state to match.

Parameters:

• night (str) – The night to scan for updates.
• specdone – If true, set spectra to done if files exist.

update(night, nside, expid=None)

Update DB based on raw data.

This will use the usual io.meta functions to find raw exposures. For each exposure, the fibermap and all following objects will be added to the DB.

Parameters:

• night (str) – The night to scan for updates.
• nside (int) – The current NSIDE value used for pixel grouping.
• expid (int) – Only update the DB for this exposure.

class desispec.pipeline.db.DataBasePostgres(host, port, dbname, user, schema=None, authorize=None)

Pipeline database using PostgreSQL as the backend.

Parameters:

• host (str) – The database server.
• port (int) – The connection port.
• dbname (str) – The database to connect.
• user (str) – The user name for the connection. The password should be stored in the ~/.pgpass file.
• schema (str) – The schema within the database. If this is specified, then the database is assumed to exist. Otherwise the schema is computed from a hash of the production location and will be created.
• authorize (str) – If creating the schema, this is the list of additional roles that should be granted access.

initdb()

Create DB tables for all tasks if they do not exist.

class desispec.pipeline.db.DataBaseSqlite(path, mode)

Pipeline database using sqlite3 as the backend.

Parameters:

• path (str) – the filesystem path of the database to open. If None, then a temporary database is created in memory.
• mode (str) – if “r”, the database is open in read-only mode. If “w”, the database is open in read-write mode and created if necessary.

initdb()

Create DB tables for all tasks if they do not exist.

desispec.pipeline.db.all_task_types()

Get the list of possible task types that are supported.

Returns:The list of supported task types. Return type:list

desispec.pipeline.db.all_tasks(night, nside, expid=None)

Get all possible tasks for a single night.

This uses the filesystem to query the raw data for a particular night and return a dictionary containing all possible tasks for each task type. For objects which span multiple nights (e.g. spectra, redrock), this returns the tasks which are touched by the given night.

Parameters:

• night (str) – The night to scan for tasks.
• nside (int) – The HEALPix NSIDE value to use.
• expid (int) – Only get tasks for this single exposure.

Returns:

a dictionary whose keys are the task types and where each value

is a list of task properties.

Return type:

dict

desispec.pipeline.db.check_tasks(tasklist, db=None, inputs=None)

Check a list of tasks and return their state.

If the database is specified, it is used to check the state of the tasks and their dependencies. Otherwise the filesystem is checked.

Parameters:

• tasklist (list) – list of tasks.
• db (pipeline.db.DB) – The optional database to use.
• inputs (dict) – optional dictionary containing the only input dependencies that should be considered.

Returns:

The current state of all tasks.

Return type:

dict

desispec.pipeline.db.load_db(dbstring, mode='w', user=None)

Load a database from a connection string.

This instantiates either an sqlite or postgresql database using a string. If this string begins with “postgresql:”, then it is taken to be the information needed to connect to a postgres server. Otherwise it is assumed to be a filesystem path to use with sqlite. The mode is only meaningful when using sqlite. Postgres permissions are controlled through the user permissions.

Parameters:

• dbstring (str) – either a filesystem path (sqlite) or a colon-separated string of connection properties in the form “postresql:<host>:<port>:<dbname>:<user>:<schema>”.
• mode (str) – for sqlite, the mode.
• user (str) – for postgresql, an alternate user name for opening the DB. This can be used to connect as a user with read-only access.

Returns:

a derived database class of the appropriate type.

Return type:

DataBase

desispec.pipeline.db.task_sort(tasks)

Sort a list of tasks by type.

This takes a list of arbitrary tasks and sorts them by type. The result is placed in an ordered dictionary of lists in run order.

Parameters:tasks (list) – the list of input tasks. Returns:ordered dictionary of tasks sorted by type. Return type:(OrderedDict)

desispec.pipeline.defs

Common definitions needed by pipeline modules.

desispec.pipeline.defs.prod_options_name = 'options.yaml'

The name of the options file inside the run directory.

desispec.pipeline.defs.state_colors = {'done': '#00ff00', 'failed': '#ff0000', 'ready': '#0000ff', 'running': '#ffff00', 'waiting': '#000000'}

State colors used for visualization.

desispec.pipeline.defs.task_name_sep = '_'

The separator string used for building object names.

desispec.pipeline.defs.task_states = ['waiting', 'ready', 'running', 'done', 'failed']

The valid states of each pipeline task.

desispec.pipeline.prod

Functions for updating and loading a production.

desispec.pipeline.prod.load_prod(mode='w', user=None)

Load the database and options for a production.

This loads the database from the production location defined by the usual DESI environment variables. It also loads the global options file for the production.

Parameters:

• mode (str) – open mode for sqlite database (“r” or “w”).
• user (str) – for postgresql, an alternate user name for opening the DB. This can be used to connect as a user with read-only access.

Returns:

(pipeline.db.DataBase, dict) The database for the production

and the global options dictionary.

Return type:

tuple

desispec.pipeline.prod.select_nights(allnights, nightstr)

Select nights based on regex matches.

Given a list of nights, select all nights matching the specified patterns and return this subset.

Parameters:

• allnights (list) – list of all nights as strings
• nightstr (str) – comma-separated list of regex patterns.

Returns:

list of nights that match the patterns.

Return type:

list

desispec.pipeline.prod.task_read(path)

Read a task list from a text file or STDIN.

Lines that begin with ‘#’ are ignored as comments. If the path is None, lines are read from STDIN until an EOF marker is received.

Parameters:path (str) – the input file name. Returns:the list of tasks. Return type:list

desispec.pipeline.prod.task_write(path, tasklist)

Write a task list to a text file or STDOUT.

If the path is None, write lines to STDOUT. In all cases, write a special termination line so that this stream or file can be passed into the task_read function.

Parameters:

• path (str) – the output file name.
• tasklist (list) – the data.

Returns:

nothing.

desispec.pipeline.prod.update_prod(nightstr=None, hpxnside=64, expid=None)

Create or update a production directory tree.

For a given production, create the directory hierarchy and the starting default options.yaml file if it does not exist. Also initialize the production DB if it does not exist. Then update the DB with one or more nights from the raw data. Nights to update may be specified by a list of simple regex matches.

Parameters:

• nightstr (str) – comma-separated list of regex patterns.
• hpxnside (int) – The nside value to use for spectral grouping.
• expid (int) – Only update a single exposure. If this is specified, then nightstr must contain only a single night.

desispec.pipeline.prod.yaml_read(path)

Read a dictionary from a file.

Parameters:path (str) – the input file name. Returns:the data. Return type:dict

desispec.pipeline.prod.yaml_write(path, input)

Write a dictionary to a file.

Parameters:

• path (str) – the output file name.
• input (dict) – the data.

Returns:

nothing.

desispec.pipeline.run

Tools for running the pipeline.

exception desispec.pipeline.run.TimeoutError

desispec.pipeline.run.dry_run(tasktype, tasklist, opts, procs, procs_per_node, db=None, launch='mpirun -np', force=False)

Compute the distribution of tasks and equivalent commands.

This function takes similar arguments as run_task_list() except simulates the data distribution and commands that would be run if given the specified number of processes and processes per node.

This can be used to debug issues with the runtime concurrency or the actual options that will be passed to the underying main() entry points for each task.

This function requires that the DESI environment variables are set to point to the current production directory.

Only tasks that are ready to run (based on the filesystem checks or the database) will actually be attempted.

NOTE: Since this function is just informative and for interactive use, we print information directly to STDOUT rather than logging.

Parameters:

• tasktype (str) – the pipeline step to process.
• tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.
• opts (dict) – the global options (for example, as read from the production options.yaml file).
• procs (int) – the number of processes to simulate.
• procs_per_node (int) – the number of processes per node to simulate.
• db (pipeline.db.DB) – The optional database to update.
• launch (str) – The launching command for a job. This is just a convenience and prepended to each command before the number of processes.
• force (bool) – If True, ignore database and filesystem state and just run the tasks regardless.

Returns:

Nothing.

desispec.pipeline.run.run_dist(tasktype, tasklist, db, nproc, procs_per_node, force=False)

Compute the runtime distribution of tasks.

For a given number of processes, parse job environment variables and compute the number of workers to use and the remaining tasks to process. Divide the processes into groups, and associate some (or all) of those groups to workers. Assign tasks to these groups of processes. Some groups may have zero tasks if there are more groups than workers needed.

Returns:

The (groupsize, groups, tasks, dist) information. Groupsize

is the processes per group. Groups is a list of tuples (one per process) giving the group number and rank within the group. The tasks are a sorted list of tasks containing the subset of the inputs that needs to be run. The dist is a list of tuples (one per group) containing the indices of tasks assigned to each group.

Return type:tuple

desispec.pipeline.run.run_task(name, opts, comm=None, logfile=None, db=None)

Run a single task.

Based on the name of the task, call the appropriate run function for that task. Log output to the specified file. Run using the specified MPI communicator and optionally update state to the specified database.

Note: This function DOES NOT check the database or filesystem to see if the task has been completed or if its dependencies exist. It assumes that some higher-level code has done that if necessary.

Parameters:

• name (str) – the name of this task.
• opts (dict) – options to use for this task.
• comm (mpi4py.MPI.Comm) – optional MPI communicator.
• logfile (str) – output log file. If None, do not redirect output to a file.
• db (pipeline.db.DB) – The optional database to update.

Returns:

the total number of processes that failed.

Return type:

int

desispec.pipeline.run.run_task_list(tasktype, tasklist, opts, comm=None, db=None, force=False)

Run a collection of tasks of the same type.

This function requires that the DESI environment variables are set to point to the current production directory.

This function first takes the communicator and uses the maximum processes per task to split the communicator and form groups of processes of the desired size. It then takes the list of tasks and uses their relative run time estimates to assign tasks to the process groups. Each process group loops over its assigned tasks.

If the database is not specified, no state tracking will be done and the filesystem will be checked as needed to determine the current state.

Only tasks that are ready to run (based on the filesystem checks or the database) will actually be attempted.

Parameters:

• tasktype (str) – the pipeline step to process.
• tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.
• opts (dict) – the global options (for example, as read from the production options.yaml file).
• comm (mpi4py.Comm) – the full communicator to use for whole set of tasks.
• db (pipeline.db.DB) – The optional database to update.
• force (bool) – If True, ignore database and filesystem state and just run the tasks regardless.

Returns:

the number of ready tasks, number that are done, and the number

that failed.

Return type:

tuple

desispec.pipeline.run.run_task_list_db(tasktype, tasklist, comm=None)

Run a list of tasks using the pipeline DB and options.

This is a wrapper around run_task_list which uses the production database and global options file.

Parameters:

• tasktype (str) – the pipeline step to process.
• tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.
• comm (mpi4py.Comm) – the full communicator to use for whole set of tasks.

Returns:

the number of ready tasks, and the number that failed.

Return type:

tuple

desispec.pipeline.run.run_task_simple(name, opts, comm=None)

Run a single task with no DB or log redirection.

This a wrapper around run_task() for use without a database and with no log redirection. See documentation for that function.

Parameters:

• name (str) – the name of this task.
• opts (dict) – options to use for this task.
• comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the total number of processes that failed.

Return type:

int

desispec.pipeline.scriptgen

Tools for generating shell and slurm scripts.

desispec.pipeline.scriptgen.batch_nersc(tasks_by_type, outroot, logroot, jobname, machine, queue, maxtime, maxnodes, nodeprocs=None, openmp=False, multiproc=False, db=None, shifterimg=None, debug=False)

Generate slurm script(s) to process lists of tasks.

Given sets of task lists and constraints about the machine, generate slurm scripts for use at NERSC.

Parameters:

• tasks_by_type (OrderedDict) – Ordered dictionary of the tasks for each type to be written to a single job script.
• outroot (str) – root output script name.
• logroot (str) – root output log name.
• jobname (str) – the name of the job.
• machine (str) – the NERSC machine name.
• queue (str) – the name of the queue
• maxtime (int) – the maximum run time in minutes.
• maxnodes (int) – the maximum number of nodes to use.
• nodeprocs (int) – the number of processes to use per node.
• openmp (bool) – if True, set OMP_NUM_THREADS to the correct value.
• multiproc (bool) – if True, use OMP_NUM_THREADS=1 and disable core binding of processes.
• db (DataBase) – the pipeline database handle.
• shifter (str) – the name of the shifter image to use.
• debug (bool) – if True, set DESI log level to DEBUG in the script.

Returns:

list of generated slurm files.

Return type:

(list)

desispec.pipeline.scriptgen.batch_shell(tasks_by_type, outroot, logroot, mpirun='', mpiprocs=1, openmp=1, db=None)

Generate bash script(s) to process lists of tasks.

Given sets of task lists, generate a script that processes each in order.

Parameters:

• tasks_by_type (OrderedDict) – Ordered dictionary of the tasks for each type to be written to a single job script.
• outroot (str) – root output script name.
• logroot (str) – root output log name.
• mpirun (str) – optional command to use for launching MPI programs.
• mpiprocs (int) – if mpirun is specified, use this number of processes.
• openmp (int) – value to set for OMP_NUM_THREADS.
• db (DataBase) – the pipeline database handle.

Returns:

list of generated script files.

Return type:

(list)

desispec.pipeline.scriptgen.dump_job_env(fh, tfactor, startup, nworker, workersize)

Write parameters needed at runtime to an open filehandle.

desispec.pipeline.scriptgen.nersc_job(jobname, path, logroot, desisetup, commands, machine, queue, nodes, cnodes, ppns, minutes, nworker, workersize, multisrun=False, openmp=False, multiproc=False, shifterimg=None, debug=False)

Create a SLURM script for use at NERSC.

Args:

desispec.pipeline.scriptgen.parse_job_env()

Retrieve job parameters from the environment.

desispec.pipeline.tasks

Classes that describe pipeline tasks.

desispec.pipeline.tasks.base

Common operations for pipeline tasks.

class desispec.pipeline.tasks.base.BaseTask

Base class for tasks.

This defines the interfaces for the classes representing pipeline tasks. This class should not be instantiated directly.

_create(db)

See BaseTask.create.

_insert(cursor, props)

See BaseTask.insert.

_retrieve(db, name)

See BaseTask.retrieve.

_run_max_mem_proc(name, db)

Return zero (i.e. not a limit)

_run_max_mem_task(name, db)

Return zero (i.e. no memory requirement)

_state_get(db, name, cur=None)

See BaseTask.state_get.

_state_set(db, name, state, cur=None)

See BaseTask.state_set.

create(db)

Initialize a database for this task type.

This may include creating one or more tables.

Parameters:db (pipeline.DB) – the database instance.

deps(name, db=None, inputs=None)

Get the dependencies for a task.

This gets a list of other tasks which are required.

Parameters:

• name (str) – the task name.
• db (pipeline.DB) – the optional database instance.
• inputs (dict) – optional dictionary containing the only input dependencies that should be considered.

Returns:

a dictionary of dependencies. The keys are arbitrary and

the values can be either scalar task names or lists of tasks.

Return type:

dict

getready(db, name, cur)

Checks whether dependencies are ready

insert(cursor, props)

Insert a task into a database.

This uses the name and extra keywords to update one or more task-specific tables.

Parameters:

• cursor (DB cursor) – the database cursor of an open connection.
• props (dict) – dictionary of properties for the task.

name_join(props)

Construct a task name from its properties.

Parameters:props (dict) – dictionary of properties. Returns:the task name. Return type:str

name_split(name)

Split a task name into its properties.

Parameters:name (str) – the task name. Returns:dictionary of properties. Return type:dict

paths(name)

The filesystem path(s) associated with this task.

Parameters:name (str) – the task name. Returns:the list of output files generated by this task. Return type:list

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

retrieve(db, name)

Retrieve all task information from the DB.

This may include additional information beyond the contents of the task name (e.g. from other tables).

Parameters:

• db (pipeline.DB) – the database instance.
• name (str) – the task name.

Returns:

dictionary of properties for the task.

Return type:

dict

run(name, opts, comm=None, db=None)

Run the task.

Parameters:

• name (str) – the name of this task.
• opts (dict) – options to use for this task.
• comm (mpi4py.MPI.Comm) – optional MPI communicator.
• db (pipeline.db.DB) – The database.

Returns:

the number of processes that failed.

Return type:

int

run_and_update(db, name, opts, comm=None)

Run the task and update DB state.

The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.

Parameters:

• db (pipeline.db.DB) – The database.
• name (str) – the name of this task.
• opts (dict) – options to use for this task.
• comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the number of processes that failed.

Return type:

int

run_cli(name, opts, procs, launch=None, log=None, db=None)

Return the equivalent command-line interface.

Parameters:

• name (str) – the name of the task.
• opts (dict) – dictionary of runtime options.
• procs (int) – The number of processes to use.
• launch (str) – optional launching command.
• log (str) – optional log file for output.
• db (pipeline.db.DB) – The database.

Returns:

a command line.

Return type:

str

run_defaults()

Default options.

This dictionary of default options will be written to the options.yaml file in a production directory. The options will then be loaded from that file at run time.

Changes to this function will only impact newly-created productions, and these options will be overridden by any changes to the options.yaml file.

Returns:dictionary of default options. Return type:dict

run_max_mem_proc(name, db=None)

Maximum memory in GB per process required.

If zero is returned, it indicates that the memory requirement is so small that the code can run fully-packed on any system.

Parameters:

• name (str) – the name of the task.
• db (pipeline.DB) – the optional database instance.

Returns:

the required RAM in GB per process.

Return type:

float

run_max_mem_task(name, db=None)

Maximum memory in GB per task required.

If zero is returned, it indicates that the memory requirement is so small that the code can run on a single node.

Parameters:

• name (str) – the name of the task.
• db (pipeline.DB) – the optional database instance.

Returns:

the required RAM in GB per process.

Return type:

float

run_max_procs()

Maximum number of processes supported by this task type.

Parameters:procs_per_node (int) – the number of processes running per node. Returns:the maximum number of processes. Zero indicates no limit. Return type:int

run_time(name, procs, db=None)

Estimated runtime for a task at maximum concurrency.

Parameters:

• name (str) – the name of the task.
• procs (int) – the total number of processes used for this task.
• db (pipeline.DB) – the optional database instance.

Returns:

estimated minutes of run time.

Return type:

int

state_get(db, name, cur=None)

Get the state of a task.

This should not be called repeatedly for many tasks- it is more efficient to get the state of many tasks in a single custom SQL query.

Parameters:

• db (pipeline.DB) – the database instance.
• name (str) – the task name.

Returns:

the state.

Return type:

str

state_set(db, name, state, cur=None)

Set the state of a task.

This should not be called repeatedly if you are setting the state of many tasks. It is more efficient to do that in a single SQL command.

Parameters:

• db (pipeline.DB) – the database instance.
• name (str) – the task name.

desispec.pipeline.tasks.base.task_type(name)

Given a task name, find the type from the list of available ones.

Parameters:name (str) – the name of the task. Returns:the type of the task. Return type:str

class desispec.pipeline.tasks.cframe.TaskCFrame

Class containing the properties of a sky fit task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

_run_max_procs()

See BaseTask.run_max_procs.

_run_time(name, procs, db)

See BaseTask.run_time.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.extract.TaskExtract

Class containing the properties of one extraction task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.fiberflat.TaskFiberflat

Class containing the properties of one extraction task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.fiberflatnight.TaskFiberflatNight

Class containing the properties of one fiberflat combined night task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

getready(db, name, cur)

Checks whether dependencies are ready

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.fibermap.TaskFibermap

Class containing the properties of one fibermap.

Since fibermaps have no dependencies and are not created by the pipeline, this class is just used to specify names, etc.

_deps(name, db, inputs)

See BaseTask.deps.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

class desispec.pipeline.tasks.fluxcalib.TaskFluxCalib

Class containing the properties of a sky fit task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.preproc.TaskPreproc

Class containing the properties of one preprocessed pixel file.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.psf.TaskPSF

Class containing the properties of one PSF task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.psfnight.TaskPSFNight

Class containing the properties of one PSF combined night task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_dict(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

getready(db, name, cur)

Checks whether dependencies are ready

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.qadata.TaskQAData

Class containing the properties of a sky fit task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.rawdata.TaskRawdata

Class containing the properties of one rawdata.

Since rawdatas have no dependencies and are not created by the pipeline, this class is just used to specify names, etc.

_deps(name, db, inputs)

See BaseTask.deps.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

class desispec.pipeline.tasks.redshift.TaskRedshift

Class containing the properties of one spectra task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

run_and_update(db, name, opts, comm=None)

Run the task and update DB state.

The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.

It is specific for redshift because the healpix_frame table has to be updated

Parameters:

• db (pipeline.db.DB) – The database.
• name (str) – the name of this task.
• opts (dict) – options to use for this task.
• comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the number of processes that failed.

Return type:

int

class desispec.pipeline.tasks.sky.TaskSky

Class containing the properties of a sky fit task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.spectra.TaskSpectra

Class containing the properties of one spectra task.

_deps(name, db, inputs)

See BaseTask.deps.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

run_and_update(db, name, opts, comm=None)

Run the task and update DB state.

The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.

It is specific for spectra because the healpix_frame table has to be updated

Parameters:

• db (pipeline.db.DB) – The database.
• name (str) – the name of this task.
• opts (dict) – options to use for this task.
• comm (mpi4py.MPI.Comm) – optional MPI communicator.

Returns:

the number of processes that failed.

Return type:

int

class desispec.pipeline.tasks.starfit.TaskStarFit

Class containing the properties of one extraction task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

class desispec.pipeline.tasks.traceshift.TaskTraceShift

Class containing the properties of one trace shift task.

_deps(name, db, inputs)

See BaseTask.deps.

_option_list(name, opts)

Build the full list of options.

This includes appending the filenames and incorporating runtime options.

_paths(name)

See BaseTask.paths.

_run(name, opts, comm, db)

See BaseTask.run.

_run_cli(name, opts, procs, db)

See BaseTask.run_cli.

_run_defaults()

See BaseTask.run_defaults.

postprocessing(db, name, cur)

For successful runs, postprocessing on DB

Tools to regroup spectra in individual exposures by healpix on the sky

class desispec.pixgroup.FrameLite(wave, flux, ivar, mask, resolution_data, fibermap, header, scores=None)

Lightweight Frame object for regrouping

This is intended for I/O without the overheads of float32 -> float64 conversion, correcting endianness, etc.

classmethod read(filename)

Return FrameLite read from filename

class desispec.pixgroup.SpectraLite(bands, wave, flux, ivar, mask, resolution_data, fibermap, exp_fibermap=None, scores=None)

Lightweight spectra I/O object for regrouping

num_spectra()

Get the number of spectra contained in this group.

Returns (int):

Number of spectra contained in this group.

num_targets()

Get the number of distinct targets.

Returns (int):

Number of unique targets with spectra in this object.

classmethod read(filename)

Return a SpectraLite object read from filename

target_ids()

Return list of unique target IDs.

The target IDs are sorted by the order that they first appear.

Returns (array):

an array of integer target IDs.

write(filename, header=None)

Write this SpectraLite object to filename

desispec.pixgroup.add_missing_frames(frames)

Adds any missing frames with ivar=0 FrameLite objects with correct shape to match those that do exist.

Parameters:frames – dict of FrameLite objects, keyed by (night,expid,camera)

Modifies frames in-place.

Example: if frames has keys (2020,1,’b0’) and (2020,1,’r0’) but not (2020,1,’z0’), this will add a blank FrameLite object for z0.

The purpose of this is to facilitate frames2spectra, which needs something for every spectro camera for every exposure that is included.

desispec.pixgroup.fibermap2tilepix(fibermap, nside=64)

Maps fibermap to which healpix are covered by which petals

Parameters:fibermap – table with columns TARGET_RA, TARGET_DEC, PETAL_LOC

Options:

nside (int): nested healpix nside (must be power of 2)

Returns dict petalpix[petal] = list(healpix covered by that petal)

desispec.pixgroup.frames2spectra(frames, pix=None, nside=64)

Combine a dict of FrameLite into a SpectraLite for healpix pix

Parameters:frames – dict of FrameLight, keyed by (night, expid, camera)

Options:

pix: only include targets in this NESTED healpix pixel number nside: Healpix nside, must be power of 2

Returns:SpectraLite object with subset of spectra from frames that are in the requested healpix pixel pix

desispec.pixgroup.get_exp2healpix_map(survey=None, program=None, expfile=None, specprod_dir=None, strict=False, nights=None, expids=None)

Maps exposures to healpixels using preproc/NIGHT/EXPID/tilepix*.json files

Options:

survey (str): filter by this survey (main, sv3, sv1, …) program (str): filter by this FAPRGRM (dark, bright, backup, other) specprod_dir (str): override $DESI_SPECTRO_REDUX/$SPECPROD

TODO…

Returns table with columns NIGHT EXPID SPECTRO HEALPIX

desispec.pixgroup.update_frame_cache(frames, framekeys, specprod_dir=None)

Update a cache of FrameLite objects to match requested frameskeys

Parameters:

• frames – dict of FrameLite objects, keyed by (night, expid, camera)
• framekeys – list of desired (night, expid, camera)

Updates frames in-place

Notes

frames is dictionary, framekeys is list. When finished, the keys of frames match the entries in framekeys

Preprocess raw DESI exposures

desispec.preproc._background(image, header, patch_width=200, stitch_width=10, stitch=False)

determine background using a 2D median with square patches of width = width that are interpolated and optionnally try and match the level of amplifiers (does not subtract the background)

Parameters:

• image (ndarray) –
• image = (() –
• is used to read CCDSEC (header) –

Options:

patch_width (integer) size in pixels of the median square patches stitch_width (integer) width in pixels of amplifier edges to match level of amplifiers stitch : do match level of amplifiers

Returns background image with same shape as input image

desispec.preproc._clipped_std_bias(nsigma)

Returns the bias on the standard deviation of a sigma-clipped dataset

Divide by the returned bias to get a corrected value:

a = nsigma
bias = sqrt((integrate x^2 exp(-x^2/2), x=-a..a) / (integrate exp(-x^2/2), x=-a..a))
     = sqrt(1 - 2a exp(-a^2/2) / (sqrt(2pi) erf(a/sqrt(2))))

See http://www.wolframalpha.com/input/?i=(integrate+x%5E2+exp(-x%5E2%2F2),+x+%3D+-a+to+a)+%2F+(integrate+exp(-x%5E2%2F2),+x%3D-a+to+a)

desispec.preproc._global_background(image, patch_width=200)

determine background using a 2D median with square patches of width = width that are interpolated (does not subtract the background)

Parameters:

• image (ndarray) –
• image = (() –

Options:

patch_width (integer) size in pixels of the median square patches

Returns background image with same shape as input image

desispec.preproc._overscan(pix, nsigma=5, niter=3)

DEPRECATED: See calc_overscan

desispec.preproc._savgol_clipped(data, window=15, polyorder=5, niter=0, threshold=3.0)

Simple method to iteratively do a SavGol filter with rejection and replacing rejected pixels by nearest neighbors

Parameters:

• data (ndarray) –
• window (int) – Window parameter for savgol
• polyorder (int) –
• niter (int) –
• threshold (float) –

Returns:

desispec.preproc.calc_overscan(pix, nsigma=5, niter=3)

Calculates overscan, readnoise from overscan image pixels

Parameters:pix (ndarray) – overscan pixels from CCD image

Optional:

nsigma (float) : number of standard deviations for sigma clipping niter (int) : number of iterative refits

Returns:Mean, sigma-clipped value readnoise (float): Return type:overscan (float)

desispec.preproc.compute_background_between_fiber_blocks(image, xyset)

Computes CCD background between blocks of fibers

Parameters:

• image – desispec.image.Image object
• xyset – desispec.xytraceset.XYTraceSet object

Returns (model, qadict):

model: np.array of same shape as image.pix qadict: dictionary of keywords for QA with min/max per amplifier

Notes

Has hardcoded number of blocks and fibers and typical spacing between blocked tuned to DESI spectrographs.

desispec.preproc.compute_overscan_step(overscan_col, median_size=7, edge_margin=50)

Compute the overscan step score ‘OSTEP’ from an array of overscan values averaged per CCD row

Parameters:overscan_col – 1D numpy.array

Options:

median_size (int): window size for median pre-filter of overscan_col edge_margin (int): ignore this number of rows at the CCD edges

Returns:OSTEP value (float scalar)

desispec.preproc.get_amp_ids(header)

Return list of amp names based upon header keywords

desispec.preproc.get_calibration_image(cfinder, keyword, entry, header=None)

Reads a calibration file

Parameters:

• cfinder – None or CalibFinder object
• keyword – BIAS, MASK, or PIXFLAT
• entry – boolean or filename or image if entry==False return False if entry==True use calibration filename from calib. config and read it if entry==str use this for the filename if entry==image return input

Options:

header : if not None, update header[‘CAL…’] = calib provenance

Returns:2D numpy array with calibration image

For the case of keyword=’BIAS’, check for nightly bias before using default bias in $DESI_SPECTRO_CALIB

desispec.preproc.numba_mean(image_flux, image_ivar, x, hw=3)

Returns mean of pixels vs. row about x+-hw

Parameters:

• image_flux – 2D array of CCD image pixels
• image_ivar – 2D array of inverse variance of image_flux
• x – 1D array of x location per row, len(x) = image_flux.shape[0]

Options:

hw (int): halfwidth over which to average

Returns (flux, ivar) 1D arrays with weighted mean and inverse variance of pixels[i, int(x-hw):int(x+hw)+1] per row i

desispec.preproc.parse_sec_keyword(value)

parse keywords like BIASSECB=’[7:56,51:4146]’ into python slices

python and FITS have almost opposite conventions,

• FITS 1-indexed vs. python 0-indexed
• FITS upperlimit-inclusive vs. python upperlimit-exclusive
• FITS[x,y] vs. python[y,x]

i.e. BIASSEC2=’[7:56,51:4146]’ -> (slice(50,4146), slice(6,56))

desispec.preproc.preproc(rawimage, header, primary_header, bias=True, dark=True, pixflat=True, mask=True, bkgsub_dark=False, nocosmic=False, cosmics_nsig=6, cosmics_cfudge=3.0, cosmics_c2fudge=0.5, ccd_calibration_filename=None, nocrosstalk=False, nogain=False, overscan_per_row=False, use_overscan_row=False, use_savgol=None, nodarktrail=False, remove_scattered_light=False, psf_filename=None, bias_img=None, model_variance=False, no_traceshift=False, bkgsub_science=False)

preprocess image using metadata in header

image = ((rawimage-bias-overscan)*gain)/pixflat

Parameters:

• rawimage – 2D numpy array directly from raw data file
• header – dict-like metadata, e.g. from FITS header, with keywords CAMERA, BIASSECx, DATASECx, CCDSECx where x = A, B, C, D for each of the 4 amplifiers (also supports old naming convention 1, 2, 3, 4).
• primary_header – dict-like metadata fit keywords EXPTIME, DOSVER DATE-OBS is also required if bias, pixflat, or mask=True

Optional bias, pixflat, and mask can each be:

False: don’t apply that step True: use default calibration data for that night ndarray: use that array filename (str or unicode): read HDU 0 and use that

Optional overscan features:

overscan_per_row : bool, Subtract the overscan_col values

row by row from the data.

use_overscan_row : bool, Subtract off the overscan_row

from the data (default: False). Requires ORSEC in the Header

use_savgol : bool, Specify whether to use Savitsky-Golay filter for

the overscan. (default: False). Requires use_overscan_row=True to have any effect.

Optional variance model if model_variance=True Optional background subtraction with median filtering accross the whole CCD if bkgsub_dark=True Optional background subtraction with median filtering between groups of fiber traces if bkgsub_science=True

Optional disabling of cosmic ray rejection if nocosmic=True Optional disabling of dark trail correction if nodarktrail=True

Optional bias image (testing only) may be provided by bias_img=

Optional tuning of cosmic ray rejection parameters:

cosmics_nsig: number of sigma above background required cosmics_cfudge: number of sigma inconsistent with PSF required cosmics_c2fudge: fudge factor applied to PSF

Optional fit and subtraction of scattered light

Returns Image object with member variables:

pix : 2D preprocessed image in units of electrons per pixel ivar : 2D inverse variance of image mask : 2D mask of image (0=good) readnoise : 2D per-pixel readnoise of image meta : metadata dictionary TODO: define what keywords are included

preprocessing includes the following steps:

• bias image subtraction
• overscan subtraction (from BIASSEC* keyword defined regions)
• readnoise estimation (from BIASSEC* keyword defined regions)
• gain correction (from GAIN* keywords)
• pixel flat correction
• cosmic ray masking
• propagation of input known bad pixel mask
• inverse variance estimation

Notes:

The bias image is subtracted before any other calculation to remove any non-uniformities in the overscan regions prior to calculating overscan levels and readnoise.

The readnoise is an image not just one number per amp, because the pixflat image also affects the interpreted readnoise.

The inverse variance is estimated from the readnoise and the image itself, and thus is biased.

desispec.preproc.read_bias(filename=None, camera=None, dateobs=None)

Return calibration bias filename for camera on dateobs or night

Options:

filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

Notes

must provide filename, or both camera and dateobs

desispec.preproc.read_dark(filename=None, camera=None, dateobs=None, exptime=None)

Return dark current 2D image ( accounting for exposure time)

Parameters:

• filename – input filename to read
• camera – e.g. ‘b0’, ‘r1’, ‘z9’
• dateobs – DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’
• exptime – the exposure time of the image in seconds

Notes

must provide filename

desispec.preproc.read_mask(filename=None, camera=None, dateobs=None)

Read bad pixel mask image for camera on dateobs.

Options:

filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

Notes

must provide filename, or both camera and dateobs

desispec.preproc.read_pixflat(filename=None, camera=None, dateobs=None)

Read calibration pixflat image for camera on dateobs.

Options:

filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’

Notes

must provide filename, or both camera and dateobs

desispec.preproc.subtract_peramp_overscan(image, hdr)

Subtract per-amp overscan using BIASSEC* keywords

Parameters:

• image – 2D image array, modified in-place
• hdr – FITS header with BIASSEC[ABCD] or BIASSEC[1234] keywords

Note: currently used in desispec.ccdcalib.compute_bias_file to model bias image, but not preproc itself (which subtracts that bias, and has more complex support for row-by-row, col-overscan, etc.)

Module for generating QA HTML

desispec.qa.html.calib(qaprod_dir=None, specprod_dir=None)

Generate HTML to orgainze calib HTML

desispec.qa.html.calib_exp(night, expid, qaprod_dir=None)

Geneate HTML for calib exposure PNGs :param night: :param expid:

Returns:

desispec.qa.html.finish(f, body, links=None)

Fill in the HTML file and end it :param f: :type f: file :param body: :type body: str :param links: :type links: str, optional

desispec.qa.html.header(title)

Parameters:title (str, optional) –

desispec.qa.html.make_exposure(night, expid, qaprod_dir=None)

Generate HTML for exposure PNGs

Parameters:

• setup (str) –
• cbset (str) –
• det (int) –

Returns:

• links (str)
• body (str)

desispec.qa.html.make_exposures(qaprod_dir=None)

Generate HTML to organize exposure HTML

Returns:

• links (str)
• body (str)

desispec.qa.html.toplevel(qaprod_dir=None)

Generate HTML to top level QA Mainly generates the highest level HTML file which has links to the Exposure and Calib QA.

This also slurps any .png files in the top-level

Parameters:

• setup (str) –
• cbset (str) –
• det (int) –

Returns:

• links (str)
• body (str)

Classes to organize and execute QA for a DESI exposure

Classes to organize and execute QA for a DESI exposure

Classes to organize and execute QA for a DESI exposure

desispec.qa.qa_frame.qaframe_from_frame(frame_file, specprod_dir=None, make_plots=False, qaprod_dir=None, output_dir=None, clobber=True)

Generate a qaframe object from an input frame_file name (and night)

Write QA to disk Will also make plots if directed :param frame_file: str :param specprod_dir: str, optional :param qa_dir: str, optional – Location of QA :param make_plots: bool, optional :param output_dir: str, optional

Returns:

Class to organize QA for multiple exposures Likely to only be used as parent of QA_Night or QA_Prod

Class to organize QA for one night of DESI exposures

Module for QA plots

desispec.qa.qa_plots.brick_redrock(outfil, zf, qabrick)

QA plots for redrock fits

Parameters:

• outfil –
• qabrick –
• zf – ZfindBase object

Returns:

Stuff?

desispec.qa.qa_plots.exposure_fiberflat(channel, expid, metric, outfile=None)

Generate an Exposure level plot of a FiberFlat metric :param channel: str, e.g. ‘b’, ‘r’, ‘z’ :param expid: int :param metric: str, allowed entires are: [‘meanflux’]

Returns:

desispec.qa.qa_plots.exposure_fluxcalib(outfil, qa_data)

QA plots for Flux calibration in an Exposure

Parameters:

• outfil – str – Name of PDF file
• qa_data – dict – QA data, including that of the individual frames

desispec.qa.qa_plots.exposure_map(x, y, metric, mlbl=None, outfile=None, title=None, ax=None, fig=None, psz=9.0, cmap=None, vmnx=None)

Generic method used to generated Exposure level QA One channel at a time

Parameters:

• x – list or ndarray
• y – list or ndarray
• metric – list or ndarray
• mlbl – str, optional
• outfile – str, optional
• title – str, optional

desispec.qa.qa_plots.exposure_s2n(qa_exp, metric, outfile='exposure_s2n.png', verbose=True, specprod_dir=None)

Generate an Exposure level plot of a S/N metric :param qa_exp: QA_Exposure :param metric: str, allowed entires are: [‘resid’] :param specprod_dir: str, optional

Returns:

desispec.qa.qa_plots.frame_fiberflat(outfil, qaframe, frame, fiberflat)

QA plots for fiber flat

Parameters:

• outfil –
• qaframe –
• frame –
• fiberflat –
• clobber – bool, optional

Returns:

Stuff?

desispec.qa.qa_plots.frame_fluxcalib(outfil, qaframe, frame, fluxcalib)

QA plots for Flux calibration in a Frame

Parameters:

• outfil – str, name of output file
• qaframe – dict containing QA info
• frame – frame object containing extraction of standard stars
• fluxcalib – fluxcalib object containing flux calibration

Returns:

desispec.qa.qa_plots.frame_s2n(s2n_dict, outfile, rescut=0.2, verbose=True)

Plot S/N diagnostics for a given frame Replaces a previous-QL script

Parameters:

• s2n_dict (dict) – dictionary of qa outputs repackaged a bit
• outfile (str) – output png filename
• rescut (float, optional) – only plot residuals (+/-) less than rescut

desispec.qa.qa_plots.frame_skyres(outfil, frame, skymodel, qaframe, quick_look=False)

Generate QA plots and files for sky residuals of a given frame

Parameters:

• outfil (str) – Name of output file
• frame (Frame object) –
• skymodel (SkyModel object) –
• qaframe (QAFrame object) –

desispec.qa.qa_plots.get_channel_clrs()

Simple dict to organize styles for channels :returns: dict :rtype: channel_dict

desispec.qa.qa_plots.get_sty_otype()

Styles for plots

desispec.qa.qa_plots.prod_ZP(qa_prod, outfile=None, channels=('b', 'r', 'z'), xaxis='MJD')

Generate a plot summarizing the ZP for a production

Parameters:

• qa_prod – QA_Prod object
• outfile – str, optional Output file name
• channels – tuple, optional List of channels to show
• xaxis – str, optional Designate x-axis. Options are: MJD, expid, texp

Returns:

desispec.qa.qa_plots.prod_avg_s2n(qa_prod, outfile=None, optypes=['ELG'], xaxis='MJD', fiducials=None)

Generate a plot summarizing average S/N in a production for a few object types in a few cameras

Parameters:

• qa_prod – QA_Prod object
• outfile – str, optional Output file name
• optypes – list, optional List of fiducial objects to show Options are: ELG, LRG, QSO
• xaxis – str, optional Designate x-axis. Options are: MJD, expid, texp
• fiducials –

Returns:

desispec.qa.qa_plots.prod_channel_hist(qa_prod, qatype, metric, xlim=None, outfile=None, pp=None, close=True)

Generate a series of histrograms (one per channel)

Parameters:

• qa_prod – QA_Prod class
• qatype – str
• metric – str
• xlim – tuple, optional
• outfile – str, optional
• pp – PdfPages, optional
• close – bool, optional

Returns:

desispec.qa.qa_plots.prod_time_series(qa_multi, qatype, metric, outfile=None, close=True, pp=None, bright_dark=0, exposures=False, night=None, horiz_line=None)

Generate a time series plot for a production Can be MJD or Exposure number

Parameters:

• qa_multi – QA_Prod or QA_Night
• qatype – str
• metric – str
• outfile – str, optional
• close – bool, optional
• pp –
• bright_dark – int, optional; (flag: 0=all; 1=bright; 2=dark)
• night – str, optional Only used for the Title
• horiz_line – float, optional Draw a horizontal line at input value

desispec.qa.qa_plots.show_meta(ax, qaframe, qaflavor, outfil)

Show meta data on the figure

Parameters:

• ax – matplotlib.ax
• qaframe – QA_Frame
• qaflavor – str

Returns:

desispec.qa.qa_plots.skyline_resid(channel, sky_wave, sky_flux, sky_res, sky_ivar, outfile=None, pp=None, close=True, dpi=700)

QA plot for residuals on sky lines ala Julien Guy :param sky_wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close: :param nslices: :param dpi:

Returns:

desispec.qa.qa_plots.skysub_gauss(sky_wave, sky_flux, sky_res, sky_ivar, outfile=None, pp=None, close=True, binsz=0.1, dpi=700, nfbin=4)

Generate a plot examining the Gaussianity of the residuals Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param sky_ivar: :param outfile: :param pp: :param close:

Returns:

desispec.qa.qa_plots.skysub_resid_dual(sky_wave, sky_flux, sky_res, outfile=None, pp=None, close=True, nslices=20, dpi=700)

Generate a plot of sky subtraction residuals Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close:

Returns:

desispec.qa.qa_plots.skysub_resid_series(sky_dict, xtype, outfile=None, pp=None, close=True, nslices=20, dpi=700)

Generate a plot of sky subtraction residuals for a series of inputs Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close:

Returns:

This includes routines to make pdf plots on the qa outputs from quicklook.

For information on QA dictionaries used here as input, visit wiki page: https://desi.lbl.gov/trac/wiki/Pipeline/QuickLook/QuicklookQAOutputs/Science

desispec.qa.qa_plots_ql.plot_RMS(qa_dict, outfile, plotconf=None, hardplots=False)

Plot RMS

Parameters:

• qa_dict – dictionary of qa outputs from running qa_quicklook.Get_RMS
• outfile – Name of plot output file

desispec.qa.qa_plots_ql.plot_SNR(qa_dict, outfile, objlist, fitsnr, rescut=0.2, sigmacut=2.0, plotconf=None, hardplots=False)

Plot SNR

Parameters:

• qa_dict – dictionary of qa outputs from running qa_quicklook.Calculate_SNR
• outfile – output png file
• objlist – list of objtype for log(snr**2) vs. mag plots
• badfibs – list of fibers with infs or nans to remove for plotting
• fitsnr – list of snr vs. mag fitting coefficients # JXP – THIS IS NOT TRUE!!
• rescut – only plot residuals (+/-) less than rescut (default 0.2)
• sigmacut – only plot residuals (+/-) less than sigma cut (default 2.0)
• NOTE – rescut taken as default cut parameter

desispec.qa.qa_plots_ql.plot_XWSigma(qa_dict, outfile, plotconf=None, hardplots=False)

Plot XWSigma

Parameters:

• qa_dict – qa dictionary from countpix qa
• outfile – file of the plot

desispec.qa.qa_plots_ql.plot_bias_overscan(qa_dict, outfile, plotconf=None, hardplots=False)

Map of bias from overscan from 4 regions of CCD

Parameters:

• qa_dict – qa dictionary from bias_from_overscan qa
• outfile – pdf file of the plot

desispec.qa.qa_plots_ql.plot_countpix(qa_dict, outfile, plotconf=None, hardplots=False)

Plot pixel counts above some threshold

Parameters:

• qa_dict – qa dictionary from countpix qa
• outfile – pdf file of the plot

desispec.qa.qa_plots_ql.plot_countspectralbins(qa_dict, outfile, plotconf=None, hardplots=False)

Plot count spectral bins.

Parameters:

• qa_dict – dictionary of qa outputs from running qa_quicklook.CountSpectralBins
• outfile – Name of figure.

desispec.qa.qa_plots_ql.plot_lpolyhist(qa_dict, outfile, plotconf=None, hardplots=False)

Plot histogram for each legendre polynomial coefficient in WSIGMA array.

Parameters:

• qa_dict – Dictionary of qa outputs from running qa_quicklook.Check_Resolution
• outfile – Name of figure.

desispec.qa.qa_plots_ql.plot_sky_continuum(qa_dict, outfile, plotconf=None, hardplots=False)

Plot mean sky continuum from lower and higher wavelength range for each fiber and accross amps.

Parameters:

• qa_dict – dictionary from sky continuum QA
• outfile – pdf file to save the plot

desispec.qa.qa_plots_ql.plot_sky_peaks(qa_dict, outfile, plotconf=None, hardplots=False)

Plot rms of sky peaks for smy fibers across amps

Parameters:

• qa_dict – dictionary from sky peaks QA
• outfile – pdf file to save the plot

Class to organize QA for a full DESI production run

Monitoring algorithms for Quicklook pipeline

class desispec.qa.qa_quicklook.Bias_From_Overscan(name, config, logger=None)

class desispec.qa.qa_quicklook.Calc_XWSigma(name, config, logger=None)

class desispec.qa.qa_quicklook.Calculate_SNR(name, config, logger=None)

class desispec.qa.qa_quicklook.Check_FiberFlat(name, config, logger=None)

class desispec.qa.qa_quicklook.Check_HDUs(name, config, logger=None)

class desispec.qa.qa_quicklook.Check_Resolution(name, config, logger=None)

class desispec.qa.qa_quicklook.CountSpectralBins(name, config, logger=None)

class desispec.qa.qa_quicklook.Count_Pixels(name, config, logger=None)

class desispec.qa.qa_quicklook.Get_RMS(name, config, logger=None)

class desispec.qa.qa_quicklook.Integrate_Spec(name, config, logger=None)

class desispec.qa.qa_quicklook.Sky_Continuum(name, config, logger=None)

class desispec.qa.qa_quicklook.Sky_Peaks(name, config, logger=None)

class desispec.qa.qa_quicklook.Sky_Rband(name, config, logger=None)

class desispec.qa.qa_quicklook.Sky_Residual(name, config, logger=None)

Use offline sky_residual function to calculate sky residuals

class desispec.qa.qa_quicklook.Trace_Shifts(name, config, logger=None)

desispec.qa.qa_quicklook.get_frame(filetype, night, expid, camera, specdir)

Make frame object from file if in development mode

desispec.qa.qa_quicklook.get_image(filetype, night, expid, camera, specdir)

Make image object from file if in development mode

desispec.qa.qa_quicklook.get_inputs(*args, **kwargs)

Get inputs required for each QA

simple low level library functions for offline and online qas

desispec.qa.qalib.SN_ratio(flux, ivar)

SN Ratio median snr for the spectra, flux should be sky subtracted.

Parameters:

• flux (array) – 2d [nspec,nwave] the signal (typically for spectra, this comes from frame object
• ivar (array) – 2d [nspec,nwave] corresponding inverse variance

Returns:

1d [nspec]

Return type:

medsnr (array)

desispec.qa.qalib.SignalVsNoise(frame, params, fidboundary=None)

Signal vs. Noise

Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera.

Parameters:

• flux (array) – 2d [nspec,nwave] the signal (typically for spectra, this comes from frame object
• ivar (array) – 2d [nspec,nwave] corresponding inverse variance
• fidboundary – list of slices indicating where to select in fiber and wavelength directions for each amp (output of slice_fidboundary function)

desispec.qa.qalib._get_mags(frame)

Extract frame.fibermap fluxes into mags depending upon camera

Parameters:frame – Frame object

Returns array of magnitudes, using 99.0 when flux<0

b camera frames return g-band magnitudes; r camera -> r-mags; z camera -> z-mags

desispec.qa.qalib.ampregion(image)

Get the pixel boundary regions for amps

Parameters:image – desispec.image.Image object

desispec.qa.qalib.continuum(wave, flux, wmin=None, wmax=None)

Find the median continuum of the spectrum inside a wavelength region.

Parameters:

• wave – 1d wavelength array
• flux – 1d counts/flux array
• and wmax (wmin) – region to consider for the continuum

desispec.qa.qalib.countbins(flux, threshold=0)

Count the number of bins above a given threshold on each fiber

Parameters:

• flux – 2d (nspec,nwave)
• threshold – threshold counts

desispec.qa.qalib.countpix(image, nsig=None)

Count the pixels above a given threshold in units of sigma.

Parameters:

• image – 2d image array
• nsig – threshold in units of sigma, e.g 2 for 2 sigma

desispec.qa.qalib.fiducialregion(frame, psf)

Get the fiducial amplifier regions on the CCD pixel to fiber by wavelength space

Parameters:

• frame – desispec.frame.Frame object
• psf – desispec.psf.PSF like object

desispec.qa.qalib.gauss(x, a, mu, sigma)

Gaussian fit of input data

desispec.qa.qalib.getrms(image)

Calculate the rms of the pixel values)

Parameters:image – 2d array

desispec.qa.qalib.integrate_spec(wave, flux)

Calculate the integral of the spectrum in the given range using trapezoidal integration

Note: limits of integration are min and max values of wavelength

Parameters:

• wave – 1d wavelength array
• flux – 1d flux array

desispec.qa.qalib.orig_SNRFit(frame, night, camera, expid, params, fidboundary=None, offline=False)

Signal vs. Noise With fitting

Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera. then fit the log(snr)=a+b*mag or log(snr)=poly(mag)

see http://arXiv.org/abs/0706.1062v2 for proper fitting of power-law distributions it is not implemented here!

qadict has the following data model

“MAGNITUDES” : ndarray - Depends on camera (DECAM_G, DECAM_R, DECAM_Z) “MEDIAN_SNR” : ndarray (nfiber) “NUM_NEGATIVE_SNR” : int “SNR_MAG_TGT” “FITCOEFF_TGT” : list “SNR_RESID” : list, can be trimmed down during the fitting “FIDSNR_TGT” “RA” : ndarray (nfiber) “DEC” : ndarray (nfiber) “OBJLIST” : list - Save a copy to make sense of the list order later “EXPTIME” : float “FIT_FILTER” : str “r2” : float - Fitting parameter

Parameters:

• frame – desispec.Frame object
• night –
• camera –
• expid – int
• params – parameters dictionary
• { – “Func”: “linear”, # Fit function type one of [“linear”,”poly”,”astro”] “FIDMAG”: 22.0, # magnitude to evaluate the fit “Filter”:”DECAM_R”, #filter name
• } –
• fidboundary – list of slices indicating where to select in fiber and wavelength directions for each amp (output of slice_fidboundary function)
• offline – bool, optional If True, save things differently for offline

Returns:

dict

Return type:

qadict

desispec.qa.qalib.s2n_flux_astro(flux, A, B)

Function for a normalized (by texp**1/2) curve to flux vs S/N

Parameters:

• flux (float or np.ndarray) – Flux value(s)
• A (float) – Scale coefficient
• B (float) – Offset coefficient

Returns:

S/N at the input flux

desispec.qa.qalib.s2n_funcs(exptime=None)

Functions for fitting S/N

Parameters:exptime – float, optional Returns:dict Return type:funcMap

desispec.qa.qalib.s2nfit(frame, camera, params)

Signal vs. Noise With fitting

Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera. then fit snr=A*mag/sqrt(A*mag+B)

see http://arXiv.org/abs/0706.1062v2 for proper fitting of power-law distributions it is not implemented here!

Instead we use scipy.optimize.curve_fit

Parameters:

• frame – desispec.Frame object
• camera – str, name of the camera
• params – parameters dictionary for S/N

Returns:

dict

MEDIAN_SNR (ndarray, nfiber): Median S/N of light in each fiber FIT_FILTER (str): Filter used for the fluxes EXPTIME (float): Exposure time XXX_FIBERID (list): Fibers matching ELG, LRG, BGS, etc. SNR_MAG_TGT (list): List of lists with S/N and mag of ELG, LRG, BGS, etc. FITCOEFF_TGT (list): List of fitted coefficients. Junk fits have np.nan OBJLIST (list): List of object types analyzed (1 or more fiber)

Return type:

qadict

desispec.qa.qalib.sky_continuum(frame, wrange1, wrange2)

QA Algorithm for sky continuum.

To be called from desispec.sky.qa_skysub and desispec.qa.qa_quicklook.Sky_Continuum.run_qa

Parameters:

• frame –
• wrange1 –
• wrange2 –

Returns:

skyfiber, contfiberlow, contfiberhigh, meancontfiber, skycont

desispec.qa.qalib.sky_resid(param, frame, skymodel, quick_look=False)

QA Algorithm for sky residual To be called from desispec.sky.qa_skysub and desispec.qa.qa_quicklook.Sky_residual.run_qa :param param: dict of QA parameters :param frame: desispec.Frame object after sky subtraction :param skymodel: desispec.SkyModel object

Returns a qa dictionary for sky resid

desispec.qa.qalib.slice_fidboundary(frame, leftmax, rightmin, bottommax, topmin)

leftmax,rightmin,bottommax,topmin - Indices in spec-wavelength space for different amps (e.g output from fiducialregion function) #- This could be merged to fiducialregion function

Returns (list):

list of tuples of slices for spec- wavelength boundary for the amps.

Module for QA support

desispec.qa.utils.get_skyres(cframes, sub_sky=False, flatten=True)

Parameters:

• cframes – str or list Single cframe or a list of them
• sub_sky – bool, optional Subtract the sky? This should probably not be done
• flatten – bool, optional Return a flat, 1D array for each variable
• combine – bool, optional combine the individual sky fibers? Median ‘smash’

Returns:

ndarray flux : ndarray res : ndarray ivar : ndarray

Return type:

wave

desispec.qproc.io

I/O routines for qproc objects

desispec.qproc.io.read_qframe(filename, nspec=None, skip_resolution=False)

Reads a frame fits file and returns its data.

Parameters:

• filename – path to a file
• skip_resolution – bool, option Speed up read time (>5x) by avoiding the Resolution matrix

Returns:

desispec.Frame object with attributes wave, flux, ivar, etc.

desispec.qproc.io.write_qframe(outfile, qframe, header=None, fibermap=None, units=None)

Write a frame fits file and returns path to file written.

Parameters:

• outfile – full path to output file, or tuple (night, expid, channel)
• qframe – desispec.qproc.QFrame object with wave, flux, ivar…

Optional:

header: astropy.io.fits.Header or dict to override frame.header fibermap: table to store as FIBERMAP HDU

Returns:full filepath of output file that was written

Note

to create a QFrame object to pass into write_qframe, qframe = QFrame(wave, flux, ivar)

desispec.qproc.qextract

desispec.qproc.qextract.qproc_boxcar_extraction(xytraceset, image, fibers=None, width=7, fibermap=None, save_sigma=True)

Fast boxcar extraction of spectra from a preprocessed image and a trace set

Parameters:

• xytraceset – DESI XYTraceSet object
• image – DESI preprocessed Image object

Optional:

fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : extraction boxcar width, default is 7 fibermap : table

Returns:QFrame object

desispec.qproc.qfiberflat.qproc_apply_fiberflat(qframe, fiberflat, return_flat=False)

Apply a fiber flat to a qframe.

Inputs:

qframe: desispec.qproc.qframe.QFrame object which will be modified fiberflat: desispec.fiberflat.FiberFlat object with the flat to apply

Optional:

return_flat : if True, returns the flat field that has been applied

Returns nothing or the flat that has been applied.

desispec.qproc.qfiberflat.qproc_compute_fiberflat(qframe, niter_meanspec=4, nsig_clipping=3.0, spline_res_clipping=20.0, spline_res_flat=5.0)

Fast estimation of fiberflat

desispec.qproc.qframe

Lightweight wrapper class for spectra extract with qextract, row-by-row extraction (boxcar or profile)

desispec.qproc.qsky.qproc_sky_subtraction(qframe, return_skymodel=False)

Fast sky subtraction directly applied to the input qframe.

Args:

qframe : DESI QFrame object

Optional:

return_skymodel returns the skymodel as an array of same shape as qframe.flux

desispec.quicklook

desispec.quicklook.arcprocess.process_arc(frame, linelist=None, npoly=2, nbins=2, domain=None)

frame: desispec.frame.Frame object, preumably resolution not evaluated. linelist: line list to fit npoly: polynomial order for sigma expansion nbins: no of bins for the half of the fitting window return: coefficients of the polynomial expansion

desispec.quicklook.arcprocess.sigmas_from_arc(wave, flux, ivar, linelist, n=2)

Gaussian fitting of listed arc lines and return corresponding sigmas in pixel units Args: linelist: list of lines (A) for which fit is to be done n: fit region half width (in bin units): n=2 bins => (2*n+1)=5 bins fitting window.

desispec.quicklook.arcprocess.write_psffile(infile, wcoeffs, wcoeffs_wavemin, wcoeffs_wavemax, outfile, wavestepsize=None)

extract psf file, add wcoeffs, and make a new psf file preserving the traces etc. psf module will load this

desispec.quicklook.palib Low level functions to be from top level PAs

desispec.quicklook.palib.apply_flux_calibration(frame, fluxcalib)

Apply flux calibration to sky subtracted qframe Use offline algorithm, but assume qframe object is input and that it is on native ccd wavelength grid Calibration vector is resampled to frame wavelength grid

frame: QFrame object fluxcalib: FluxCalib object

Modifies frame.flux and frame.ivar

desispec.quicklook.palib.get_resolution(wave, nspec, tset, usesigma=False)

Calculates approximate resolution values at given wavelengths in the format that can directly feed resolution data of desispec.frame.Frame object.

wave: wavelength array nsepc: no of spectra (int) tset: desispec.xytraceset like object usesigma: allows to use sigma from psf file for resolution computation.

returns : resolution data (nspec,nband,nwave); nband = 1 for usesigma = False, otherwise nband=21

desispec.quicklook.palib.project(x1, x2)

return a projection matrix so that arrays are related by linear interpolation x1: Array with one binning x2: new binning

Return Pr: x1= Pr.dot(x2) in the overlap region

desispec.quicklook.palib.resample_spec(wave, flux, outwave, ivar=None)

rebinning conserving S/N Algorithm is based on http://www.ast.cam.ac.uk/%7Erfc/vpfit10.2.pdf Appendix: B.1

Args: wave : original wavelength array (expected (but not limited) to be native CCD pixel wavelength grid outwave: new wavelength array: expected (but not limited) to be uniform binning flux : df/dx (Flux per A) sampled at x ivar : ivar in original binning. If not None, ivar in new binning is returned.

Note: Full resolution computation for resampling is expensive for quicklook.

desispec.interpolation.resample_flux using weights by ivar does not conserve total S/N. Tests with arc lines show much narrow spectral profile, thus not giving realistic psf resolutions This algorithm gives the same resolution as obtained for native CCD binning, i.e, resampling has insignificant effect. Details,plots in the arc processing note.

class desispec.quicklook.pas.PipelineAlg(name, inptype, outtype, config, logger=None)

Simple base class for Pipeline algorithms

get_default_config()

return a dictionary of 3-tuples, field 0 is the name of the parameter field 1 is the default value of the parameter field 2 is the comment for human readeable format. Field 2 can be used for QLF to dynamically setup the display

Pipeline Preprocessing algorithms for Quicklook

class desispec.quicklook.procalgs.ApplyFiberFlat(name, config, logger=None)

PA to Apply the fiberflat field to the given frame

class desispec.quicklook.procalgs.ApplyFiberFlat_QL(name, config, logger=None)

PA to Apply the fiberflat field (QL) to the given frame

class desispec.quicklook.procalgs.ApplyFiberFlat_QP(name, config, logger=None)

PA to Apply the fiberflat field (QP) to the given qframe

class desispec.quicklook.procalgs.ApplyFluxCalibration(name, config, logger=None)

PA to apply flux calibration to the given sframe

class desispec.quicklook.procalgs.BoxcarExtract(name, config, logger=None)

do_boxcar(tset, outwave, boxwidth=2.5, nspec=500, maskFile=None, usesigma=False, quick_resolution=False)

Extracts spectra row by row, given the centroids

Parameters:

• image – desispec.image object
• tset – desispec.xytraceset like object
• outwave – wavelength array for the final spectra output
• boxwidth – HW box size in pixels
• usesigma – if True, use sigma from psf file (ysigma) to calculate resolution data.
• quick_resolution – whether to calculate the resolution matrix or use QuickResolution object

Returns flux, ivar, resolution

get_default_config()

return a dictionary of 3-tuples, field 0 is the name of the parameter field 1 is the default value of the parameter field 2 is the comment for human readeable format. Field 2 can be used for QLF to dynamically setup the display

class desispec.quicklook.procalgs.ComputeFiberflat(name, config, logger=None)

PA to compute fiberflat field correction from a DESI continuum lamp frame

class desispec.quicklook.procalgs.ComputeFiberflat_QL(name, config, logger=None)

PA to compute fiberflat field correction from a DESI continuum lamp frame

class desispec.quicklook.procalgs.ComputeFiberflat_QP(name, config, logger=None)

class desispec.quicklook.procalgs.ComputeSky(name, config, logger=None)

PA to compute sky model from a DESI frame

class desispec.quicklook.procalgs.ComputeSky_QL(name, config, logger=None)

PA to compute sky model from a DESI frame

class desispec.quicklook.procalgs.Extract_QP(name, config, logger=None)

get_default_config()

return a dictionary of 3-tuples, field 0 is the name of the parameter field 1 is the default value of the parameter field 2 is the comment for human readeable format. Field 2 can be used for QLF to dynamically setup the display

class desispec.quicklook.procalgs.Extraction_2d(name, config, logger=None)

Offline 2D extraction for offline QuickLook

class desispec.quicklook.procalgs.Flexure(name, config, logger=None)

Use desi_compute_trace_shifts to output modified psf file

class desispec.quicklook.procalgs.Initialize(name, config, logger=None)

This PA takes information from the fibermap and raw header and adds it to the general info section of the merged dictionary

class desispec.quicklook.procalgs.Preproc(name, config, logger=None)

class desispec.quicklook.procalgs.ResolutionFit(name, config, logger=None)

Fitting of Arc lines on extracted arc spectra, polynomial expansion of the fitted sigmas, and updating the coefficients to the new traceset file

class desispec.quicklook.procalgs.SkySub(name, config, logger=None)

class desispec.quicklook.procalgs.SkySub_QL(name, config, logger=None)

This is for QL Sky subtraction. The input frame object should be fiber flat corrected. Unlike offline, if no skymodel file is given as input, a sky compute method is called to create a skymodel object and then subtraction is performed. Outputing that skymodel to a file is optional and can be configured.

class desispec.quicklook.procalgs.SkySub_QP(name, config, logger=None)

Sky subtraction. The input frame object should be fiber flat corrected. No sky model is saved for now

class desispec.quicklook.qas.MonitoringAlg(name, inptype, config, logger=None)

Simple base class for monitoring algorithms

class desispec.quicklook.qas.QASeverity

An enumeration.

boxcar extraction for Spectra from Desi Image

desispec.quicklook.qlboxcar.do_boxcar(image, tset, outwave, boxwidth=2.5, nspec=500, maskFile=None, usesigma=False, quick_resolution=False)

Extracts spectra row by row, given the centroids

Parameters:

• image – desispec.image object
• tset – desispec.xytraceset like object
• outwave – wavelength array for the final spectra output
• boxwidth – HW box size in pixels
• usesigma – if True, use sigma from psf file (ysigma) to calculate resolution data.
• quick_resolution – whether to calculate the resolution matrix or use QuickResolution object

Returns flux, ivar, resolution

class desispec.quicklook.qlconfig.Config(configfile, night, camera, expid, singqa, amps=True, rawdata_dir=None, specprod_dir=None, outdir=None, qlf=False, psfid=None, flatid=None, templateid=None, templatenight=None, qlplots=False, store_res=None)

A class to generate Quicklook configurations for a given desi exposure. expand_config will expand out to full format as needed by quicklook.setup

dump_pa(paname)

dump the PA outputs to respective files. This has to be updated for fframe and sframe files as QL anticipates for dumpintermediate case.

dump_qa()

yaml outputfile for the set of qas for a given pa Name and default locations of files are handled by desispec.io.meta.findfile

expand_config()

config: desispec.quicklook.qlconfig.Config object

io_qa(qaname)

Specify the filenames: json and png for the given qa output

io_qa_pa(paname)

Specify the filenames: json and png of the pa level qa files”

paargs

Many arguments for the PAs are taken default. Some of these may need to be variable psfspfile is for offline extraction case

palist

palist for this config see :class: Palist for details.

qalist

qalist for the given palist

class desispec.quicklook.qlconfig.Palist(thislist=None, algorithms=None)

Generate PA list and QA list for the Quicklook Pipeline for the given exposure

desispec.quicklook.qlconfig.check_config(outconfig, singqa)

Given the expanded config, check for all possible file existence etc….

exception desispec.quicklook.qlexceptions.ParameterException(value)

class desispec.quicklook.qllogger.QLLogger(name=None, loglevel=20)

Simple logger class using logging

Generic plotting algorithms for QuickLook QAs

desispec.quicklook.ql_plotlib.ql_2dplot(ax, xvals, yvals, plottitle, xtitle, ytitle, xlim=None, ylim=None)

Make 2d plot of specific metrics provided in configuration file

Parameters:

• ax – matplotlib subplot
• xvals – QA metric to be plotted along the xaxis
• yvals – QA metric to be plotted along the yaxis
• plottitle – plot title from configuration file
• xtitle – x axis label
• ytitle – y axis label

Optional:

xlim: list containing x range (i.e. [x_lo,x_hi]) ylim: list containing y range (i.e. [y_lo,y_hi])

Returns:matplotlib sublot containing plotted metrics

desispec.quicklook.ql_plotlib.ql_3dplot(ax, xvals, yvals, zvals, plottitle, xtitle, ytitle, zlim=None, heatmap=None)

Make 3d scatter plot of specific metrics provided in configuration file

Parameters:

• ax – matplotlib subplot
• xvals – QA metric to be plotted along the xaxis
• yvals – QA metric to be plotted along the yaxis
• zvals – QA metric to be plotted
• plottitle – plot title from configuration file
• xtitle – x axis label
• ytitle – y axis label

Optional:

zlim: list containing scatter plot range (i.e. [z_lo,z_hi])

Returns:matplotlib sublot containing plotted metrics

desispec.quicklook.ql_plotlib.ql_patchplot(ax, vals, plottitle, grid, heatmap=None)

Make patch plot of specific metrics provided in configuration file

Parameters:

• ax – matplotlib subplot
• vals – QA metric to be plotted
• plottitle – plot title from configuration file
• grid – shape of patch plot

Optional:

heat: specify color of heatmap (must conform to matplotlib)

Returns:matplotlib sublot containing plotted metrics

desispec.quicklook.ql_plotlib.ql_qaplot(fig, plotconf, qadict, camera, expid, outfile)

Get plotting configuration info and setup plots

Parameters:

• fig – matplotlib figure
• plotconf – list of config info for each plot
• qadict – QA metrics dictionary
• expid (camera,) – to be used in output png title
• outfile – output png file

Returns:

png file containing all desired plots

Given a psf output file eg. output from bootcalib.write_psf or desimodel/data/specpsf/PSF* files this defines an interface that other codes can use the trace and wavelength solutions

Mostly making parallel to specter.psf.PSF baseclass and inheriting as needed, but only xtrace, ytrace and wavelength solution available for this case. No resolution information yet.

class desispec.quicklook.qlpsf.PSF(filename)

Base class for 2D psf

angstroms_per_pixel(ispec, wavelength)

Return CCD pixel width in Angstroms for spectrum ispec at given wavlength(s). Wavelength may be scalar or array.

wavelength(ispec=None, y=None)

returns wavelength evaluated at y

x(ispec=None, wavelength=None)

returns CCD x centroids for the spectra ispec can be None, scalar or a vector wavelength can be None or a vector

y(ispec=None, wavelength=None)

returns CCD y centroids for the spectra ispec can be None, scalar or a vector wavelength can be a vector but not allowing None #- similar as in specter.psf.PSF.y

desispec.quicklook.qlresolution

Quicklook version of resolution object that can calculate resolution efficiently from psf information

Author: Sami Kama

class desispec.quicklook.qlresolution.QuickResolution(mu=None, sigma=None, wdict=None, waves=None, ndiag=9)

Quicklook version of the resolution mimicking desispec.resolution.Resolution with some reduction in dimentionality. Contains code from Resolution implementation Note that this is similar to desispec.resolution.Resolution, though faster and differing in implementation details that should be cross checked before merging these or replacing one with the other

desispec.quickfiberflat

Here will be the fiberflat routines specific to quicklook.

7. Dhungana, 2016

desispec.quicklook.quickfiberflat.apply_fiberflat(frame, fiberflat)

Args: frame: desispec.frame.Frame object

fiberflat: desispec.fiberflat.Fiberflat object

desispec.quicklook.quickfiberflat.compute_fiberflat()

computes fiberflat: A boss like algorithm writing in progress and will fit in here.

Args:

desispec.quicklook.quicklook.get_chan_spec_exp(inpname, camera=None)

Get channel, spectrograph and expid from the filename itself

Parameters:

• inpname – can be raw or pix, or frame etc filename
• camera – is required for raw case, eg, r0, b5, z8 irrelevant for others

desispec.quicklook.quicklook.mapkeywords(kw, kwmap)

Maps the keyword in the configuration to the corresponding object returned by the desispec.io module. e.g Bias Image file is mapped to biasimage object… for the same keyword “BiasImage”

desispec.quicklook.quicklook.runpipeline(pl, convdict, conf)

Runs the quicklook pipeline as configured

Parameters:

• pl – is a list of [pa,qas] where pa is a pipeline step and qas the corresponding qas for that pa
• convdict – converted dictionary e.g : conf[“IMAGE”] is the real psf file but convdict[“IMAGE”] is like desispec.image.Image object and so on. details in setup_pipeline method below for examples.
• conf – a configured dictionary, read from the configuration yaml file. e.g: conf=configdict=yaml.safe_load(open(‘configfile.yaml’,’rb’))

desispec.quicklook.quicklook.setup_pipeline(config)

Given a configuration from QLF, this sets up a pipeline [pa,qa] and also returns a conversion dictionary from the configuration dictionary so that Pipeline steps (PA) can take them. This is required for runpipeline.

desispec.quicklook.quicksky

Here will be the sky computing and sky subtraction routines for QL

desispec.quicklook.quicksky.compute_sky(fframe, fibermap=None, nsig_clipping=4.0, apply_resolution=False)

Adding in the offline algorithm here to be able to apply resolution for sky compute. We will update this here as needed for quicklook. The original weighted sky compute still is the default.

Args: fframe: fiberflat fielded frame object

fibermap: fibermap object apply_resolution: if True, uses the resolution in the frame object to evaluate sky allowing fiber to fiber variation of resolution.

desispec.quicklook.quicksky.subtract_sky(fframe, skymodel)

skymodel: skymodel object. fframe: frame object to do the sky subtraction, should be already fiber flat fielded need same number of fibers and same wavelength grid

desispec.resolution

Standardized handling of sparse wavelength resolution matrices.

Use python -m desispec.resolution to run unit tests.

class desispec.resolution.Resolution(data, offsets=None)

Canonical representation of a resolution matrix.

Inherits all of the method of scipy.sparse.dia_matrix, including todense() for converting to a dense 2D numpy array of matrix elements, most of which will be zero (so you generally want to avoid this).

Parameters:data – Must be in one of the following formats listed below.

Options:

offsets: list of diagonals that the data represents. Only used if

data is a 2D dense array.

Raises:ValueError – Invalid input for initializing a sparse resolution matrix.

Data formats:

1. a scipy.sparse matrix in DIA format with the required diagonals (but not necessarily in the canoncial order);
2. a 2D square numpy arrray (i.e., a dense matrix) whose non-zero values beyond default_ndiag will be silently dropped; or
3. a 2D numpy array[ndiag, nwave] that encodes the sparse diagonal values in the same format as scipy.sparse.dia_matrix.data .

The last format is the one used to store resolution matrices in FITS files.

to_fits_array()

Convert to an array of sparse diagonal values.

This is the format used to store resolution matrices in FITS files. Note that some values in the returned rectangular array do not correspond to actual matrix elements since the diagonals get smaller as you move away from the central diagonal. As long as you treat this array as an opaque representation for FITS I/O, you don’t care about this. To actually use the matrix, create a Resolution object from the fits array first.

Returns:

An array of (num_diagonals,nbins) sparse matrix

element values close to the diagonal.

Return type:numpy.ndarray

desispec.resolution._gauss_pix(x, mean=0.0, sigma=1.0)

Utility function to integrate Gaussian density within pixels

Parameters:

• x (1D array) – pixel centers
• mean (float) – mean of Gaussian
• sigma (float) – sigma of Gaussian

Returns:

array of integals of the Gaussian density in the pixels.

Note

All pixels must be the same size

desispec.resolution._sort_and_symmeterize(data, offsets)

Sort data,offsets and pad to ensure equal number of upper/lower diagonals

Parameters:

• data – 2D array of diagonals, following scipy.sparse.dia_matrix.data ordering
• offsets – 1D array of offsets; must be complete from min to max but doesn’t have to be sorted

Returns:

fulldata, fulloffsets

desispec.scripts

Main functions and commandline parsing.

desispec.scripts.average_fiberflat

desispec.scripts.bootcalib

Utility functions to perform a quick calibration of DESI data

TODO: 1. Expand to r, i cameras 2. QA plots 3. Test with CR data

Extract spectra from DESI pre-processed raw data

desispec.scripts.extract._extract_and_save(img, psf, bspecmin, bnspec, specmin, wave, raw_wave, fibers, fibermap, outbundle, outmodel, bundlesize, args, log)

Performs the main extraction and saving of extracted frames found in the body of the main loop. Refactored to be callable by both MPI and non-MPI versions of the code. This should be viewed as a shorthand for the following commands.

desispec.scripts.extract.barycentric_correction_multiplicative_factor(header)

Returns mult. barycentric correction factor using coords in header

header must contrain MJD or MJD-OBS; and TARGTRA,TARGTDEC or SKYRA,SKYDEC or TELRA,TELDEC or RA,DEC

desispec.scripts.extract.gpu_specter_check_input_options(args)

Perform pre-flight checks on input options

returns ok(True/False), message

desispec.scripts.fiberflat

Utility functions to compute a fiber flat correction and apply it We try to keep all the (fits) io separated.

Regroup spectra by healpix

exspec extracts individual bundles of spectra with one bundle per output file. This script merges them back together into a single file combining all bundles.

This workflow is hacky. Release early, release often, but also refactor often.

Stephen Bailey, LBL March 2014

desispec.scripts.night

Automated nightly processing.

Interactive control of the pipeline

Run one or more pipeline tasks.

desispec.scripts.preproc

Command line wrappers for pre-processing a DESI raw exposure

desispec.scripts.preproc._preproc_file_kwargs_wrapper(opts)

This function just unpacks opts dict for preproc_file so that it can be used with multiprocessing.Pool.map

desispec.scripts.preproc.preproc_file(infile, camera, outfile=None, outdir=None, fibermap=None, zero_masked=False, **preproc_opts)

Preprocess a single camera from a single input file

Parameters:

• infile – input raw data file
• camera – camera, e.g. ‘b0’, ‘r1’, ‘z9’

Options:

outfile: output preprocessed image file to write outdir: output directory; derive filename from infile NIGHT and EXPID fibermap: fibermap filename to include in output zero_masked (bool): set masked pixels to 0 preproc_opts: dictionary to pass to preproc

Returns error code (1=error, 0=success) but will not raise exception if there is an I/O or preprocessing failure (allows other parallel procs to proceed).

Note: either outfile or outdir must be provided

This script processes an exposure by applying fiberflat, sky subtraction, spectro-photometric calibration depending on input. Optionally, includes tsnr in the scores hdu.

desispec.scripts.quicklook

Command line wrapper for running a QL pipeline

QuickLook team @Southern Methodist University (SMU) First version Spring 2016 Latest revision July 2018

Running QuickLook:

desi_quicklook -i qlconfig_science.yaml -n 20191001 -c r0 -e 3577

This requires having necessary input files and setting the following environment variables:

QL_SPEC_DATA: directory containing raw/fibermap files (full path: $QL_SPEC_DATA/night/expid) QL_SPEC_REDUX: directory for QL output (full path: $QL_SPEC_REDUX/exposures/night/expid) DESI_CALIBRATION_DATA: directory containing calibration files

Necessary Quicklook command line arguments:

-i,–config_file : path to QL configuration file -n,–night : night to be processed -c,–camera : camera to be processed -e,–expid : exposure ID to be processed

Optional QuickLook arguments:

–rawdata_dir : directory containing raw/fibermap files (overrides $QL_SPEC_DATA) –specprod_dir : directory for QL output (overrides $QL_SPEC_REDUX)

Plotting options:

-p (including path to plotting configuration file) : generate configured plots -p (only using -p with no configuration file) : generate QL hardcoded plots

desispec.scripts.quicklook.parse()

Should have either a pre existing config file, or need to generate one using config module

This script finds cosmics in a pre-processed image and write the result in the mask extension of an output image (output can be same as input).

Run PSF estimation.

desispec.scripts.specex.compatible(head1, head2)

Return bool for whether two FITS headers are compatible for merging PSFs

desispec.scripts.specex.mean_psf(inputs, output)

Average multiple input PSF files into an output PSF file

Parameters:

• inputs – list of input PSF files
• output – output filename

desispec.scripts.specex.merge_psf(inputs, output)

Merge individual per-bundle PSF files into full PSF

Parameters:

• inputs – list of input PSF filenames
• output – output filename

desispec.scripts.specex.run(comm, cmds, cameras)

Run PSF fits with specex on a set of ccd images in parallel using the run method of the desispec.workflow.schedule.Schedule (Schedule) class.

Parameters:

• comm – MPI communicator containing all processes available for work and scheduling (usually MPI_COMM_WORLD); at least 21 processes should be available, one for scheduling and (group_size=) 20 to fit all bundles for a given ccd image. Otherwise there is no constraint on the number of ranks available, but (comm.Get_size()-1)%group_size will be unused, since every job is assigned exactly group_size=20 ranks. The variable group_size is set at the number of bundles on a ccd, and there is currently no support for any other number, due to the way merging of bundles is currently done.
• cmds – dictionary keyed by a camera string (e.g. ‘b0’, ‘r1’, …) with values being the ‘desi_compute_psf …’ string that one would run on the command line.
• cameras – list of camera strings identifying the entries in cmds to be run as jobs in parallel jobs, one entry per ccd image to be fit; entries not in the dictionary will be logged as an error while still continuing with the others and not crashing.

The function first defines the procedure to call specex for a given ccd image with the “fitbundles” inline function, passes the fitbundles function to the Schedule initialization method, and then calls the run method of the Schedule class to call fitbundles len(cameras) times, each with group_size = 20 processes.

Compute some information scores on spectra in frames

Get the normalized best template to do flux calibration.

desispec.scripts.stdstars.get_gaia_ab_correction()

Get the dictionary with corrections from AB magnitudes to Vega magnitudes (as the official gaia catalog is in vega)

desispec.scripts.stdstars.get_magnitude(stdwave, model, model_filters, cur_filt)

Obtain magnitude for a filter taking into account the ab/vega correction if needed. Wwe assume the flux is in units of 1e-17 erg/s/cm^2/A

desispec.scripts.stdstars.main(args, comm=None)

finds the best models of all standard stars in the frame and normlize the model flux. Output is written to a file and will be called for calibration.

desispec.scripts.trace_shifts

Update healpix-grouped spectra from a set of input cframe files

desispec.sky

Utility functions to compute a sky model and subtract it.

desispec.sky._model_variance(frame, cskyflux, cskyivar, skyfibers)

look at chi2 per wavelength and increase sky variance to reach chi2/ndf=1

desispec.sky.calculate_throughput_corrections(frame, skymodel)

Calculate the throughput corrections for each fiber based on the skymodel.

Parameters:

• frame (Frame object) – frame containing the data that may need to be corrected
• skymodel (SkyModel object) – skymodel object that contains the information about the sky for the given exposure/frame

Output:

corrections (1D array): 1D array where the index corresponds to the fiber % 500 and the values are the multiplicative corrections that would

be applied to the fluxes in frame.flux to correct them based on the input skymodel

desispec.sky.compute_non_uniform_sky(frame, nsig_clipping=4.0, max_iterations=10, model_ivar=False, add_variance=True, angular_variation_deg=1)

Compute a sky model.

Sky[fiber,i] = R[fiber,i,j] ( Flux_0[j] + x[fiber]*Flux_x[j] + y[fiber]*Flux_y[j] + … )

Input flux are expected to be flatfielded! We don’t check this in this routine.

Parameters:

• frame – Frame object, which includes attributes - wave : 1D wavelength grid in Angstroms - flux : 2D flux[nspec, nwave] density - ivar : 2D inverse variance of flux - mask : 2D inverse mask flux (0=good) - resolution_data : 3D[nspec, ndiag, nwave] (only sky fibers)
• nsig_clipping – [optional] sigma clipping value for outlier rejection

Optional:

max_iterations : int , number of iterations model_ivar : replace ivar by a model to avoid bias due to correlated flux and ivar. this has a negligible effect on sims. add_variance : evaluate calibration error and add this to the sky model variance angular_variation_deg : degree of 2D polynomial correction as a function of fiber focal plane coordinates (default=1). One set of coefficients per wavelength

returns SkyModel object with attributes wave, flux, ivar, mask

desispec.sky.compute_polynomial_times_sky(frame, nsig_clipping=4.0, max_iterations=30, model_ivar=False, add_variance=True, angular_variation_deg=1, chromatic_variation_deg=1)

Compute a sky model.

Sky[fiber,i] = R[fiber,i,j] Polynomial(x[fiber],y[fiber],wavelength[j]) Flux[j]

Input flux are expected to be flatfielded! We don’t check this in this routine.

Parameters:

• frame – Frame object, which includes attributes - wave : 1D wavelength grid in Angstroms - flux : 2D flux[nspec, nwave] density - ivar : 2D inverse variance of flux - mask : 2D inverse mask flux (0=good) - resolution_data : 3D[nspec, ndiag, nwave] (only sky fibers)
• nsig_clipping – [optional] sigma clipping value for outlier rejection

Optional:

max_iterations : int , number of iterations model_ivar : replace ivar by a model to avoid bias due to correlated flux and ivar. this has a negligible effect on sims. add_variance : evaluate calibration error and add this to the sky model variance

returns SkyModel object with attributes wave, flux, ivar, mask

desispec.sky.compute_sky(frame, nsig_clipping=4.0, max_iterations=100, model_ivar=False, add_variance=True, angular_variation_deg=0, chromatic_variation_deg=0, adjust_wavelength=False, adjust_lsf=False, only_use_skyfibers_for_adjustments=True, pcacorr=None, fit_offsets=False, fiberflat=None)

Compute a sky model.

Input flux are expected to be flatfielded! We don’t check this in this routine.

Parameters:

• frame – Frame object, which includes attributes - wave : 1D wavelength grid in Angstroms - flux : 2D flux[nspec, nwave] density - ivar : 2D inverse variance of flux - mask : 2D inverse mask flux (0=good) - resolution_data : 3D[nspec, ndiag, nwave] (only sky fibers)
• nsig_clipping – [optional] sigma clipping value for outlier rejection

Optional:

max_iterations : int , number of iterations model_ivar : replace ivar by a model to avoid bias due to correlated flux and ivar. this has a negligible effect on sims. add_variance : evaluate calibration error and add this to the sky model variance angular_variation_deg : Degree of polynomial for sky flux variation with focal plane coordinates (default=0, i.e. no correction, a uniform sky) chromatic_variation_deg : Wavelength degree for the chromatic x angular terms. If negative, use as many 2D polynomials of x and y as wavelength entries. adjust_wavelength : adjust the wavelength of the sky model on sky lines to improve the sky subtraction adjust_lsf : adjust the LSF width of the sky model on sky lines to improve the sky subtraction only_use_skyfibers_for_adjustments: interpolate adjustments using sky fibers only pcacorr : SkyCorrPCA object to interpolate the wavelength or LSF adjustment from sky fibers to all fibers fit_offsets : fit offsets for regions defined in calib fiberflat : desispec.FiberFlat object used for the fit of offsets

returns SkyModel object with attributes wave, flux, ivar, mask

desispec.sky.compute_uniform_sky(frame, nsig_clipping=4.0, max_iterations=100, model_ivar=False, add_variance=True, adjust_wavelength=True, adjust_lsf=True, only_use_skyfibers_for_adjustments=True, pcacorr=None, fit_offsets=False, fiberflat=None)

Compute a sky model.

Sky[fiber,i] = R[fiber,i,j] Flux[j]

Input flux are expected to be flatfielded! We don’t check this in this routine.

Parameters:

• frame – Frame object, which includes attributes - wave : 1D wavelength grid in Angstroms - flux : 2D flux[nspec, nwave] density - ivar : 2D inverse variance of flux - mask : 2D inverse mask flux (0=good) - resolution_data : 3D[nspec, ndiag, nwave] (only sky fibers)
• nsig_clipping – [optional] sigma clipping value for outlier rejection

Optional:

max_iterations : int , number of iterations model_ivar : replace ivar by a model to avoid bias due to correlated flux and ivar. this has a negligible effect on sims. add_variance : evaluate calibration error and add this to the sky model variance adjust_wavelength : adjust the wavelength of the sky model on sky lines to improve the sky subtraction adjust_lsf : adjust the LSF width of the sky model on sky lines to improve the sky subtraction only_use_skyfibers_for_adjustments : interpolate adjustments using sky fibers only pcacorr : SkyCorrPCA object to interpolate the wavelength or LSF adjustment from sky fibers to all fibers fit_offsets : fit offsets for regions defined in calib fiberflat : desispec.FiberFlat object used for the fit of offsets

returns SkyModel object with attributes wave, flux, ivar, mask

desispec.sky.qa_skysub(param, frame, skymodel, quick_look=False)

Calculate QA on SkySubtraction

Note: Pixels rejected in generating the SkyModel (as above), are not rejected in the stats calculated here. Would need to carry along current_ivar to do so.

Parameters:

• param – dict of QA parameters : see qa_frame.init_skysub for example
• frame – desispec.Frame object; Should have been flat fielded
• skymodel – desispec.SkyModel object
• quick_look – bool, optional If True, do QuickLook specific QA (or avoid some)

Returns:

dict of QA outputs

Need to record simple Python objects for yaml (str, float, int)

Return type:

qadict

desispec.sky.subtract_sky(frame, skymodel, apply_throughput_correction=True, zero_ivar=True)

Subtract skymodel from frame, altering frame.flux, .ivar, and .mask

Parameters:

• frame – desispec.Frame object
• skymodel – desispec.SkyModel object

Option:

apply_throughput_correction : if True, fit for an achromatic throughput correction.

This is to absorb variations of Focal Ratio Degradation with fiber flexure.

zero_ivar : if True , set ivar=0 for masked pixels

desispec.specscore

Spectral scores routines.

desispec.specscore.compute_coadd_scores(coadd, specscores=None, update_coadd=True)

Compute scores for a coadded Spectra object

Parameters:coadd – a Spectra object from a coadd

Options:

update_coadd: if True, update coadd.scores specscores: scores Table from the uncoadded spectra including a TARGETID column

Returns tuple of dictionaries (scores, comments); see compute_frame_scores

specscores is used to update TSNR2 scores by summing inputs

desispec.specscore.compute_coadd_tsnr_scores(specscores)

Compute coadded TSNR2 scores (TSNR2=Template Signal-to-Noise squared)

Parameters:specscores – uncoadded scores with TSNR2* columns (dict or Table-like)

Returns (tsnrscores, comments) tuple of dictionaries

desispec.specscore.compute_frame_scores(frame, band=None, suffix=None, flux_per_angstrom=None)

Computes scores in spectra of a frame.

The scores are sum,mean,medians in a predefined and fixed wavelength range for each DESI camera arm, or band, b, r or z. The band argument is optional because it can be automatically chosen from the wavelength range in the frame. The suffix is added to the key name in the output dictionnary, for instance ‘RAW’, ‘SKYSUB’, ‘CALIB’ … The boolean argument flux_per_angstrom is needed if there is no ‘BUNIT’ keyword in frame.meta (frame fits header)

Parameters:

• frame (Frame or QFrame) – A Frame or a QFrame object.
• band (str, optional) – Spectrograph band, b, r, z, autodetected by default.
• suffix (str, optional) – Character string added to the keywords in the output dictionary, for instance suffix=’RAW’
• flux_per_angstrom (bool, optional) – If True the spectra are assumed flux_per_angstrom, i.e. flux densities. If False, the spectra are assumed to be counts or photo-electrons per bin. None by default in which case the frame.units string is read to find out whether the flux quantity is per unit wavelenght or per bin.

Returns:

A tuple containg a dict of 1D arrays of size = number of spectra in frame and a dict of string with comments on the type of scores.

Return type:

tuple()

desispec.spectra

Class for dealing with a group of spectra from multiple bands and the associated fibermap information.

class desispec.spectra.Spectra(bands=[], wave={}, flux={}, ivar={}, mask=None, resolution_data=None, fibermap=None, exp_fibermap=None, meta=None, extra=None, single=False, scores=None, scores_comments=None, extra_catalog=None)

Represents a grouping of spectra.

This class contains an “extended” fibermap that has information about the night and exposure of each spectrum. For each band, this class has the wavelength grid, flux, ivar, mask, and resolution arrays.

Parameters:

• bands (list) – List of strings used to identify the bands.
• wave (dict) – Dictionary of arrays specifying the wavelength grid.
• flux (dict) – Dictionary of arrays specifying the flux for each spectrum.
• ivar (dict) – Dictionary of arrays specifying the inverse variance.
• mask (dict, optional) – Dictionary of arrays specifying the bitmask.
• resolution_data (dict, optional) – Dictionary of arrays specifying the block diagonal resolution matrix. The object for each band must be in one of the formats supported by the Resolution class constructor.
• Table-like, optional (exp_fibermap,) – Extended fibermap to use. If not specified, a fake one is created.
• Table-like, optional – Exposure-specific fibermap columns, which may not apply to a coadd.
• meta (dict, optional) – Dictionary of arbitrary properties.
• extra (dict, optional) – Optional dictionary of dictionaries containing extra floating point arrays. The top-level is a dictionary over bands and each value is a dictionary containing string keys and values which are arrays of the same size as the flux array.
• single (bool, optional) – If True, store data in memory as single precision.
• scores – QA scores table.
• scores_comments – dict[column] = comment to include in output file
• extra_catalog (numpy or astropy Table, optional) – optional table of metadata, rowmatched to fibermap, e.g. a redshift catalog for these spectra

_get_slice(index, bands=None)

Slice spectra by index. :param bands: optional list of bands to select. :type bands: list

Note: This function is intended to be private,

to be used by __getitem__() and select().

bands

the list of valid bands.

Type:(list)

ftype

the data type used for floating point numbers.

Type:(numpy.dtype)

num_spectra()

Get the number of spectra contained in this group.

Returns (int):

Number of spectra contained in this group.

num_targets()

Get the number of distinct targets.

Returns (int):

Number of unique targets with spectra in this object.

select(nights=None, exposures=None, bands=None, targets=None, fibers=None, invert=False, return_index=False)

Select a subset of the data.

This filters the data based on a logical AND of the different criteria, optionally inverting that selection.

Parameters:

• nights (list) – optional list of nights to select.
• exposures (list) – optional list of exposures to select.
• bands (list) – optional list of bands to select.
• targets (list) – optional list of target IDs to select.
• fibers (list) – list/array of fiber indices to select.
• invert (bool) – after combining all criteria, invert selection.
• return_index (bool) – if True, also return the indices of selected spectra.

Returns:

a new Spectra object containing the selected data. indices (list, optional): indices of selected spectra. Only provided if return_index is True.

Return type:

spectra

target_ids()

Return list of unique target IDs.

The target IDs are sorted by the order that they first appear.

Returns (array):

an array of integer target IDs.

update(other)

Overwrite or append new data.

Given another Spectra object, compare the fibermap information with the existing one. For spectra that already exist, overwrite existing data with the new values. For spectra that do not exist, append that data to the end of the spectral data.

Parameters:other (Spectra) – the new data to add. Returns:nothing (object updated in place).

Note: if fibermap, scores and extra_catalog exist in the new data, they are appended to the existing tables. If those new tables have different columns, only columns with identical names will be appended. Spectra.meta is unchanged.

wavelength_grid(band)

Return the wavelength grid for a band.

Parameters:band (str) – the name of the band.

Returns (array):

an array containing the wavelength values.

desispec.spectra.stack(speclist)

Stack a list of spectra, return a new spectra object

Parameters:speclist – list of Spectra objects

returns stacked Spectra object

Note: all input spectra must have the same bands, wavelength grid, and include or not the same optional elements (mask, fibermap, extra, …). The returned Spectra have the meta from the first input Spectra.

Also see Spectra.update, which is less efficient but more flexible for handling heterogeneous inputs

desispec.trace_shifts

desispec.trace_shifts.boxcar_extraction_from_filenames(image_filename, psf_filename, fibers=None, width=7)

Fast boxcar extraction of spectra from a preprocessed image and a trace set

Parameters:

• image_filename – input preprocessed fits filename
• psf_filename – input PSF fits filename

Optional:

fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : extraction boxcar width, default is 7

Returns:2D np.array of shape (nfibers,n0=image.shape[0]), sum of pixel values per row of length=width per fiber ivar : 2D np.array of shape (nfibers,n0), ivar[f,j] = 1/( sum_[j,b:e] (1/image.ivar) ), ivar=0 if at least 1 pixel in the row has image.ivar=0 or image.mask!=0 wave : 2D np.array of shape (nfibers,n0), determined from the traces Return type:flux

desispec.trace_shifts.compute_dx_dy_using_psf(psf, image, fibers, lines)

Computes trace shifts along x and y from a preprocessed image, a PSF (with trace coords), and a set of emission lines, by doing a forward model of the image. Calls compute_fiber_bundle_trace_shifts_using_psf.

Parameters:

• psf – specter psf object
• image – DESI preprocessed image object
• fibers – 1D array with list of fibers
• lines – 1D array of wavelength of emission lines (in Angstrom)

Returns:

1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD dy : 1D array of shifts along y coordinates on CCD sx : 1D array of uncertainties on dx sy : 1D array of uncertainties on dy fiber : 1D array of fiber ID wave : 1D array of wavelength

Return type:

x

desispec.trace_shifts.compute_dx_from_cross_dispersion_profiles(xcoef, ycoef, wavemin, wavemax, image, fibers, width=7, deg=2, image_rebin=4)

Measure x offsets from a preprocessed image and a trace set

Parameters:

• xcoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to XCCD
• ycoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to YCCD
• wavemin – float
• wavemax – float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1 used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])
• image – DESI preprocessed image object
• fibers – 1D np.array of int (default is all fibers, the first fiber is always = 0)

Optional:

width : extraction boxcar width, default is 5 deg : degree of polynomial fit as a function of y, only used to find and mask outliers image_rebin : rebinning of CCD rows to run faster (with rebin=4 loss of precision <0.01 pixel)

Returns:1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD ex : 1D array of uncertainties on dx fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength Return type:x

desispec.trace_shifts.compute_dy_from_spectral_cross_correlation(flux, wave, refflux, ivar=None, hw=3.0, calibrate=False)

Measure y offsets from two spectra expected to be on the same wavelength grid. refflux is the assumed well calibrated spectrum. A relative flux calibration of the two spectra is done internally.

Parameters:

• flux – 1D array of spectral flux as a function of wavelenght
• wave – 1D array of wavelength (in Angstrom)
• refflux – 1D array of reference spectral flux

Optional:

ivar : 1D array of inverse variance of flux hw : half width in Angstrom of the cross-correlation chi2 scan, default=3A corresponding approximatly to 5 pixels for DESI

Returns:1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD ex : 1D array of uncertainties on dx fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength Return type:x

desispec.trace_shifts.compute_dy_from_spectral_cross_correlations_of_frame(flux, ivar, wave, xcoef, ycoef, wavemin, wavemax, reference_flux, n_wavelength_bins=4)

Measures y offsets from a set of resampled spectra and a reference spectrum that are on the same wavelength grid. reference_flux is the assumed well calibrated spectrum. Calls compute_dy_from_spectral_cross_correlation per fiber

Parameters:

• flux – 2D np.array of shape (nfibers,nwave)
• ivar – 2D np.array of shape (nfibers,nwave) , inverse variance of flux
• wave – 1D array of wavelength (in Angstrom) of size nwave
• refflux – 1D array of reference spectral flux of size nwave

Optional:

n_wavelength_bins : number of bins along wavelength

Returns:1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dy : 1D array of shifts along y coordinates on CCD ey : 1D array of uncertainties on dy fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength Return type:x

desispec.trace_shifts.compute_dy_using_boxcar_extraction(xytraceset, image, fibers, width=7, degyy=2)

Measures y offsets (internal wavelength calibration) from a preprocessed image and a trace set using a cross-correlation of boxcar extracted spectra. Uses boxcar_extraction , resample_boxcar_frame , compute_dy_from_spectral_cross_correlations_of_frame

Parameters:

• xytraceset – XYTraceset object
• image – DESI preprocessed image object

Optional:

fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : int, extraction boxcar width, default is 7 degyy : int, degree of polynomial fit of shifts as a function of y, used to reject outliers.

Returns:1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dy : 1D array of shifts along y coordinates on CCD ey : 1D array of uncertainties on dy fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength Return type:x

desispec.trace_shifts.compute_fiber_bundle_trace_shifts_using_psf(fibers, line, psf, image, maxshift=2.0)

Computes trace shifts along x and y from a preprocessed image, a PSF (with trace coords), and a given emission line, by doing a forward model of the image.

Parameters:

• fibers – 1D array with list of fibers
• line – float, wavelength of an emission line (in Angstrom)
• psf – specter psf object
• image – DESI preprocessed image object

Optional:

maxshift : float maximum shift in pixels for 2D chi2 scan

Returns:1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD dy : 1D array of shifts along y coordinates on CCD sx : 1D array of uncertainties on dx sy : 1D array of uncertainties on dy Return type:x

desispec.trace_shifts.legx(wave, wavemin, wavemax)

Reduced coordinate (range [-1,1]) for calls to legval and legfit

Parameters:

• wave – ND np.array
• wavemin – float, min. val
• wavemax – float, max. val

Returns:

array of same shape as wave

desispec.trace_shifts.monomials(x, y, degx, degy)

Computes monomials as a function of x and y of a 2D polynomial of degrees degx and degy

Parameters:

• x – ND array
• y – ND array of same shape as x
• degx – int (>=0), polynomial degree along x
• degy – int (>=0), polynomial degree along y

Returns :

monomials : ND array of shape ( (degx+1)*(degy+1) , x shape )

desispec.trace_shifts.polynomial_fit(z, ez, xx, yy, degx, degy)

Computes and 2D polynomial fit of z as a function of (x,y) of degrees degx and degy

Parameters:

• z – ND array
• ez – ND array of same shape as z, uncertainties on z
• x – ND array of same shape as z
• y – ND array of same shape as z
• degx – int (>=0), polynomial degree along x
• degy – int (>=0), polynomial degree along y

Returns:

1D array of size (degx+1)*(degy+1) with polynomial coefficients (as defined by routine monomials) covariance : 2D array of covariance of coeff error_floor : float , extra uncertainty needed to get chi2/ndf=1 polval : ND array of same shape as z with values of pol(x,y) mask : ND array of same shape as z indicating the masked data points in the fit

Return type:

coeff

desispec.trace_shifts.recompute_legendre_coefficients(xcoef, ycoef, wavemin, wavemax, degxx, degxy, degyx, degyy, dx_coeff, dy_coeff)

Modifies legendre coefficients of an input trace set using polynomial coefficents (as defined by the routine monomials)

Parameters:

• xcoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to XCCD
• ycoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to YCCD
• wavemin – float
• wavemax – float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1 used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])
• degxx – int, degree of polynomial for x shifts as a function of x (x is axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction)
• degxy – int, degree of polynomial for x shifts as a function of y (y is axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis)
• degyx – int, degree of polynomial for y shifts as a function of x
• degyy – int, degree of polynomial for y shifts as a function of y
• dx_coeff – 1D np.array of polynomial coefficients of size (degxx*degxy) as defined by the routine monomials.
• dy_coeff – 1D np.array of polynomial coefficients of size (degyx*degyy) as defined by the routine monomials.

Returns:

2D np.array of shape (nfibers,ncoef) with modified Legendre coefficents ycoef : 2D np.array of shape (nfibers,ncoef) with modified Legendre coefficents

Return type:

xcoef

desispec.trace_shifts.resample_boxcar_frame(frame_flux, frame_ivar, frame_wave, oversampling=2)

Resamples the spectra in a frame obtained with boxcar extraction to the same wavelength grid, with oversampling. Uses resample_flux routine.

Parameters:

• frame_flux – 2D np.array of shape (nfibers,nwave), sum of pixel values per row of length=width per fiber
• frame_ivar – 2D np.array of shape (nfibers,nwave), ivar[f,j] = 1/( sum_[j,b:e] (1/image.ivar) ), ivar=0 if at least 1 pixel in the row has image.ivar=0 or image.mask!=0
• frame_wave – 2D np.array of shape (nfibers,nwave), determined from the traces

Optional:

oversampling : int , oversampling factor , default is 2

Returns:2D np.array of shape (nfibers,nwave*oversampling) ivar : 2D np.array of shape (nfibers,nwave*oversampling) frame_wave : 1D np.array of size (nwave*oversampling) Return type:flux

desispec.trace_shifts.shift_ycoef_using_external_spectrum(psf, xytraceset, image, fibers, spectrum_filename, degyy=2, width=7)

Measure y offsets (external wavelength calibration) from a preprocessed image , a PSF + trace set using a cross-correlation of boxcar extracted spectra and an external well-calibrated spectrum. The PSF shape is used to convolve the input spectrum. It could also be used to correct for the PSF asymetry (disabled for now). A relative flux calibration of the spectra is performed internally.

Parameters:

• psf – specter PSF
• xytraceset – XYTraceset object
• image – DESI preprocessed image object
• fibers – 1D np.array of fiber indices
• spectrum_filename – path to input spectral file ( read with np.loadtxt , first column is wavelength (in vacuum and Angstrom) , second column in flux (arb. units)

Optional:

width : int, extraction boxcar width, default is 7 degyy : int, degree of polynomial fit of shifts as a function of y, used to reject outliers.

Returns:2D np.array of same shape as input, with modified Legendre coefficents for each fiber to convert wavelenght to YCCD Return type:ycoef

desispec.trace_shifts.write_traces_in_psf(input_psf_filename, output_psf_filename, xytraceset)

Writes traces in a PSF.

Parameters:

• input_psf_filename – Path to input fits file which has to contain XTRACE and YTRACE HDUs
• output_psf_filename – Path to output fits file which has to contain XTRACE and YTRACE HDUs
• xytraceset – xytraceset

Utility functions for desispec

desispec.util.combine_ivar(ivar1, ivar2)

Returns the combined inverse variance of two inputs, making sure not to divide by 0 in the process.

ivar1 and ivar2 may be scalar or ndarray but must have the same dimensions

desispec.util.dateobs2night(dateobs)

Convert DATE-OBS ISO8601 UTC string to YEARMMDD int night of KPNO sunset

desispec.util.header2night(header)

Return YEARMMDD night from FITS header, handling common problems

desispec.util.healpix_degrade_fixed(nside, pixel)

Degrade a NEST ordered healpix pixel with a fixed ratio.

This degrades the pixel to a lower nside value that is fixed to half the healpix “factor”.

Parameters:

• nside (int) – a valid NSIDE value.
• pixel (int) – the NESTED pixel index.

Returns (tuple):

a tuple of ints, where the first value is the new NSIDE and the second value is the degraded pixel index.

desispec.util.itemindices(a)

Return dict[key] -> list of indices i where a[i] == key

Parameters:a – array-like of hashable values

Return dict[key] -> list of indices i where a[i] == key

The dict keys are inserted in the order that they first appear in a, and the value lists of indices are sorted

e.g. itemindices([10,30,20,30]) -> {10: [0], 30: [1, 3], 20: [2]}

desispec.util.mask32(mask)

Return an input mask as unsigned 32-bit

Raises ValueError if 64-bit input can’t be cast to 32-bit without losing info (i.e. if it contains values > 2**32-1)

desispec.util.mjd2night(mjd)

Convert MJD to YEARMMDD int night of KPNO sunset

desispec.util.mpi_count_failures(num_cmd, num_err, comm=None)

Sum num_cmd and num_err across MPI ranks

Parameters:

• num_cmd (int) – number of commands run
• num_err (int) – number of failures

Options:

comm: mpi4py communicator

Returns:sum(num_cmd), sum(num_err) summed across all MPI ranks

If comm is None, returns input num_cmd, num_err

desispec.util.night2ymd(night)

parse night YEARMMDD string into tuple of integers (year, month, day)

desispec.util.option_list(opts)

Convert key, value pairs into command-line options.

Parameters:opts (dict-like) – Convert a dictionary into command-line options. Returns:A list of command-line options. Return type:list

desispec.util.ordered_unique(ar, return_index=False)

Find the unique elements of an array in the order they first appear

Like numpy.unique, but preserves original order instead of sorting

Parameters:ar – array-like data to find unique elements

Options:

return_index: if True also return indices in ar where items first appear

desispec.util.parse_fibers(fiber_string, include_end=False)

Short func that parses a string containing a comma separated list of integers, which can include “:” or “..” or “-” labeled ranges

Parameters:fiber_string (str) – list of integers or integer ranges

Options:

include_end (bool): if True, include end-value in ranges

Returns (array 1-D):

1D numpy array listing all of the integers given in the list, including enumerations of ranges given.

Note: this follows python-style ranges, i,e, 1:5 or 1..5 returns 1, 2, 3, 4 unless include_end is True, which then returns 1,2,3,4,5

desispec.util.parse_int_args(arg_string, include_end=False)

Short func that parses a string containing a comma separated list of integers, which can include “:” or “..” or “-” labeled ranges

Parameters:arg_string (str) – list of integers or integer ranges

Options:

include_end (bool): if True, include end-value in ranges

Returns (array 1-D):

1D numpy array listing all of the integers given in the list, including enumerations of ranges given.

Note: this follows python-style ranges, i,e, 1:5 or 1..5 returns 1,2,3,4 unless include_end is True, which then returns 1,2,3,4,5

desispec.util.pid_exists(pid)

Check whether pid exists in the current process table.

UNIX only. Should work the same as psutil.pid_exists().

Parameters:pid (int) – A process ID. Returns:True if the process exists in the current process table. Return type:pid_exists (bool)

desispec.util.runcmd(cmd, args=None, inputs=[], outputs=[], clobber=False)

Runs a command, checking for inputs and outputs

Parameters:

• cmd – command string to run with subprocess.call()
• inputs – list of filename inputs that must exist before running
• outputs – list of output filenames that should be created
• clobber – if True, run even if outputs already exist

Returns:

error code from command or input/output checking; 0 is good

Notes

If any inputs are missing, don’t run cmd. If outputs exist and have timestamps after all inputs, don’t run cmd.

desispec.util.sprun(com, capture=False, input=None)

Run a command with subprocess and handle errors.

This runs a command and returns the lines of STDOUT as a list. Any contents of STDERR are logged. If an OSError is raised by the child process, that is also logged. If another exception is raised by the child process, the traceback from the child process is printed.

Parameters:

• com (list) – the command to run.
• capture (bool) – if True, return the stdout contents.
• input (str) – the string data (can include embedded newlines) to write to the STDIN of the child process.

Returns:

the return code and optionally the lines of STDOUT

from the child process.

Return type:

tuple(int, (list))

desispec.util.ymd2night(year, month, day)

convert year, month, day integers into cannonical YEARMMDD night string

desispec.workflow.schedule

Tools for scheduling MPI jobs using mpi4py

desispec.xytraceset

Lightweight wrapper class for trace coordinates and wavelength solution, to be returned by read_xytraceset().