desispec API¶
desispec¶
Tools for DESI spectroscopic processing.
desispec.averagefluxcalibration¶
desispec.badcolumn¶
Utility functions to treat bad CCD columns
- desispec.badcolumn.add_badcolumn_mask(frame, xyset, badcolumns_table, threshold_value=0.005, threshold_specfrac=0.6)[source]¶
Modifies the input frame spectral mask and fiber mask (frame.mask and frame.fibermap[“FIBERSTATUS”].mask) by adding (bitwise OR) the BADCOLUMN bits for values/fibers affected by bad CCD columns.
- Parameters:
frame – desispec.frame.Frame instance
xyset – desispec.xytraceset.XYTraceSet instance (valid for the frame)
badcolumns_table – astropy.table.Table with columns “COLUMN” (CCD column index) and “ELEC_PER_SEC”, value in CCD column, in electron / sec
threshold_value – threshold for masking spectral pixels, in electron / sec
threshold_specfrac – fraction of specmask.BADCOLUMN masked pixels to trigger a fibermask bitmask fibermask.BADCOLUMN.
- Returns:
nothing, the input frame is modified
- desispec.badcolumn.compute_badcolumn_fibermask(frame_mask, camera_arm, threshold_specfrac=0.6)[source]¶
fills a mask for fibers affected by a bad CCD column.
- Parameters:
frame_mask – 2D integer numpy array (nfibers x nwavelength), with bitmask defined in desispec.maskbits.specmask
threshold_specfrac – fraction of specmask.BADCOLUMN masked pixels to trigger a fibermask bitmask fibermask.BADAMP… .
camera_arm – ‘B’,’R’ or ‘Z’
- Returns:
fiber mask 1D numpy array of size nfibers (bitwise OR of the orginal mask with the newly set bits)
- desispec.badcolumn.compute_badcolumn_specmask(frame, xyset, badcolumns_table, threshold_value=0.005)[source]¶
fills a mask for spectral values affected by a bad CCD column.
- Parameters:
frame – desispec.frame.Frame instance
xyset – desispec.xytraceset.XYTraceSet instance (valid for the frame)
badcolumns_table – astropy.table.Table with columns “COLUMN” (CCD column index) and “ELEC_PER_SEC”, value in CCD column, in electron / sec
threshold_value – threshold for masking spectral pixels, in electron / sec
- Returns:
mask numpy array of same shape as frame.flux, or None if nothing is masked
- desispec.badcolumn.flux_bias_function(delta_x)[source]¶
Multiply the excess counts in a bad column by this function to determine the bias on the extracted flux of a spectral trace situated at a distance dx (in pixels) from the bad CCD column. This has been derived empirically on the r and z cameras (with inverse variance cross-dispersion profile weighted extractions). https://github.com/desihub/desispec/pull/1371#issuecomment-904961936
- Parameters:
delta_x – float or numpy array
- Returns:
flux bias, same dimension as input delta_x
desispec.bootcalib¶
Utility functions to perform a quick calibration of DESI data
TODO: 1. Expand to r, i cameras 2. QA plots 3. Test with CR data
- desispec.bootcalib.bootcalib(deg, flatimage, arcimage)[source]¶
- Parameters:
deg – Legendre polynomial degree to use to fit
flatimage – desispec.image.Image object of flatfield
arcimage – desispec.image.Image object of arc
Mostly inherited from desispec/bin/desi_bootcalib directly as needed
- Returns:
xfit, fdicts, gauss, all_wave_soln
TODO: document what those return objects are
- desispec.bootcalib.extract_sngfibers_gaussianpsf(img, img_ivar, xtrc, sigma, box_radius=2, verbose=True)[source]¶
Extract spectrum for fibers one-by-one using a Gaussian PSF
- desispec.bootcalib.fiber_gauss_new(flat, xtrc, xerr, box_radius=2, max_iter=5, debug=False, verbose=False)[source]¶
Find the PSF sigma for each fiber This serves as an initial guess to what follows
- Parameters:
- Returns:
list of Gaussian sigma
- Return type:
gauss
- desispec.bootcalib.fiber_gauss_old(flat, xtrc, xerr, box_radius=2, max_iter=5, debug=False, verbose=False)[source]¶
Find the PSF sigma for each fiber This serves as an initial guess to what follows
- Parameters:
- Returns:
list of Gaussian sigma
- Return type:
gauss
- desispec.bootcalib.find_arc_lines(spec, rms_thresh=7.0, nwidth=5)[source]¶
Find and centroid arc lines in an input spectrum
- desispec.bootcalib.find_fiber_peaks(flat, ypos=None, nwidth=5, debug=False, thresh=None)[source]¶
Find the peaks of the fiber flat spectra Preforms book-keeping error checking
- Parameters:
flat – ndarray of fiber flat image
ypos – int [optional] Row for finding peaks Default is half-way up the image
nwidth – int [optional] Width of peak (end-to-end)
debug – bool, optional
- Returns:
- xpk, ypos, cut
list of xpk (nearest pixel) at ypos ndarray of cut through the image
- desispec.bootcalib.fit_traces(xset, xerr, func='legendre', order=6, sigrej=20.0, RMS_TOLER=0.03, verbose=False)[source]¶
Fit the traces Default is 6th order Legendre polynomials
- Parameters:
xset (ndarray) – traces
xerr (ndarray) – Error in the trace values (999.=Bad)
RMS_TOLER (float, optional [0.02]) – Tolerance on size of RMS in fit
- Returns:
xnew, fits
xnew (ndarray) – New fit values (without error)
fits (list) – List of the fit dicts
- desispec.bootcalib.fix_ycoeff_outliers(xcoeff, ycoeff, deg=5, tolerance=2)[source]¶
Fix outliers in coefficients for wavelength solution, assuming a continuous function of CCD coordinates
- Parameters:
xcoeff[nfiber – 2D array of Legendre coefficients for X(wavelength)
ncoeff] – 2D array of Legendre coefficients for X(wavelength)
ycoeff[nfiber – 2D array of Legendre coefficients for Y(wavelength)
ncoeff] – 2D array of Legendre coefficients for Y(wavelength)
- Options:
deg : integer degree of polynomial to fit tolerance : replace fibers with difference of wavelength solution larger than this number of pixels after interpolation
- Returns:
new_ycoeff[nfiber, ncoeff] with outliers replaced by interpolations
For each coefficient, fit a polynomial vs. fiber number with one pass of sigma clipping. Remaining outliers are than replaced with the interpolated fit value.
- desispec.bootcalib.id_arc_lines_using_triplets(id_dict, w, dwdy_prior, d2wdy2_prior=1.5e-05, toler=0.2, ntrack=50, nmax=40)[source]¶
Match (as best possible), a set of the input list of expected arc lines to the detected list
- Parameters:
id_dict (dictionnary with Pixel locations of detected arc lines in "pixpk" and fluxes in "flux") –
w (ndarray) – array of expected arc lines to be detected and identified
dwdy (float) – Average dispersion in the spectrum
d2wdy2_prior (float) – Prior on second derivative
toler (float, optional) – Tolerance for matching (20%)
ntrack (max. number of solutions to be tracked) –
- Returns:
id_dict – dict of identified lines
- Return type:
- desispec.bootcalib.load_arcline_list(camera, vacuum=True, lamps=None)[source]¶
Loads arc line list from NIST files Parses and rejects
Taken from PYPIT
- desispec.bootcalib.load_gdarc_lines(camera, llist, vacuum=True, lamps=None, good_lines_filename=None)[source]¶
Loads a select set of arc lines for initial calibrating
- Parameters:
- Returns:
dlamb (float) – Dispersion for input camera
wmark (float) – wavelength to key off of [???]
gd_lines (ndarray) – Array of lines expected to be recorded and good for ID
line_guess (int or None) – Guess at the line index corresponding to wmark (default is to guess the 1/2 way point)
- desispec.bootcalib.load_parse_dict()[source]¶
Dicts for parsing Arc line lists from NIST
Rejected lines are in the rejected_lines.yaml file
- desispec.bootcalib.parse_nist(ion, vacuum=True)[source]¶
Parse a NIST ASCII table.
Note that the long —- should have been commented out and also the few lines at the start.
Taken from PYPIT
- Parameters:
- Returns:
A Table obtained from the data file with some columns added or renamed.
- Return type:
- desispec.bootcalib.parse_nist_tbl(tbl, parse_dict)[source]¶
Parses a NIST table using various criteria
- Parameters:
tbl (Table) – Read previously from NIST ASCII file
parse_dict (dict) – Dict of parsing criteria. Read from load_parse_dict
- Returns:
tbl – Rows meeting the criteria
- Return type:
Table
- desispec.bootcalib.qa_arc_spec(all_spec, all_soln, pp, figsz=None)[source]¶
Generate QA plots of the arc spectra with IDs
- Parameters:
all_spec – Arc 1D fiber spectra
all_soln – Wavelength solutions
pp – PDF file pointer
figsz – figure size, optional
- desispec.bootcalib.qa_fiber_Dx(xfit, fdicts, pp=None, figsz=None)[source]¶
Show the spread in the trace per fiber
Used to diagnose the traces
- Parameters:
xfit – traces
fdicts – dict of the traces
pp – PDF file pointer
figsz – figure size, optional
- desispec.bootcalib.qa_fiber_arcrms(all_soln, pp, figsz=None)[source]¶
Show the RMS of the wavelength solutions vs. fiber
- Parameters:
all_soln – Wavelength solutions
pp – PDF file pointer
figsz – figure size, optional
- desispec.bootcalib.qa_fiber_dlamb(all_spec, all_soln, pp, figsz=None)[source]¶
Show the Dlamb of the wavelength solutions vs. fiber
- Parameters:
all_soln – Wavelength solutions
pp – PDF file pointer
figsz – figure size, optional
- desispec.bootcalib.qa_fiber_gauss(gauss, pp=None, figsz=None)[source]¶
Show the Gaussian (sigma) fits to each fiber
- Parameters:
gauss – Gaussian of each fiber
pp – PDF file pointer
figsz – figure size, optional
- desispec.bootcalib.qa_fiber_peaks(xpk, cut, pp=None, figsz=None, nper=100)[source]¶
Generate a QA plot for the fiber peaks
- Parameters:
xpk – x positions on the CCD of the fiber peaks at a ypos
cut – Spatial cut through the detector
pp – PDF file pointer
figsz – figure size, optional
nper – number of fibers per row in the plot, optional
- desispec.bootcalib.qa_fiber_trace(flat, xtrc, outfil=None, Nfiber=25, isclmin=0.5)[source]¶
Generate a QA plot for the fiber traces
- desispec.bootcalib.reject_lines(tbl, rej_dict, rej_tol=0.1)[source]¶
Rejects lines from a NIST table
Taken from PYPIT
- desispec.bootcalib.script_bootcalib(arc_idx, flat_idx, cameras=None, channels=None, nproc=10)[source]¶
Runs desi_bootcalib on a series of preproc files
- Returns:
script_bootcalib([0,1,2,3,4,5,6,7,8,9], [10,11,12,13,14])
- desispec.bootcalib.trace_crude_init(image, xinit0, ypass, invvar=None, radius=2.0, maxshift0=0.5, maxshift=0.15, maxerr=0.2)[source]¶
Python port of trace_crude_idl.pro from IDLUTILS
Modified for initial guess
- Parameters:
image (2D ndarray) – Image for tracing
xinit (ndarray) – Initial guesses for trace peak at ypass
ypass (int) – Row for initial guesses
- Returns:
xset (Trace for each fiber)
xerr (Estimated error in that trace)
- desispec.bootcalib.trace_fweight(fimage, xinit, ycen=None, invvar=None, radius=2.0, debug=False)[source]¶
Python port of trace_fweight.pro from IDLUTILS
- Parameters:
fimage (2D ndarray) – Image for tracing
xinit (ndarray) – Initial guesses for x-trace
invvar (ndarray, optional) – Inverse variance array for the image
radius (float, optional) – Radius for centroiding; default to 3.0
- desispec.bootcalib.write_psf(outfile, xfit, fdicts, gauss, wv_solns, legendre_deg=5, without_arc=False, XCOEFF=None, fiberflat_header=None, arc_header=None, fix_ycoeff=True)[source]¶
Write the output to a Base PSF format
desispec.calibfinder¶
Reading and selecting calibration data from $DESI_SPECTRO_CALIB using content of image headers
- desispec.calibfinder._load_smsp()[source]¶
Loads $DESI_SPECTRO_CALIB/spec/smsp.txt into global _sp2sm and _sm2sp dicts
- desispec.calibfinder.badfibers(headers, keys=['BROKENFIBERS', 'BADCOLUMNFIBERS', 'LOWTRANSMISSIONFIBERS'], yaml_file=None)[source]¶
find list of bad fibers from $DESI_SPECTRO_CALIB using the keywords found in the headers
- Parameters:
headers – list of fits headers, or list of dictionnaries
- Optional:
keys: list of keywords, among [“BROKENFIBERS”,”BADCOLUMNFIBERS”,”LOWTRANSMISSIONFIBERS”]. Default is all of them. yaml_file: path to a specific yaml file. By default, the code will automatically find the yaml file from the environment variable DESI_SPECTRO_CALIB and the CAMERA keyword in the headers
Returns List of bad fibers as a 1D array of intergers
- desispec.calibfinder.ccdregionmask(headers)[source]¶
Looks for regions of CCD to mask for a given NIGHT EXPID CAMERA and returns a list of dictionnaries. NIGHT EXPID CAMERA are retrieved from the input image headers and compared to corresponding columns in the cvs table $DESI_SPECTRO_CALIB/ccd/ccd-region-mask.csv
- Parameters:
headers – list of fits headers, or list of dictionnaries
Returns list of dictionnaries with keys XMIN, XMAX, YMIN,YMAX
- desispec.calibfinder.findcalibfile(headers, key, yaml_file=None)[source]¶
read and select calibration data file from $DESI_SPECTRO_CALIB using the keywords found in the headers
- Parameters:
headers – list of fits headers, or list of dictionnaries
key – type of calib file, e.g. ‘PSF’ or ‘FIBERFLAT’
- Optional:
yaml_file: path to a specific yaml file. By default, the code will automatically find the yaml file from the environment variable DESI_SPECTRO_CALIB and the CAMERA keyword in the headers
Returns path to calibration file
- desispec.calibfinder.parse_date_obs(value)[source]¶
converts DATE-OBS keywork to int with for instance DATE-OBS=2016-12-21T18:06:21.268371-05:00
- desispec.calibfinder.sm2sp(sm, night=None)[source]¶
Converts spectrograph sm hardware number to sp logical number
- Parameters:
sm – spectrograph sm number 1-10 or str sm[1-10]
Returns “spP” if input is str “smM”, or int P if input is int M
Note: uses $DESI_SPECTRO_CALIB/spec/smsp.txt
TODO: add support for different mappings based on night
- desispec.calibfinder.sp2sm(sp)[source]¶
Converts spectrograph sp logical number to sm hardware number
- Parameters:
sp – spectrograph int 0-9 or str sp[0-9]
Returns “smM” if input is str “spP”, or int M if input is int P
Note: uses $DESI_SPECTRO_CALIB/spec/smsp.txt
TODO: add support for different mappings based on night
desispec.ccdcalib¶
- desispec.ccdcalib._find_zeros(night, cameras, nzeros=25, nskip=2)[source]¶
Find all OBSTYPE=ZERO exposures on a given night
- Parameters:
- Options:
nzeros (int): number of zeros desired from valid all-cam observations to not worry about partials nskip (int): number of initial zeros to skip
- Returns:
- dictionary of expids (values) for each camera (keys)
for calibration zeros
- noncalib_expdict (dict): dictionary of expids (values) for each camera (keys)
for non calibration zeros.
- Return type:
calib_expdict (dict)
Uses production exposure tables to veto known bad ZEROs, but it will also find any ZEROs on disk for that night, regardless of whether they are in the exposures table or not.
- desispec.ccdcalib.compare_bias(rawfile, biasfile1, biasfile2, ny=8, nx=40)[source]¶
Compare rawfile image to bias images in biasfile1 and biasfile2
- Parameters:
rawfile – full filepath to desi*.fits.fz raw data file
biasfile1 – filepath to bias model made from OBSTYPE=ZERO exposures
biasfile2 – filepath to bias model made from OBSTYPE=ZERO exposures
- Options:
ny (even int): number of patches in y (row) direction nx (even int): number of patches in x (col) direction
Returns tuple (mdiff1[ny,nx], mdiff2[ny,nx]) median(image-bias) in patches
The rawfile camera is derived from the biasfile CAMERA header keyword.
median(raw-bias) is calculated in ny*nx patches using only the DATASEC portion of the images. Since the DESI CCD bias features tend to vary faster with row than column, default patches are (4k/8 x 4k/40) = 500x100.
- desispec.ccdcalib.compare_dark(preprocfile1, preprocfile2, ny=8, nx=40)[source]¶
Compare preprocessed dark images based on different dark models
- Parameters:
preprocfile1 – filepath to bias model made from OBSTYPE=ZERO exposures
preprocfile2 – filepath to bias model made from OBSTYPE=ZERO exposures
- Options:
ny (even int): number of patches in y (row) direction nx (even int): number of patches in x (col) direction
Returns tuple (mdiff1[ny,nx], mdiff2[ny,nx]) median(image-bias) in patches
median(raw-bias) is calculated in ny*nx patches using only the DATASEC portion of the images. Since the DESI CCD bias features tend to vary faster with row than column, default patches are (4k/8 x 4k/40) = 500x100.
- desispec.ccdcalib.compute_bias_file(rawfiles, outfile, camera, explistfile=None, extraheader=None)[source]¶
Compute a bias file from input ZERO rawfiles
- Parameters:
- Options:
explistfile: filename with text list of NIGHT EXPID to use extraheader: dict-like key/value header keywords to add
Notes: explistfile is only used if rawfiles=None; it should have one NIGHT EXPID entry per line.
- desispec.ccdcalib.compute_dark_file(rawfiles, outfile, camera, bias=None, nocosmic=False, scale=False, exptime=None)[source]¶
Compute classic dark model from input dark images
- Parameters:
- Options:
bias (str or list): bias file to use, or list of bias files nocosmic (bool): use medians instead of cosmic identification scale (bool): apply scale correction for EM0 teststand data exptime (float): write EXPTIME header keyword; all inputs must match
Note: if bias is None, no bias correction is applied. If it is a single file, then use that bias for all darks. If it is a list, it must have len(rawfiles) and gives the per-file bias to use.
Note: this computes a classic dark model without any non-linear terms. see bin/compute_dark_nonlinear for current DESI dark model.
TODO: separate algorithm from I/O
- desispec.ccdcalib.compute_nightly_bias(night, cameras, outdir=None, nzeros=25, minzeros=15, nskip=2, anyzeros=False, comm=None)[source]¶
Create nightly biases for cameras on night
- Parameters:
- Options:
outdir (str): write files to this output directory nzeros (int): number of OBSTYPE=ZERO exposures to use minzeros (int): minimum number of OBSTYPE=ZERO exposures required nskip (int): number of initial zeros to skip anyzeros (bool): allow any ZEROs, not just those taken for CCD calib seq comm: MPI communicator for parallelism
- Returns:
number of cameras that failed across all ranks
- Return type:
nfail (int)
Writes biasnight*.fits files in outdir or $DESI_SPECTRO_REDUX/$SPECPROD/calibnight/night/
Note: compute_bias_file requires ~12 GB memory per camera, so limit the size of the MPI communicator depending upon the memory available.
- desispec.ccdcalib.fit_const_plus_dark(exp_arr, image_arr)[source]¶
fit const + dark*t model given images and exptimes
- Parameters:
exp_arr – list of exposure times
image_arr – list of average dark images
returns: (const, dark) tuple of images
NOTE: the image_arr should not be divided by the exposure time
- desispec.ccdcalib.fit_dark(exp_arr, image_arr)[source]¶
fit dark*t model given images and exptimes
- Parameters:
exp_arr – list of exposure times
image_arr – list of average dark images
returns: dark_image
NOTE: the image_arr should not be divided by the exposure time
- desispec.ccdcalib.make_dark_scripts(outdir, days=None, nights=None, cameras=None, linexptime=None, nskip_zeros=None, tempdir=None, nosubmit=False, first_expid=None, night_for_name=None, use_exptable=True, queue='realtime', copy_outputs_to_split_dirs=False, prepared_exptable=None, system_name=None, min_vccdsec=None)[source]¶
Generate batch script to run desi_compute_dark_nonlinear
- Parameters:
outdir (str) – output directory
nights (days or) – days or nights to include
- Options:
cameras (list of str): cameras to include, e.g. b0, r1, z9 linexptime (float): exptime after which dark current is linear nskip_zeros (int): number of ZEROs at beginning of day/night to skip tempdir (str): tempfile working directory nosubmit (bool): generate scripts but don’t submit them to batch queue first_expid (int): ignore expids prior to this use_exptable (bool): use shortened copy of joined exposure tables instead of spectable (need to have right $SPECPROD set) queue (str): which batch queue to use for submission copy_outputs_to_split_dirs (bool): whether to copy outputs to bias_frames/dark_frames subdirs prepared_exptable (exptable): if a table is submitted here, no further spectra will be searched and this will be used instead system_name (str): the system for which batch files should be created, defaults to guessing current system min_vccdsec(str): minimum time a ccd needs to be turned on to be allowed in the model
Args/Options are passed to the desi_compute_dark_nonlinear script
- desispec.ccdcalib.make_regular_darks(outdir=None, lastnight=None, cameras=None, window=30, linexptime=None, nskip_zeros=None, tempdir=None, nosubmit=False, first_expid=None, night_for_name=None, use_exptable=True, queue='realtime', copy_outputs_to_split_dirs=None, transmit_obslist=True, system_name=None, no_obslist=False, min_vccdsec=None)[source]¶
Generate batch script to run desi_compute_dark_nonlinear
- Options:
outdir (str): output directory lastnight (int): last night to take into account (inclusive), defaults to tonight
window (int): length of time window to take into account cameras (list of str): cameras to include, e.g. b0, r1, z9 linexptime (float): exptime after which dark current is linear nskip_zeros (int): number of ZEROs at beginning of day/night to skip tempdir (str): tempfile working directory nosubmit (bool): generate scripts but don’t submit them to batch queue first_expid (int): ignore expids prior to this use_exptable (bool): use shortened copy of joined exposure tables instead of spectable (need to have right $SPECPROD set) queue (str): which batch queue to use for submission transmit_obslist(bool): if True will give use the obslist from here downstream system_name(str): allows to overwrite the system for which slurm scripts are created, will default to guessing the current system no_obslist(str): just use exactly the specified night-range, but assume we do not have exposure tables for this (useful when there is no exposure_table yet) min_vccdsec(float): minimum time a ccd needs to be turned on to be allowed in the model (default: 21600=6h)
Args/Options are passed to the desi_compute_dark_nonlinear script
- desispec.ccdcalib.model_y1d(image, smooth=0)[source]¶
Model image as a sigma-clipped mean 1D function of row
- Parameters:
image – 2D array to model
- Options:
smooth (int): if >0, Savitzky-Golay filter curve by this window
Returns 1D model of image with len = image.shape[0]
- desispec.ccdcalib.select_zero_expids(calib_exps, noncalib_exps, night, cam, nzeros=25, minzeros=15, nskip=2, anyzeros=False)[source]¶
Select which ZERO exposure IDs to use for nightly bias
- Parameters:
- Options:
nzeros (int): number of zeros desired from valid all-cam observations to not worry about partials minzeros (int): minimum number of zeros to sufficiently compute a nightly bias nskip (int): number of initial zeros to skip
- Returns:
the list of expids selected
- Return type:
expids (array)
desispec.coaddition¶
Coadd spectra.
- desispec.coaddition.calc_mean_std_ra_dec(ras, decs)[source]¶
Calculate mean/std of ras, decs accounting for RA wraparound and cos(dec)
- Parameters:
ras (array) – input RA values in degrees
decs (array) – input declination values in degrees
Returns: mean_ra, std_ra, mean_dec, std_dec
where the means are in degrees and the standard deviations are in arcsec, including cos(dec) correction.
For efficiency, this does not try to handle dec= +/-90 poles correctly, nor arbitrarily large spreads of angles. i.e. this is ok for a mean of fiber positions scattered about a single target, but not for e.g. a science analysis of the central location of a cluster of galaxies.
- desispec.coaddition.coadd(spectra, cosmics_nsig=None, onetile=False)[source]¶
Coadd spectra for each target and each camera, modifying input spectra obj.
- Parameters:
spectra – desispec.spectra.Spectra object
- Options:
cosmics_nsig: float, nsigma clipping threshold for cosmics rays (default 4) onetile: bool, if True, inputs are from a single tile
- Notes: if onetile is True, additional tile-specific columns
like LOCATION and FIBER are included the FIBERMAP; otherwise these are only in the EXP_FIBERMAP since for the same target they could be different on different tiles.
- desispec.coaddition.coadd_cameras(spectra, cosmics_nsig=0.0, onetile=False)[source]¶
Return coadd across both exposures and cameras
- Parameters:
spectra – desispec.spectra.Spectra object
- Options:
cosmics_nsig: float, nsigma clipping threshold for cosmics rays onetile: bool, if True, inputs are from a single tile
If onetile is True, additional tile-specific columns like LOCATION and FIBER are included the FIBERMAP; otherwise these are only in the EXP_FIBERMAP since for the same target they could be different on different tiles.
Note: unlike coadd, this does not modify the input spectra object
- desispec.coaddition.coadd_fibermap(fibermap, onetile=False)[source]¶
Coadds fibermap
- Parameters:
fibermap (Table or ndarray) – fibermap of individual exposures
- Options:
onetile (bool): this is a coadd of a single tile, not across tiles
Returns: (coadded_fibermap, exp_fibermap) Tables
coadded_fibermap contains the coadded_fibermap for the columns that can be coadded, while exp_fibermap is the subset of columns of the original fibermap that can’t be meaningfully coadded because they are per-exposure quantities like FIBER_X.
If onetile is True, the coadded_fibermap includes additional columns like MEAN_FIBER_X that are meaningful if coadding a single tile, but not if coadding across tiles.
- desispec.coaddition.decorrelate_divide_and_conquer(Cinv, Cinvf, wavebin, flux, ivar, rdata)[source]¶
Decorrelate an inverse covariance using the matrix square root.
Implements the decorrelation part of the spectroperfectionism algorithm described in Bolton & Schlegel 2009 (BS) http://arxiv.org/abs/0911.2689.
with the divide and conquer approach, i.e. per diagonal block of the matrix, with an overlapping ‘skin’ from one block to another.
- Parameters:
Cinv – Square 2D array: input inverse covariance matrix
Cinvf – 1D array: input
wavebin – minimal size of wavelength bin in A, used to define the core and skin size
flux – 1D array: output flux (has to be allocated)
ivar – 1D array: output flux inverse variance (has to be allocated)
rdata – 2D array: output resolution matrix per diagonal (has to be allocated)
- desispec.coaddition.fast_resample_spectra(spectra, wave)[source]¶
Fast resampling of spectra file. The output resolution = Id. The neighboring flux bins are correlated.
- Parameters:
spectra – desispec.spectra.Spectra object
wave – 1D numy array with new wavelenght grid
- Returns:
desispec.spectra.Spectra object, resolution data=Id
- desispec.coaddition.get_resampling_matrix(global_grid, local_grid, sparse=False)[source]¶
Build the rectangular matrix that linearly resamples from the global grid to a local grid.
The local grid range must be contained within the global grid range.
- Parameters:
global_grid (numpy.ndarray) – Sorted array of n global grid wavelengths.
local_grid (numpy.ndarray) – Sorted array of m local grid wavelengths.
- Returns:
Array of (m,n) matrix elements that perform the linear resampling.
- Return type:
- desispec.coaddition.resample_spectra_lin_or_log(spectra, linear_step=0, log10_step=0, fast=False, wave_min=None, wave_max=None, nproc=1)[source]¶
Resampling of spectra file.
- Parameters:
spectra – desispec.spectra.Spectra object
linear_step – if not null the ouput wavelenght grid will be linear with this step
log10_step – if not null the ouput wavelenght grid will be logarthmic with this step
- Options:
fast: simple resampling. fast but at the price of correlated output flux bins and no information on resolution wave_min: if set, use this min wavelength wave_max: if set, use this max wavelength
- Returns:
desispec.spectra.Spectra object
- desispec.coaddition.spectroperf_resample_spectra(spectra, wave, nproc=1)[source]¶
Resampling of spectra file using the spectrophotometic approach
- Parameters:
spectra – desispec.spectra.Spectra object
wave – 1D numy array with new wavelenght grid
- Returns:
desispec.spectra.Spectra object
- desispec.coaddition.use_for_coadd(fiberstatus, band)[source]¶
Determine which exposures should be used for a per-camera coadd
- Parameters:
fiberstatus (array) – FIBERSTATUS bitmasks, one per exposure
band (str) – camera band, ‘b’, ‘r’, or ‘z’
Returns: boolean array of whether exposure should be used in coadd or not
This is factored into a separate function because it is used in coadd, coadd_cameras, and coadd_fibermap
desispec.correct_cte¶
Methods to fit CTE effects and remove them from images.
- desispec.correct_cte.add_cte(img, cte_transfer_func=None, **cteparam)[source]¶
Add CTE to a region affected by CTE.
Assumes all of img is affected.
- Parameters:
img (np.ndarray) – input image
cte_transfer_func (callable) – function describing how many electrons trap soaks up and emits
**cteparam (dictionary) – additional arguments to cte_transfer_func
- Returns:
np.ndarray
img with additional CTE applied
- desispec.correct_cte.apply_multiple_cte_effects(amp, locations, ctefuns)[source]¶
Apply a series of traps to an amp.
This function assumes that it is only being given a single amp that has some traps on it. The readout direction is always assumed to be toward smaller numbers on the second axis.
- Parameters:
amp (np.ndarray) – image of amplifier
locations (np.ndarray [int]) – locations of traps
ctefuns (callables) – functions that can be called to apply CTE to an amp
- Return type:
amplifier after applying traps
- desispec.correct_cte.chi_simplified_regnault(param, cleantraces=None, ctetraces=None, uncertainties=None)[source]¶
Chi loss function for a Regnault transfer function.
- desispec.correct_cte.correct_amp(image, ctefun, niter=1, apply_only=False)[source]¶
Correct an amp for CTE.
In the limit that CTE is perturbative, one can correct for CTE as follows: for i in range(niter): I’ = CTE(I) I = I - (I’ - I) This function implements that approach.
However, for large traps, this approach correlates the noise and leads to poor performance; we do not use it for DESI.
- desispec.correct_cte.correct_image(image, **kw)[source]¶
Correct an image for CTE.
This function wraps correct_amp, looping over amplifiers and reversing readout directions as appropriate. The approach taken here correlates the noise and is not actually used for DESI.
- Parameters:
- Returns:
outimage (desispec.io.image.Image)
CTE-corrected image
- desispec.correct_cte.correct_image_via_model(image, niter=10, cteparam=None)[source]¶
Correct for CTE via an image model.
The idea here is that you can roughly extract spectra from a CTE-affected image just by extracting as usual. You can then create a noise-free image from that extraction in the context of a PSF and traces. You can then apply CTE to that noise-free image. The difference between the CTE-affected image and the original image is a noise-free version of what CTE is doing to your data, which you can then subtract.
You can then re-extract from the corrected image, and repeat, improving the CTE correction.
As pseudocode, this corresponds to: for i in range(niter): M = get_model(I) M’ = CTE(M) I = I - (M’ - M) This function implements that approach.
- Parameters:
- Returns:
outimage – image after correction for CTE
- Return type:
Image
- desispec.correct_cte.fit_cte(images)[source]¶
Fits CTE models to a list of images.
This fits the parameters of the Regnault transfer function model to an image with a trap. It works by comparing an unaffected amplifier to an amplifier with a trap along the boundary between the two amplifiers, solving for the trap parameters that make the unaffected image + the CTE effect a good match to the CTE-affected image.
It will not work if both amplifiers have CTE, or if there are multiple CTE effects to solve for.
It assumes that amp A is across from amp C and amp B is across from amp D.
It would likely fail badly for a two-amp mode image.
- Parameters:
images (list) – A list of images. Usually these are all flats with different exposure lengths on the same device.
- Returns:
dict[dict[tuple(tuple, float)]]
The top-level dictionary has amplifier names as keys.
The next-level dictionary has a string identifying the specific trap as
keys.
The tuple contains the parameters of the CTE function as its first
element, and the chi^2 per degree of freedom of the fit as its second
element.
- desispec.correct_cte.fit_cte_night(night, camera)[source]¶
Fit the CTE parameters for a particular night.
- desispec.correct_cte.get_amps_and_cte(image, cteparam=None)[source]¶
Get amp and CTE information from image metadata.
- Parameters:
- Returns:
amp_regions, cte_regions
amp_regions (dict) – dictionary of slices defining each amp
cte_regions (dict) – dictionary. The keys are the names of the amplifier. Each entry is a list containing one entry for each trap. on that amplifier. Each item in the list is dictionary containing the start location, stop location, and a callable giving the CTE function for a trap.
- desispec.correct_cte.get_cte_images(night, camera)[source]¶
Get the images needed for a CTE fit for a particular night.
This function looks up the appropriate exposure tables to find the CTE-detection image and the previous image, which is usually a normal flat field image.
- desispec.correct_cte.get_image_model(preproc, psf=None)[source]¶
Compute model for an image using aperture extraction.
This computes a simple model for an image based on an aperture extraction.
- Parameters:
preproc (Image) – Image to model
- Returns:
np.ndarray
Model image
- desispec.correct_cte.get_rowbyrow_image_model(preproc, fibermap=None, spectral_smoothing_sigma_length=31, nspec=500, psf=None)[source]¶
Compute row-by-row image model.
This model uses a simultaneous PSF fit in each row to get better performance than get_image_model at the expense of reduced speed. The extracted fluxes are then combined with the PSF to produce a 2D model image.
- Parameters:
- Returns:
np.ndarray
Model image.
- desispec.correct_cte.simplified_regnault(pixval, in_trap, amplitude, fracleak)[source]¶
CTE transfer function of Regnault+.
The model is transfer = in_trap * fracleak - pixval * (1 - in_trap / amplitude) with slight elaborations to prevent more electrons than exist from entering or leaving the trap.
- Parameters:
- Returns:
int
number of electrons that leak out of the trap into the pixel
desispec.cosmics¶
Utility functions to find cosmic rays
- desispec.cosmics._reject_cosmic_rays_ala_sdss_single(pix, ivar, selection, psf_gradients, nsig, cfudge, c2fudge)[source]¶
Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)
This routine is a single call, similar to IDL routine reject_cr_single Called by reject_cosmic_rays_ala_sdss with an iteration
Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image
A faster version of this routine using numba in implemented in _reject_cosmic_rays_ala_sdss_single_numba
- Parameters:
pix – input desispec.Image.pix (counts in pixels, 2D image)
ivar – inverse variance of pix
selection – array of booleans
psf_gradients – 1D array of size 4, for 4 axes: horizontal,vertical and 2 diagonals
nsig – number of sigma above background required
cfudge – number of sigma inconsistent with PSF required
c2fudge – fudge factor applied to PSF
- desispec.cosmics._reject_cosmic_rays_ala_sdss_single_numba(pix, ivar, selection, psf_gradients, nsig, cfudge, c2fudge)¶
Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)
This routine is a single call, similar to IDL routine reject_cr_single Called by reject_cosmic_rays_ala_sdss with an iteration
Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image
This routine is much faster than _reject_cosmic_rays_ala_sdss_single (if you have numba installed, otherwise it is catastrophically slower)
- Parameters:
pix – input desispec.Image.pix (counts in pixels, 2D image)
ivar – inverse variance of pix
selection – array of booleans
psf_gradients – 1D array of size 4, for 4 axes: horizontal,vertical and 2 diagonals
nsig – number of sigma above background required
cfudge – number of sigma inconsistent with PSF required
c2fudge – fudge factor applied to PSF
- desispec.cosmics.reject_cosmic_rays(img, nsig=5.0, cfudge=3.0, c2fudge=0.9, niter=100, dilate=True)[source]¶
Cosmic ray rejection Input is a pre-processed image : desispec.Image The image mask is modified
- Parameters:
img – input desispec.Image
- desispec.cosmics.reject_cosmic_rays_1d(frame, nsig=3, psferr=0.05)[source]¶
Use resolution matrix in frame to detect spikes in the spectra that are narrower than the PSF, and mask them
- desispec.cosmics.reject_cosmic_rays_ala_sdss(img, nsig=6.0, cfudge=3.0, c2fudge=0.5, niter=6, dilate=True)[source]¶
Cosmic ray rejection following the implementation in SDSS/BOSS. (see idlutils/src/image/reject_cr_psf.c and idlutils/pro/image/reject_cr.pro)
This routine is calling several times reject_cosmic_rays_ala_sdss_single, similar to IDL routine reject_cr. There is an optionnal dilatation of the mask by one pixel, as done in sdssproc.pro for SDSS
Input is a pre-processed image : desispec.Image Ouput is a rejection mask of the same size as the image
- Parameters:
img – input desispec.Image
nsig – number of sigma above background required
cfudge – number of sigma inconsistent with PSF required
c2fudge – fudge factor applied to PSF
niter – number of iterations on neighboring pixels of rejected pixels
dilate – force +1 pixel dilation of rejection mask
desispec.darktrail¶
Utility functions to correct dark trails found for some LBNL CCD amplifiers due to an imperfect CDS (correlated double sampling) suppression of the reset level. The cause of the effect is unknown (as this is written), and we have not been able to correcting it with adjustment of the readout electronic parameters.
The function here are used to do a software correction of the effect.
- desispec.darktrail.correct_dark_trail(image, xyslice, left, width, amplitude)[source]¶
remove the dark trails from a preprocessed image with a processing of the form:
image[j,i] += sum_{k<i} image[j,k] * a * |i-k|/width * exp( -|i-k|/width)
- Parameters:
image – desispec.Image class instance
xyslice – tuple of python slices (yy,xx) where yy is indexing CCD rows (firt index in 2D numpy array) and xx is the other
left – if true , the trails are to the left of the image spots, as is the case for amplifiers B and D.
width – width parameter in unit of pixels
amplitude – amplitude of the dark trail
desispec.database¶
Tools for loading DESI data into databases.
At the present time, this covers:
Raw metadata database.
Redshift results database (a.k.a “SpectraDB”).
It does not include:
Pipeline processing status database.
Imaging and targeting databases.
desispec.database.duplicates¶
Find rows in a file that contain the same value in a certain column. The file in question could potentially be very large.
- desispec.database.duplicates.find_duplicate_rows(data, column)[source]¶
Find rows in data for which column has the same value.
- Parameters:
data (
astropy.table.Table
) – Data set to analyze.column (
str
) – Search for duplicates in this column.
- Returns:
A mapping of unique values of column to row numbers in data.
- Return type:
- desispec.database.duplicates.get_options(*args)[source]¶
Parse command-line options.
- Parameters:
args (iterable) – If arguments are passed, use them instead of
sys.argv
.- Returns:
The parsed options.
- Return type:
- desispec.database.duplicates.main()[source]¶
Entry point for command-line script.
- Returns:
An integer suitable for passing to
sys.exit()
.- Return type:
desispec.database.metadata¶
Code for interacting with the file metadatabase.
- class desispec.database.metadata.Brick(*args: Any, **kwargs: Any)[source]¶
Representation of a region of the sky.
- class desispec.database.metadata.BrickStatus(*args: Any, **kwargs: Any)[source]¶
Representation of the status of a particular
Brick
.
- class desispec.database.metadata.ExposureFlavor(*args: Any, **kwargs: Any)[source]¶
List of exposure flavors. Used to constrain the possible values.
- class desispec.database.metadata.Frame(*args: Any, **kwargs: Any)[source]¶
Representation of a particular pointing, exposure, spectrograph and band (a.k.a. ‘channel’ or ‘arm’).
- class desispec.database.metadata.FrameStatus(*args: Any, **kwargs: Any)[source]¶
Representation of the status of a particular
Frame
.
- class desispec.database.metadata.Night(*args: Any, **kwargs: Any)[source]¶
List of observation nights. Used to constrain the possible values.
- class desispec.database.metadata.Status(*args: Any, **kwargs: Any)[source]¶
List of possible processing statuses.
- class desispec.database.metadata.Tile(*args: Any, **kwargs: Any)[source]¶
Representation of a particular pointing of the telescope.
- property area¶
Area of the tile in steradians.
- brick_offset(brick)[source]¶
Offset a brick in the same way as a tile.
- Parameters:
brick (
Brick
) – A brick.- Returns:
A tuple containing the shifted ra1 and ra2.
- Return type:
tuple()
- property circum_square¶
Defines a square-like region on the sphere which circumscribes the tile.
- property cos_radius¶
Cosine of the radius, precomputed for speed.
- overlapping_bricks(session, map_petals=False)[source]¶
Perform a geometric calculation to find bricks that overlap a tile.
- Parameters:
session (
sqlalchemy.orm.session.Session
) – Database connection.map_petals (bool, optional) – If
True
a map of petal number to a list of overlapping bricks is returned.
- Returns:
If map_petals is
False
, a list ofPolygon
objects. Otherwise, adict
mapping petal number to theBrick
objects that overlap that petal.- Return type:
- property radius¶
Radius of tile in degrees.
- desispec.database.metadata.get_all_tiles(session, obs_pass=0, limit=0)[source]¶
Get all tiles from the database.
- desispec.database.metadata.load_data(session, datapath)[source]¶
Load a night or multiple nights into the frame table.
- desispec.database.metadata.load_simulated_data(session, obs_pass=0)[source]¶
Load simulated frame and brick data.
- Parameters:
session (
sqlalchemy.orm.session.Session
) – Database connection.obs_pass (
int
, optional) – If set, only simulate one pass.
- desispec.database.metadata.main()[source]¶
Entry point for command-line script.
- Returns:
An integer suitable for passing to
sys.exit()
.- Return type:
desispec.database.redshift¶
Code for loading a spectroscopic production database. This includes both targeting and redshift data.
Notes
EDR development:
Load as much data as possible from patched fiberassign files.
If necessary, create intermediate files for ingestion by running the lsdr9-photometry code.
In the tractorphot files, TARGETID really is unique. However, some TARGETIDs are not present because they do not match an object to within 1 arcsec. For these, fill in what can be filled in from target files or fiberassign files.
For every duplicate potential targetid, use only the most recent desitarget version containing that targetid, rather than the default associated with the fiberassign file.
Read redshifts from
zall-*.fits
files to get the*_primary
columns, or run the primary generation code on the data (find_primary_spectra()
).
Future devlopment:
Plan for how to support fuji+guadalupe combined analysis. May need to look into cross-schema views, or daughter tables that inherit from both schemas.
Anticipating loading afterburners and VACs into the database.
Load redshifts from all redrock files in
tiles/cumulative
, rather than from theztile-*-cumulative.fits
summary file.
- class desispec.database.redshift.Exposure(*args: Any, **kwargs: Any)[source]¶
Representation of the EXPOSURES HDU in the exposures file.
Notes
The column
program
is filled in viafaflavor2program()
.
- class desispec.database.redshift.Fiberassign(*args: Any, **kwargs: Any)[source]¶
Representation of the FIBERASSIGN table in a fiberassign file.
Notes
Targets are assigned to a
location
. Alocation
happens to correspond to afiber
, but this correspondence could change over time, and therefore should not be assumed to be a rigid 1:1 mapping.PLATE_RA
,PLATE_DEC
are sometimes missing. These can be copies ofTARGET_RA
,TARGET_DEC
, but in principle they could be different if chromatic offsets in targeting positions were ever implemented.
- class desispec.database.redshift.Frame(*args: Any, **kwargs: Any)[source]¶
Representation of the FRAMES HDU in the exposures file.
Notes
The column
frameid
is a combination ofexpid
and the camera name:frameid = 100*expid + cameraid(camera)
where
cameraid()
isdesispec.database.util.cameraid()
.
- class desispec.database.redshift.Photometry(*args: Any, **kwargs: Any)[source]¶
Contains only photometric quantities associated with a
TARGETID
.This table is deliberately designed so that
TARGETID
can serve as a primary key. Any quantities created or modified by desitarget are defined in theTarget
class.However we avoid the use of the term “tractor” for this table, because not every target will have tractor photometry,
Notes
The various
LC
(light curve) columns, which are vector-valued, are not yet implemented.
- class desispec.database.redshift.Potential(*args: Any, **kwargs: Any)[source]¶
Representation of the POTENTIAL_ASSIGNMENTS table in a fiberassign file.
- class desispec.database.redshift.SchemaMixin¶
Mixin class to allow schema name to be changed at runtime. Also automatically sets the table name.
- class desispec.database.redshift.Target(*args: Any, **kwargs: Any)[source]¶
Representation of the pure-desitarget quantities in the
TARGETPHOT
table in the targetphot files.
- class desispec.database.redshift.Tile(*args: Any, **kwargs: Any)[source]¶
Representation of the tiles file.
Notes
Most of the data that are currently in the tiles file are derivable from the exposures table with much greater precision:
CREATE VIEW f5.tile AS SELECT tileid, -- SURVEY, FAPRGRM, FAFLAVOR? COUNT(*) AS nexp, SUM(exptime) AS exptime, MIN(tilera) AS tilera, MIN(tiledec) AS tiledec, SUM(efftime_etc) AS efftime_etc, SUM(efftime_spec) AS efftime_spec, SUM(efftime_gfa) AS efftime_gfa, MIN(goaltime) AS goaltime, -- OBSSTATUS? SUM(lrg_efftime_dark) AS lrg_efftime_dark, SUM(elg_efftime_dark) AS elg_efftime_dark, SUM(bgs_efftime_bright) AS bgs_efftime_bright, SUM(lya_efftime_dark) AS lya_efftime_dark, -- GOALTYPE? MIN(mintfrac) AS mintfrac, MAX(night) AS lastnight FROM f5.exposure GROUP BY tileid;
However because of some unresolved discrepancies, we’ll just load the full tiles file for now.
- class desispec.database.redshift.Zpix(*args: Any, **kwargs: Any)[source]¶
Representation of the
ZCATALOG
table in zpix files.
- class desispec.database.redshift.Ztile(*args: Any, **kwargs: Any)[source]¶
Representation of the
ZCATALOG
table in ztile files.
- desispec.database.redshift._add_ls_id(data)[source]¶
Add LS_ID to targetphot data.
- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
Updated data table.
- Return type:
- desispec.database.redshift._deduplicate_targetid(data)[source]¶
Find targetphot rows that are not already loaded into the Photometry table and resolve any duplicate TARGETID.
- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
An array of rows that are safe to load.
- Return type:
numpy.array
- desispec.database.redshift._frameid(data)[source]¶
Update the
frameid
column.- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
Updated data table.
- Return type:
- desispec.database.redshift._remove_loaded_targetid(data)[source]¶
Remove rows with TARGETID already loaded into the database.
- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
An array of rows that are safe to load.
- Return type:
numpy.array
- desispec.database.redshift._remove_loaded_unique_id(data)[source]¶
Remove rows with UNIQUE ID already loaded into the database.
- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
An array of rows that are safe to load.
- Return type:
numpy.array
- desispec.database.redshift._survey_program(data)[source]¶
Add
SURVEY
,PROGRAM
,SPGRP
columns to zpix and ztile tables.- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
Updated data table.
- Return type:
- Raises:
KeyError – If a necessary header could not be found.
- desispec.database.redshift._target_unique_id(data)[source]¶
Add composite
ID
column for later conversion.- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
Updated data table.
- Return type:
- desispec.database.redshift._tileid(data)[source]¶
Update the
tileid
column. Also check for the presence ofPLATE_RA
,PLATE_DEC
.- Parameters:
data (
astropy.table.Table
) – The initial data read from the file.- Returns:
Updated data table.
- Return type:
- desispec.database.redshift.get_options(*args)[source]¶
Parse command-line options.
- Parameters:
args (iterable) – If arguments are passed, use them instead of
sys.argv
.- Returns:
The parsed options.
- Return type:
- desispec.database.redshift.load_file(filepaths, tcls, hdu=1, preload=None, expand=None, insert=None, convert=None, index=None, rowfilter=None, q3c=None, chunksize=50000, maxrows=0)[source]¶
Load data file into the database, assuming that column names map to database column names with no surprises.
- Parameters:
filepaths (
str
orlist
) – Full path to the data file or set of data files.tcls (
sqlalchemy.ext.declarative.api.DeclarativeMeta
) – The table to load, represented by its class.hdu (
int
orstr
, optional) – Read a data table from this HDU (default 1).preload (callable, optional) – A function that takes a
Table
as an argument. Use this for more complicated manipulation of the data before loading, for example a function that depends on multiple columns. The return value should be the updated Table.expand (
dict
, optional) – If set, map FITS column names to one or more alternative column names.insert (
dict
, optional) – If set, insert one or more columns, before an existing column. The existing column will be copied into the new column(s).convert (
dict
, optional) – If set, convert the data for a named (database) column using the supplied function.index (
str
, optional) – If set, add a column that just counts the number of rows.rowfilter (callable, optional) – If set, apply this filter to the rows to be loaded. The function should return
bool
, withTrue
meaning a good row.q3c (
str
, optional) – If set, create q3c index on the table, using the RA column named q3c.chunksize (
int
, optional) – If set, load database chunksize rows at a time (default 50000).maxrows (
int
, optional) – If set, stop loading after maxrows are loaded. Alteratively, set maxrows to zero (0) to load all rows.
- Returns:
The grand total of rows loaded.
- Return type:
- desispec.database.redshift.load_redrock(datapath=None, hdu='REDSHIFTS', q3c=False)[source]¶
Load redrock files into the zcat table.
This function is deprecated since there should now be a single redshift catalog file.
- desispec.database.redshift.main()[source]¶
Entry point for command-line script.
- Returns:
An integer suitable for passing to
sys.exit()
.- Return type:
- desispec.database.redshift.setup_db(options=None, **kwargs)[source]¶
Initialize the database connection.
- Parameters:
options (
argpare.Namespace
) – Parsed command-line options.kwargs (keywords) – If present, use these instead of options. This is more user-friendly than setting up a
Namespace
object in, e.g. a Jupyter Notebook.
- Returns:
True
if the configured database is a PostgreSQL database.- Return type:
- desispec.database.redshift.update_truth(filepath, hdu=2, chunksize=50000, skip=('SLOPES', 'EMLINES'))[source]¶
Add data from columns in other HDUs of the Truth table.
- Parameters:
filepath (
str
) – Full path to the data file.hdu (
int
orstr
, optional) – Read a data table from this HDU (default 2).chunksize (
int
, optional) – If set, update database chunksize rows at a time (default 50000).skip (
tuple()
, optional) – Do not load columns with these names (default,('SLOPES', 'EMLINES')
)
desispec.database.util¶
Classes and functions for use by all database code.
- desispec.database.util.cameraid(camera)[source]¶
Converts camera (e.g. ‘b0’) to an integer in a simple but ultimately arbitrary way.
- desispec.database.util.convert_dateobs(timestamp, tzinfo=None)[source]¶
Convert a string timestamp into a
datetime.datetime
object.- Parameters:
timestamp (
str
) – Timestamp in string format.tzinfo (
datetime.tzinfo
, optional) – If set, add time zone to the timestamp.
- Returns:
The converted timestamp.
- Return type:
- desispec.database.util.decode_surveyid(surveyid)[source]¶
Converts surveyid to its corresponding name.
- desispec.database.util.fiberassignid(targetid, tileid, location)[source]¶
Convert inputs into an arbitrary large integer.
- desispec.database.util.frameid(expid, camera)[source]¶
Converts the pair expid, camera into an arbitrary integer suitable for use as a primary key.
- desispec.database.util.parse_pgpass(hostname='nerscdb03.nersc.gov', username='desidev_admin')[source]¶
Read a
~/.pgpass
file.
- desispec.database.util.programid(program)[source]¶
Converts program (e.g. ‘bright’) to an integer in a simple but ultimately arbitrary way.
- desispec.database.util.spgrpid(spgrp)[source]¶
Converts spgrp (e.g. ‘cumulative’) to an integer in a simple but ultimately arbitrary way.
- desispec.database.util.surveyid(survey)[source]¶
Converts survey (e.g. ‘main’) to an integer in a simple but ultimately arbitrary way.
- desispec.database.util.targetphotid(targetid, tileid, survey)[source]¶
Convert inputs into an arbitrary large integer.
- desispec.database.util.zpixid(targetid, survey, program)[source]¶
Convert inputs into an arbitrary large integer.
- desispec.database.util.ztileid(targetid, spgrp, spgrpval, tileid)[source]¶
Convert inputs into an arbitrary large integer.
desispec.desi_bias_dark_1d_model¶
Usage:
python3 desi_bias_dark_1d_model.py
desispec.desi_create_bias_dark¶
Usage Manual¶
Pipeline for gnerating new bias+dark files
Input: night array to use
Example
python3 desi_create_bias_dark.py -n 20200729 20200730 -r 0060633 0060634 0060635 0060636 0060637 0060638 0060639 0060640 0060641 0060642
desispec.desi_dashboard¶
- class desispec.desi_dashboard.DESI_DASHBOARD[source]¶
Code to generate the statistic of desi_pipe production status Usage: python3 desi_dashboard.py –prod realtime10 –prod_dir /global/cscratch1/sd/zhangkai/desi/ –output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_dashboard/ –output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_dashboard/
desispec.desi_proc_time_distribution¶
- class desispec.desi_proc_time_distribution.DESI_PROC_TIME_DISTRIBUTION[source]¶
Code to generate a webpage for monitoring of desi_dailyproc production status
Normal Mode:
desi_proc_time_distribution -n all --n_night 3 --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_proc_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_proc_dashboard/ python3 desi_proc_time_distribution.py -n all --n_night 3 --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_proc_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_proc_dashboard/
Cron job script:
*/30 * * * * /global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/bin/python3 /global/project/projectdirs/desi/users/zhangkai/desi/code/desispec/py/desispec/workflow/proc_dashboard_funcs.py -n all --n_night 30 --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_proc_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_proc_dashboard/ >/global/project/projectdirs/desi/users/zhangkai/desi_proc_dashboard.log 2>/global/project/projectdirs/desi/users/zhangkai/desi_proc_dashboard.err &
- _add_html_table(table, night)[source]¶
Add a collapsible and extendable table to the html file for one specific night
- Parameters:
table – the table generated by ‘calculate_one_night’
night – like 20200131
- Returns:
The string to be added to the html file
- _initialize_page()[source]¶
Initialize the html file for showing the statistics, giving all the headers and CSS setups.
- calculate_one_night(night)[source]¶
For a given night, return the file counts and other other information for each exposure taken on that night
- Parameters:
night –
- Returns:
A dictionary containing the statistics with expid as key name:
FLAVOR: FLAVOR of this exposure OBSTYPE: OBSTYPE of this exposure EXPTIME: Exposure time SPECTROGRAPHS: a list of spectrographs used n_spectrographs: number of spectrographs n_psf: number of PSF files n_ff: number of fiberflat files n_frame: number of frame files n_sframe: number of sframe files n_cframe: number of cframe files n_sky: number of sky files
desispec.efftime¶
Effective exposure time formulae.
- desispec.efftime.compute_efftime(table, kterm=0.114, ebv_r_coeff=2.165, fiber_diameter_arcsec=1.52)[source]¶
Computes the effective exposure times using transparency and fiber acceptance from the GFAs offline analysis, and the sky magnitudes from the spectroscopy. Uses the formulae described in https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed for the dark, bright and backup programs.
- Parameters:
table – astropy.table.Table with columns - EXPTIME exposure time in seconds - SKY_MAG_R_SPEC AB magnitude per arcsec2 of sky, in decam-r filter - EBV E(B-V) from SFD map - TRANSPARENCY_GFA transparency (number between 0 and 1, ~1 for photometric nights irrespectively of AIRMASS) - AIRMASS (airmass >=1) - FIBERFAC_GFA PMGSTARS forced photometry amount of light in 1.52 asec diameter aperture normalized to nominal, assuming a point source profile - FIBERFAC_ELG_GFA PMGSTARS forced photometry amount of light in 1.52 asec diameter aperture normalized to nominal, assuming an ELG-like profile (r_half = 0.45 asec exponential) - FIBERFAC_BGS_GFA PMGSTARS forced photometry amount of light in 1.52 asec diameter aperture normalized to nominal, assuming a BGS-like profile (r_half = 1.5 asec de Vaucouleurs) - FIBER_FRACFLUX_GFA fraction of light in fiber for point source - FIBER_FRACFLUX_ELG_GFA fraction of light in fiber for typical ELG source - FIBER_FRACFLUX_BGS_GFA fraction of light in fiber for typical BGS source
desispec.emlinefit¶
Routines for desi_emlinefit_afterburner.
- desispec.emlinefit.emlines_gaussfit(emname, zspec, waves, fluxes, ivars, rf_fit_hw=40, min_rf_fit_hw=20, rf_cont_w=200, p0_sigma=2.5, p0_flux=10, p0_share=0.58, min_sigma=1e-05, max_sigma=10.0, min_flux=-1000000000.0, max_flux=1000000000.0, min_share=0.1, max_share=1, log=None)[source]¶
Fits the [OII] doublet line profile with 2 related Gaussians.
- Parameters:
emname – “OII” or “OIII” or “HALPHA”, “HBETA”, “HGAMMA”, “HDELTA” (string)
zspec – redshift (float)
waves – wavelength in Angstroms (numpy array)
fluxes – flux observed in the broad band (in erg/s/cm2/A) (numpy array)
ivars – inverse variance on the flux (numpy array)
rf_fit_hw (optional, defaults to 40) – rest-frame wavelength width (in A) used for fitting on each side of the line (float)
min_rf_fit_hw (optional, defaults to 20) – minimum requested rest-frame width (in A) on each side of the line to consider the fitting (float)
rf_cont_w (optional, defaults to 200) – rest-frame wavelength extent (in A) to fit the continuum (float)
p0_sigma (optional, defaults to 2.5) – initial guess on the line width in A (float)
p0_flux (optional, defaults to 0.1) – initial guess on the line flux in 1e-17 * erg/cm2/s (float)
p0_share (optional, defaults to 0.58) – initial guess on the share between the two [OII] lines (float)
min_sigma (optional, defaults to 1e-5) – minimum allowed value for the line width in A (float)
max_sigma (optional, defaults to 10.) – maximum allowed value for the line width in A (float)
min_flux (optional, defaults to 1e-5) – minimum allowed value for the flux in 1e-17 * erg/cm2/s (float)
max_flux (optional, defaults to 1e9) – maximum allowed value for the flux in 1e-17 * erg/cm2/s (float)
min_share (optional, defaults to 1e-1) – minimum allowed value for the share (float)
max_share (optional, defaults to 1) – maximum allowed value for the share (float)
log (optional, defaults to get_logger()) – Logger object
- Returns:
a dictionary with various quantities, noticely “FLUX” and “FLUX_IVAR” (dictionary of floats) list of all keys:
CHI2, NDOF: *reduced* chi2 and nb of degrees of freedom CONT, CONT_IVAR: continuum in 1e-17 * erg/cm2/s/A FLUX, FLUX_IVAR: flux in 1e-17 * erg/cm2/s SIGMA, SIGMA_IVAR: line width in A (observed frame) SHARE, SHARE_IVAR: f1/(f0+f1) for OII and OIII doublets EW, EW_IVAR: rest-frame equivalent width (in A) waves: wavelength values (in A) used for the fitting (numpy array of floats) fluxes: flux values (in 1e-17 * erg/cm2/s/A) used for the fitting (numpy array of floats) ivars: ivar values used for the fitting (numpy array of floats) models: model flux values (in 1e-17 * erg/cm2/s/A) from the fit (numpy array of floats)
succeed: did the fit succeed? (boolean)
- Return type:
emdict
Note
Adapted/simplified from elgredshiftflag from J. Comparat (used for eBOSS/ELG): https://svn.sdss.org/repo/eboss/elgredshiftflag/
Returns np.nan in emdict (and NDOF=-99) if not enough pixels to fit or if fit fails.
Default settings designed for ELGs (e.g. max_sigma); need to re-assess if run on other targets.
For “OII”, let the doublet line ratio free during the fit.
For “OIII”, fits the 4960 and 5007 lines with a fixed line ratio.
For the Balmer lines, SHARE is not fitted and set to np.nan.
- desispec.emlinefit.get_emlines(zspecs, waves, fluxes, ivars, emnames=['OII', 'HDELTA', 'HGAMMA', 'HBETA', 'OIII', 'HALPHA'], rf_fit_hw=40, min_rf_fit_hw=20, rf_cont_w=200, log=None)[source]¶
Get Gaussian-fitted emission line fluxes.
- Parameters:
zspecs – redshifts (numpy array of shape (Nspec))
waves – wavelengths (numpy array of shape (Nwave))
fluxes – fluxes (numpy array of shape (Nspec, Nwave))
ivars – inverse variances (numpy array of shape (Nspec, Nwave))
emnames (optional, defaults to ["OII", "HDELTA", "HGAMMA", "HBETA", "OIII", "HALPHA"]) – list of lines to fit (list of string)
rf_fit_hw (optional, defaults to 40) – rest-frame wavelength width (in A) used for fitting on each side of the line (float)
min_rf_fit_hw (optional, defaults to 20) – minimum requested rest-frame width (in A) on each side of the line to consider the fitting (float)
rf_cont_w (optional, defaults to 200) – rest-frame wavelength extent (in A) to fit the continuum (float)
log (optional, defaults to get_logger()) – Logger object
- Returns:
- a dictionary with a subdictionary for each emname in emnames, with:,
CHI2, NDOF: reduced chi2 and nb of degrees of freedom CONT, CONT_IVAR: continuum in 1e-17 * erg/cm2/s/A FLUX, FLUX_IVAR: flux in 1e-17 * erg/cm2/s SIGMA, SIGMA_IVAR: line width in A (observed frame) SHARE, SHARE_IVAR: f1/(f0+f1) for OII and OIII doublets EW, EW_IVAR: rest-frame equivalent width in A waves, fluxes, ivars, models: data used for fitting + fitted model
- Return type:
emdict
- desispec.emlinefit.get_rf_em_waves(emname)[source]¶
Returns the rest-frame, vacuum, wavelengths.
- Parameters:
emname – “OII”, “HDELTA”, “HGAMMA”, “HBETA”, “OIII”, “HALPHA” (string)
from – “OII”, “HDELTA”, “HGAMMA”, “HBETA”, “OIII”, “HALPHA” (string)
- Returns:
rest-frame wavelength(s) (np array)
- Return type:
rf_em_waves
Notes
For OII and OIII returns a two-elements array; one-element array otherwise. https://github.com/desihub/fastspecfit/blob/60393296e0cc466858f70a5d021d02693eff9375/py/fastspecfit/data/emlines.ecsv
desispec.exposure_qa¶
Utility functions to compute an exposure QA scores.
- desispec.exposure_qa.compute_exposure_qa(night, expid, specprod_dir=None)[source]¶
Computes the exposure_qa
- Parameters:
night – int, YYYYMMDD
expid – int, exposure id
specprod_dir – str, optional, specify the production directory. default is $DESI_SPECTRO_REDUX/$SPECPROD
- Returns:
two tables (astropy.table.Table), fiberqa (with one row per target and at least a TARGETID column) and petalqa (with one row per petal and at least a PETAL_LOC column)
- desispec.exposure_qa.get_qa_params()[source]¶
Returns a dictionnary with the content of data/qa/qa-params.yaml
desispec.fiberbitmasking¶
Functions to properly take FIBERSTATUS into account in the variances for data reduction
- desispec.fiberbitmasking.get_all_nonamp_fiberbitmask_val()[source]¶
Return a mask for all fatally bad FIBERSTATUS bits except BADAMPB/R/Z
Note
Does not include STUCKPOSITIONER or RESTRICTED, which could still be on a valid sky location, or even a target for RESTRICTED. Also does not include POORPOSITION which is bad for stdstars but not necessarily fatal for otherwise processing a normal fiber.
- desispec.fiberbitmasking.get_fiberbitmask_comparison_value(kind, band)[source]¶
Takes a string argument and returns a 32-bit integer representing the logical OR of all relevant fibermask bits for that given reduction step
- Parameters:
kind – str : string designating which combination of bits to use based on the operation. Possible values are “all”, “sky” (or “skysub”), “flat”, “flux” (or “fluxcalib”), “star” (or “stdstars”)
band – str : BADAMP band bits to set. Values include ‘b’, ‘r’, ‘z’, or combinations thereof such as ‘brz’
- Returns:
- 32 bit bitmask corresponding to the fiberbitmask of the desired kind
in the desired cameras (bands).
- Return type:
bitmask
- desispec.fiberbitmasking.get_fiberbitmasked_frame(frame, bitmask=None, ivar_framemask=True)[source]¶
Wrapper script of get_fiberbitmasked_frame_arrays that will return a modified version of the frame instead of just the flux and ivar
Note
The input “frame” variable itself is modified and returned, not a copy.
- desispec.fiberbitmasking.get_fiberbitmasked_frame_arrays(frame, bitmask=None, ivar_framemask=True, return_mask=False)[source]¶
Function that takes a frame object and a bitmask and returns ivar (and optionally mask) array(s) that have fibers with offending bits in fibermap[‘FIBERSTATUS’] set to 0 in ivar and optionally flips a bit in mask.
- Parameters:
frame – frame object
bitmask – int32 or list/array of int32’s derived from desispec.maskbits.fibermask OR string indicating a keyword for get_fiberbitmask_comparison_value()
ivar_framemask – bool (default=True), tells code whether to multiply the output variance by (frame.mask==0)
return_mask – bool, (default=False). Returns the frame.mask with the logic of FIBERSTATUS applied.
- Returns:
- frame.ivar where the fibers with FIBERSTATUS & bitmask > 0
set to zero ivar
- mask: (optional) frame.mask logically OR’ed with BADFIBER bit in cases with
a bad FIBERSTATUS
- Return type:
ivar
Examples
Example bitmask list:
bitmask = [fmsk.BROKENFIBER,fmsk.UNASSIGNED,fmsk.BADFIBER, fmsk.BADTRACE,fmsk.MANYBADCOL, fmsk.MANYREJECTED] bitmask = get_fiberbitmask_comparison_value(kind='fluxcalib', band='brz') bitmask = 'fluxcalib' bitmask = 4128780
desispec.fibercrosstalk¶
Utility functions to correct for the fibercrosstalk
- desispec.fibercrosstalk.compute_contamination(frame, dfiber, kernel, params, xyset, fiberflat=None, fractional_error=0.1)[source]¶
Computes the contamination of a frame from a given fiber offset
- Parameters:
frame – a desispec.frame.Frame object
dfiber – int, fiber offset (-2,-1,1,2)
kernel – 1D numpy array, convolution kernel
params – nested dictionnary, parameters of the crosstalk model
xyset –
desispec.xytraceset.XYTraceSet object with trace coordinates to shift the spectra
fiberflat : desispec.fiberflat.FiberFlat object, optional if the frame has already been fiber flatfielded fractionnal_error : float, optional consider this systematic relative error on the correction
- Returns:
the contamination of the frame, 2D numpy array of same shape as frame.flux, and its variance
- Return type:
contamination , contamination_var
- desispec.fibercrosstalk.compute_crosstalk_kernels(max_fiber_offset=2, fiber_separation_in_pixels=7.3, asymptotic_power_law_index=2.5)[source]¶
Computes the fiber crosstalk convolution kernels assuming a power law PSF tail
- Parameters:
max_fiber_offset – positive int, maximum fiber offset, 2 by default
optional – positive int, maximum fiber offset, 2 by default
fiber_separation_in_pixels – float, distance between neighboring fiber traces in the CCD in pixels, default=7.3
optional – float, distance between neighboring fiber traces in the CCD in pixels, default=7.3
asymptotic_power_law_index – float, power law index of PSF tail
optional – float, power law index of PSF tail
- Returns:
A dictionnary of kernels, with key the positive fiber offset 1,2,…. Each entry is an 1D array.
- desispec.fibercrosstalk.correct_fiber_crosstalk(frame, fiberflat=None, xyset=None)[source]¶
Apply a fiber cross talk correction. Modifies frame.flux and frame.ivar.
- Parameters:
frame – desispec.frame.Frame object
fiberflat – desispec.fiberflat.FiberFlat object
optional – desispec.fiberflat.FiberFlat object
xyset – desispec.xytraceset.XYTraceSet object with trace coordinates to shift the spectra (automatically found with calibration finder otherwise)
optional – desispec.xytraceset.XYTraceSet object with trace coordinates to shift the spectra (automatically found with calibration finder otherwise)
- desispec.fibercrosstalk.eval_crosstalk(camera, wave, fibers, dfiber, params, apply_scale=True, nfiber_per_bundle=25)[source]¶
Computes the crosstalk as a function of wavelength from a fiber offset dfiber (positive and negative) for an input set of fibers
- Parameters:
camera – str, camera identifier (b8,r7,z3, …)
wave – 1D array, wavelength
fibers – list or 1D array of int, list of contaminated fibers
dfiber – int, positive or negative fiber offset, contaminating fibers = contaminated fibers + dfiber
params – nested dictionnary, parameters of the crosstalk model
apply_scale – boolean, optional apply or not the scale factor if found in the list of parameters
nfiber_per_bundle – int, optional number of fibers per bundle, only the fibers in the same bundle are considered
- Returns:
2D array of crosstalk fraction (between 0 and 1) of shape ( len(fibers),len(wave) )
- desispec.fibercrosstalk.read_crosstalk_parameters()[source]¶
Reads the crosstalk parameters in desispec/data/fiber-crosstalk.yaml
- Returns:
nested dictionary with parameters per camera
desispec.fiberflat¶
Utility functions to compute a fiber flat correction and apply it We try to keep all the (fits) io separated.
- desispec.fiberflat.apply_fiberflat(frame, fiberflat)[source]¶
- Apply fiberflat to frame. Modifies frame.flux and frame.ivar.
Checks whether an heliocentric correction has been applied to the frame wavelength in which case also apply it to the flat field array.
- Parameters:
frame – desispec.Frame object
fiberflat – desispec.FiberFlat object
The frame is divided by the fiberflat, except where the fiberflat=0.
- frame.mask gets bit specmask.BADFIBERFLAT set where
fiberflat.fiberflat == 0
fiberflat.ivar == 0
fiberflat.mask != 0
- desispec.fiberflat.autocalib_fiberflat(fiberflats)[source]¶
Combine fiberflats of all spectrographs from different lamps to maximize uniformity :param fiberflats: list of desispec.FiberFlat object
returns a dictionary of desispec.FiberFlat objects , one per spectrograph
- desispec.fiberflat.average_fiberflat(fiberflats)[source]¶
Average several fiberflats :param fiberflats: list of desispec.FiberFlat object
returns a desispec.FiberFlat object
- desispec.fiberflat.compute_fiberflat(frame, nsig_clipping=10.0, accuracy=0.0005, minval=0.1, maxval=10.0, max_iterations=15, smoothing_res=5.0, max_bad=100, max_rej_it=5, min_sn=0, diag_epsilon=0.001)[source]¶
Compute fiber flat by deriving an average spectrum and dividing all fiber data by this average. Input data are expected to be on the same wavelength grid, with uncorrelated noise. They however do not have exactly the same resolution.
- Parameters:
frame (desispec.Frame) – input Frame object with attributes wave, flux, ivar, resolution_data
nsig_clipping – [optional] sigma clipping value for outlier rejection
accuracy – [optional] accuracy of fiberflat (end test for the iterative loop)
minval – [optional] mask pixels with flux < minval * median fiberflat.
maxval – [optional] mask pixels with flux > maxval * median fiberflat.
max_iterations – [optional] maximum number of iterations
smoothing_res – [optional] spacing between spline fit nodes for smoothing the fiberflat
max_bad – [optional] mask entire fiber if more than max_bad-1 initially unmasked pixels are masked during the iterations
max_rej_it – [optional] reject at most the max_rej_it worst pixels in each iteration
min_sn – [optional] mask portions with signal to noise less than min_sn
diag_epsilon – [optional] size of the regularization term in the deconvolution
- Returns:
- desispec.FiberFlat object with attributes
wave, fiberflat, ivar, mask, meanspec
Notes: - we first iteratively :
compute a deconvolved mean spectrum
compute a fiber flat using the resolution convolved mean spectrum for each fiber
smooth the fiber flat along wavelength
clip outliers
then we compute a fiberflat at the native fiber resolution (not smoothed)
the routine returns the fiberflat, its inverse variance , mask, and the deconvolved mean spectrum
the fiberflat is the ratio data/mean , so this flat should be divided to the data
NOTE THAT THIS CODE HAS NOT BEEN TESTED WITH ACTUAL FIBER TRANSMISSION VARIATIONS, OUTLIER PIXELS, DEAD COLUMNS …
- desispec.fiberflat.qa_fiberflat(param, frame, fiberflat)[source]¶
Calculate QA on FiberFlat object
- Parameters:
param – dict of QA parameters
frame – Frame
fiberflat – FiberFlat
- Returns:
- dict of QA outputs
Need to record simple Python objects for yaml (str, float, int)
- Return type:
qadict
desispec.fiberflat_vs_humidity¶
Utility functions to compute a fiber flat corrected for variations with humidity in the shack
- desispec.fiberflat_vs_humidity._fit_flat(wavelength, flux, ivar, fibers, mean_fiberflat_vs_humidity, humidity_array)[source]¶
Finds best fit interpolation of fiberflat templates that matches an input flux frame Works only if wavelength array intersects the range [4000,4600]A, i.e. the blue cameras
- Parameters:
wavelength – 1D numpy array (n_wavelength) in Angstrom
flux – 2D numpy array (n_fibers,n_wavelength) unit does not matter
ivar – 2D numpy array (n_fibers,n_wavelength) inverse variance of flux
fibers – list or 1D number arrays of fibers to use among range(n_fibers)
mean_fiberflat_vs_humidity – 3D numpy array (n_humidity,n_fibers,n_wavelength)
humidity_array – 1D numpy array (n_humidity)
Returns best_fit_flat best_fit_humidity (2D numpy array (n_fibers,n_wavelength) and float)
- desispec.fiberflat_vs_humidity._interpolated_fiberflat_vs_humidity(fiberflat_vs_humidity, humidity_indices, humidity_index)[source]¶
Interpolates between fiberflat templates indexed by humidity.
- Parameters:
fiberflat_vs_humidity – 3D numpy array (n_humidity,n_fibers,n_wavelength)
humidity_indices – 1D numpy array (n_humidity)
humidity_index – float, humidity index value (same unit as humidity_indices)
Returns 2D numpy array (n_fibers,n_wavelength)
- desispec.fiberflat_vs_humidity.compute_humidity_corrected_fiberflat(calib_fiberflat, mean_fiberflat_vs_humidity, humidity_array, current_humidity, frame)[source]¶
Apply a humidity-dependent correction to an input fiber flat Returns frame_fiberflat = calib_fiberflat / flat_vs_humidity_model(calib) * flat_vs_humidity_model(frame)
- Parameters:
calib_fiberflat – desispec.FiberFlat object
mean_fiberflat_vs_humidity – 3D numpy array (n_humidity,n_fibers,n_wavelength)
humidity_array – 1D numpy array (n_humidity)
current_humidity – float (same unit as humidity_array)
frame – desispec.Frame object
Returns modified desispec.FiberFlat object
desispec.fiberfluxcorr¶
Routines to compute fiber flux corrections based on the fiber location, the exposure seeing, and the target morphology.
- desispec.fiberfluxcorr.flat_to_psf_flux_correction(fibermap, exposure_seeing_fwhm=1.1)[source]¶
Multiplicative factor to apply to the flat-fielded spectroscopic flux of a fiber to calibrate the spectrum of a point source, given the current exposure seeing
- Parameters:
fibermap – fibermap of frame, astropy.table.Table
exposure_seeing_fwhm – seeing FWHM in arcsec
Returns: 1D numpy array with correction factor to apply to fiber fielded fluxes, valid for point sources.
- desispec.fiberfluxcorr.psf_to_fiber_flux_correction(fibermap, exposure_seeing_fwhm=1.1)[source]¶
Multiplicative factor to apply to the psf flux of a fiber to obtain the fiber flux, given the current exposure seeing. The fiber flux is the flux one would collect for this object in a fiber of 1.5 arcsec diameter, for a 1 arcsec seeing, FWHM (same definition as for the Legacy Surveys).
- Parameters:
fibermap – fibermap of frame, astropy.table.Table
exposure_seeing_fwhm – seeing FWHM in arcsec
Returns: 1D numpy array with correction factor to apply to fiber fielded fluxes, valid for any sources.
desispec.fluxcalibration¶
Flux calibration routines.
- desispec.fluxcalibration.ZP_from_calib(exptime, wave, calib)[source]¶
Calculate the ZP in AB magnitudes given the calibration and the wavelength arrays :param exptime: float; exposure time in seconds :param wave: 1D array (A) :param calib: 1D array (converts erg/s/A to photons/s/A)
- Returns:
1D array of ZP values in AB magnitudes
- Return type:
ZP_AB
- desispec.fluxcalibration._compute_coef(coord, node_coords)[source]¶
Function used by interpolate_on_parameter_grid2
- Parameters:
coord – 1D array of coordinates of size n_axis
node_coords – 2D array of coordinates of nodes, shape = (n_nodes,n_axis)
- Returns:
1D array of linear coefficients for each node, size = n_nodes
- Return type:
coef
- desispec.fluxcalibration._smooth_template(template_id, camera_index, template_flux)[source]¶
Used for multiprocessing.Pool
- desispec.fluxcalibration.applySmoothingFilter(flux, width=200)[source]¶
Return a smoothed version of the input flux array using a median filter
- Parameters:
flux – 1D array of flux
width – size of the median filter box
- Returns:
median filtered flux of same size as input
- Return type:
smooth_flux
- desispec.fluxcalibration.apply_flux_calibration(frame, fluxcalib)[source]¶
Applies flux calibration to input flux and ivar
- Parameters:
frame – Spectra object with attributes wave, flux, ivar, resolution_data
fluxcalib – FluxCalib object with wave, calib, …
Modifies frame.flux and frame.ivar
- desispec.fluxcalibration.compute_flux_calibration(frame, input_model_wave, input_model_flux, input_model_fibers, nsig_clipping=10.0, deg=2, debug=False, highest_throughput_nstars=0, exposure_seeing_fwhm=1.1, stdcheck=True, nsig_flux_scale=3)[source]¶
Compute average frame throughput based on data frame.(wave,flux,ivar,resolution_data) and spectro-photometrically calibrated stellar models (model_wave,model_flux). Wave and model_wave are not necessarily on the same grid
- Parameters:
frame – Frame object with attributes wave, flux, ivar, resolution_data
input_model_wave – 1D[nwave] array of model wavelengths
input_model_flux – 2D[nstd, nwave] array of model fluxes
input_model_fibers – 1D[nstd] array of model fibers
nsig_clipping – (optional) sigma clipping level
exposure_seeing_fwhm – (optional) seeing FWHM in arcsec of the exposure
stdcheck – check if the model stars are actually standards according to the fibermap and only rely on those
nsig_flux_scale – n sigma cutoff on the flux scale among standard stars
- Returns:
desispec.FluxCalib object calibration: mean calibration (without resolution)
Notes
we first resample the model on the input flux wave grid
then convolve it to the data resolution (the input wave grid is supposed finer than the spectral resolution)
then iteratively - fit the mean throughput (deconvolved, this is needed because of sharp atmospheric absorption lines) - compute a scale factor to fibers (to correct for small mis-alignement for instance) - perform outlier rejection
- There is one subtelty with the relation between calibration and resolution.
The input frame flux is on average flux^frame_fiber = R_fiber*C*flux^true where C is the true calibration (or throughput) which is a function of wavelength. This is the system we solve.
But we want to return a calibration vector per fiber C_fiber defined by flux^cframe_fiber = flux^frame_fiber/C_fiber, such that flux^cframe can be compared with a convolved model of the truth, flux^cframe_fiber = R_fiber*flux^true, i.e. (R_fiber*C*flux^true)/C_fiber = R_fiber*true_flux, giving C_fiber = (R_fiber*C*flux^true)/(R_fiber*flux^true)
There is no solution for this for all possible input specta. The solution for a flat spectrum is returned, which is very close to C_fiber = R_fiber*C (but not exactly).
- desispec.fluxcalibration.interpolate_on_parameter_grid(data_wave, data_flux, data_ivar, template_flux, teff, logg, feh, template_chi2)[source]¶
- 3D Interpolation routine among templates based on a grid of parameters teff, logg, feh.
The tricky part is to define a cube on the parameter grid populated with templates, and it is not always possible. The routine never extrapolates, so that we stay in the range of input parameters.
- Parameters:
data_wave – 1D[nwave] array of wavelength (concatenated list of input wavelength of different cameras and exposures)
data_flux – 1D[nwave] array of normalized flux = (input flux)/median_filter(input flux) (concatenated list)
data_ivar – 1D[nwave] array of inverse variance of normalized flux
template_flux – 2D[ntemplates,nwave] array of normalized flux of templates (after resample, convolution and division by median_filter)
teff – 1D[ntemplates]
logg – 1D[ntemplates]
feh – 1D[ntemplates]
template_chi2 – 1D[ntemplatess] array of precomputed chi2 = sum(data_ivar*(data_flux-template_flux)**2)
- Returns:
best fit coefficient of linear combination of templates chi2 : chi2 of the linear combination
- Return type:
coefficients
- desispec.fluxcalibration.isStdStar(fibermap, bright=None)[source]¶
Determines if target(s) are standard stars
- Parameters:
fibermap – table including DESI_TARGET or SV1_DESI_TARGET bit mask(s)
- Optional:
bright: if True, only bright time standards; if False, only darktime, otherwise both
Returns bool or array of bool
- desispec.fluxcalibration.match_templates(wave, flux, ivar, resolution_data, stdwave, stdflux, teff, logg, feh, ncpu=1, z_max=0.005, z_res=2e-05, template_error=0, comm=None)[source]¶
For each input spectrum, identify which standard star template is the closest match, factoring out broadband throughput/calibration differences.
- Parameters:
wave – A dictionary of 1D array of vacuum wavelengths [Angstroms]. Example below.
flux – A dictionary of 1D observed flux for the star
ivar – A dictionary 1D inverse variance of flux
resolution_data – resolution corresponding to the star’s fiber
stdwave – 1D standard star template wavelengths [Angstroms]
stdflux – 2D[nstd, nwave] template flux
teff – 1D[nstd] effective model temperature
logg – 1D[nstd] model surface gravity
feh – 1D[nstd] model metallicity
ncpu – number of cpu for multiprocessing
comm – MPI communicator; if given, ncpu will be ignored and only rank 0 will return results that are not None
- Returns:
numpy.array of linear coefficient of standard stars redshift : redshift of standard star chipdf : reduced chi2
- Return type:
coef
Notes
wave and stdwave can be on different grids that don’t necessarily overlap
wave does not have to be uniform or monotonic. Multiple cameras can be supported by concatenating their wave and flux arrays
- desispec.fluxcalibration.normalize_templates(stdwave, stdflux, mag, band, photsys)[source]¶
Returns spectra normalized to input magnitudes.
- Parameters:
stdwave – 1D array of standard star wavelengths [Angstroms]
stdflux – 1D observed flux
mag – float desired magnitude
band – G,R,Z,W1 or W2
photsys – N or S (for Legacy Survey North or South)
- Returns:
same as input normflux : normalized flux array
- Return type:
stdwave
Only SDSS_r band is assumed to be used for normalization for now.
- desispec.fluxcalibration.qa_fluxcalib(param, frame, fluxcalib)[source]¶
- Parameters:
param – dict of QA parameters
frame – Frame
fluxcalib – FluxCalib
- Returns:
- dict of QA outputs
Need to record simple Python objects for yaml (str, float, int)
- Return type:
qadict
- desispec.fluxcalibration.redshift_fit(wave, flux, ivar, resolution_data, stdwave, stdflux, z_max=0.005, z_res=5e-05, template_error=0.0)[source]¶
Redshift fit of a single template
- Parameters:
wave – A dictionary of 1D array of vacuum wavelengths [Angstroms]. Example below.
flux – A dictionary of 1D observed flux for the star
ivar – A dictionary 1D inverse variance of flux
resolution_data – resolution corresponding to the star’s fiber
stdwave – 1D standard star template wavelengths [Angstroms]
stdflux – 1D[nwave] template flux
z_max – float, maximum blueshift and redshift in scan, has to be positive
z_res – float, step of of redshift scan between [-z_max,+z_max]
template_error – float, assumed template flux relative error
- Returns:
redshift of standard star
- Return type:
redshift
Notes
wave and stdwave can be on different grids that don’t necessarily overlap
wave does not have to be uniform or monotonic. Multiple cameras can be supported by concatenating their wave and flux arrays
- desispec.fluxcalibration.resample_template(data_wave_per_camera, resolution_obj_per_camera, template_wave, template_flux, template_id)[source]¶
Resample a spectral template on the data wavelength grid. Then convolve the spectra by the resolution for each camera. Also returns the result of applySmoothingFilter. This routine is used internally in a call to multiprocessing.Pool.
- Parameters:
data_wave_per_camera – A dictionary of 1D array of vacuum wavelengths [Angstroms], one entry per camera and exposure.
resolution_obj_per_camera – A dictionary of Resolution objects corresponding for the fiber, one entry per camera and exposure.
template_wave – 1D array, input spectral template wavelength [Angstroms] (arbitrary spacing).
template_flux – 1D array, input spectral template flux density.
template_id – int, template identification index, used to ensure matching of input/output after a multiprocessing run.
- Returns:
int, template identification index, same as input. output_wave : A dictionary of 1D array of vacuum wavelengths output_flux : A dictionary of 1D array of output template flux output_norm : A dictionary of 1D array of output template smoothed flux
- Return type:
template_id
desispec.focus¶
Utility functions for spectrographs focus.
- desispec.focus.piston_and_tilt_to_gauge_offsets(camera_spectro, focus_plane_coefficients)[source]¶
Computes gauge offsets to apply for a given best focus plane as a function of xccd and yccd
- Parameters:
camera_spectro – str, camera identifier starting with b, r or z for NIR
focus_plane_coefficients – array with 3 values (c0,c1,c2), defining the best focus plane offsets: c0+c1*(xccd/2048-1)+c2*(yccd/2048-1) in microns
- Returns:
dictionnary of offsets to apply, in mm, for the “TOP”,”LEFT” and “RIGHT” gauge
desispec.frame¶
Lightweight wrapper class for spectra, to be returned by io.read_frame
desispec.gpu¶
Utility functions for working with GPUs
- class desispec.gpu.NoGPU[source]¶
Context manager to temporarily disable GPU usage, e.g.
- with desispec.gpu.NoGPU()
blat()
- desispec.gpu.is_gpu_available()[source]¶
Return whether cupy and numba.cuda are installed and a GPU is available to use, and $DESI_NO_GPU is not set
- desispec.gpu.redistribute_gpu_ranks(comm, method='round-robin')[source]¶
Redistribute which MPI ranks are assigned to which GPUs
- Parameters:
comm – MPI communicator, or None
- Options:
method: ‘round-robin’ (default) or ‘contiguous’
- Returns:
device_id assigned (-1 if no GPUs)
‘round-robin’ assigns cyclically, e.g. 8 ranks on 4 GPUs would be assigned [0,1,2,3,0,1,2,3].
‘continuous’ assigns contiguous ranks to the same GPU, e.g. [0,0,1,1,2,2,3,3,4,4].
CAUTION: If the MPI communicator spans multiple nodes, this assumes that all nodes have the same number of GPUs, the same number of ranks per node, and that the MPI ranks are themselves contiguously assigned to nodes (which is often the case, but not required in general).
If comm is None, assign the process to GPU 0 (if present).
Note: this also calls free_gpu_memory to release memory on each GPU before assigning ranks to other GPUs.
desispec.heliocentric¶
heliocentric correction routine
- desispec.heliocentric.barycentric_velocity_corr_kms(ra, dec, mjd)[source]¶
Barycentric velocity correction routine.
See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final barycentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the heliocentric frame.
- Parameters:
system (dec - Declination [degrees] in ICRS) –
system –
DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –
- Returns:
- vcorr - Velocity correction term, in km/s, to add to measured
radial velocity to convert it to the heliocentric frame.
- desispec.heliocentric.barycentric_velocity_multiplicative_corr(ra, dec, mjd)[source]¶
Barycentric velocity correction routine.
See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final barycentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the barycentric frame.
- Parameters:
system (dec - Declination [degrees] in ICRS) –
system –
DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –
- Returns:
(1+vcorr/c) - multiplicative term to correct the wavelength
- desispec.heliocentric.heliocentric_velocity_corr_kms(ra, dec, mjd)[source]¶
Heliocentric velocity correction routine.
See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final heliocentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the heliocentric frame.
- Parameters:
system (dec - Declination [degrees] in ICRS) –
system –
DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –
- Returns:
- vcorr - Velocity correction term, in km/s, to add to measured
radial velocity to convert it to the heliocentric frame.
- desispec.heliocentric.heliocentric_velocity_multiplicative_corr(ra, dec, mjd)[source]¶
Heliocentric velocity correction routine.
See http://docs.astropy.org/en/stable/coordinates/velocities.html for more details. The computed correction can be added to any observed radial velocity to determine the final heliocentric radial velocity. In other words, wavelength calibrated with lamps have to be multiplied by (1+vcorr/cspeed) to bring them to the heliocentric frame.
- Parameters:
system (dec - Declination [degrees] in ICRS) –
system –
DOUBLE. (mjd - Decimal Modified Julian date. Note this should probably be type) –
- Returns:
(1+vcorr/c) - multiplicative term to correct the wavelength
- desispec.heliocentric.main()[source]¶
Entry-point for command-line scripts.
Comparison test with IDL routine:
pro helio dec=20 epoch=2000 longitude=-111.59989 latitude=31.96403 altitude=2097 for j=0,2 do begin mjd=58600+100*j jd=mjd+2400000.5 for i=0,20 do begin ra=(360.*i)/20 vkms = heliocentric(ra, dec, epoch, jd=jd, longitude=longitude, latitude=latitude, altitude=altitude) print,"ra,dec,mjd,vkms=",ra,dec,mjd,vkms endfor endfor end ra,dec,mjd,vkms= 0.00000 20 58600 -12.407597 ra,dec,mjd,vkms= 18.0000 20 58600 -5.1777692 ra,dec,mjd,vkms= 36.0000 20 58600 2.8805023 ra,dec,mjd,vkms= 54.0000 20 58600 10.978417 ra,dec,mjd,vkms= 72.0000 20 58600 18.323295 ra,dec,mjd,vkms= 90.0000 20 58600 24.196169 ra,dec,mjd,vkms= 108.000 20 58600 28.022160 ra,dec,mjd,vkms= 126.000 20 58600 29.426754 ra,dec,mjd,vkms= 144.000 20 58600 28.272459 ra,dec,mjd,vkms= 162.000 20 58600 24.672266 ra,dec,mjd,vkms= 180.000 20 58600 18.978587 ra,dec,mjd,vkms= 198.000 20 58600 11.748759 ra,dec,mjd,vkms= 216.000 20 58600 3.6904873 ra,dec,mjd,vkms= 234.000 20 58600 -4.4074277 ra,dec,mjd,vkms= 252.000 20 58600 -11.752306 ra,dec,mjd,vkms= 270.000 20 58600 -17.625179 ra,dec,mjd,vkms= 288.000 20 58600 -21.451170 ra,dec,mjd,vkms= 306.000 20 58600 -22.855764 ra,dec,mjd,vkms= 324.000 20 58600 -21.701469 ra,dec,mjd,vkms= 342.000 20 58600 -18.101277 ra,dec,mjd,vkms= 360.000 20 58600 -12.407597 ra,dec,mjd,vkms= 0.00000 20 58700 -23.152438 ra,dec,mjd,vkms= 18.0000 20 58700 -27.333872 ra,dec,mjd,vkms= 36.0000 20 58700 -29.104026 ra,dec,mjd,vkms= 54.0000 20 58700 -28.289624 ra,dec,mjd,vkms= 72.0000 20 58700 -24.970386 ra,dec,mjd,vkms= 90.0000 20 58700 -19.471222 ra,dec,mjd,vkms= 108.000 20 58700 -12.330428 ra,dec,mjd,vkms= 126.000 20 58700 -4.2469952 ra,dec,mjd,vkms= 144.000 20 58700 3.9878138 ra,dec,mjd,vkms= 162.000 20 58700 11.567919 ra,dec,mjd,vkms= 180.000 20 58700 17.751326 ra,dec,mjd,vkms= 198.000 20 58700 21.932760 ra,dec,mjd,vkms= 216.000 20 58700 23.702914 ra,dec,mjd,vkms= 234.000 20 58700 22.888512 ra,dec,mjd,vkms= 252.000 20 58700 19.569274 ra,dec,mjd,vkms= 270.000 20 58700 14.070110 ra,dec,mjd,vkms= 288.000 20 58700 6.9293160 ra,dec,mjd,vkms= 306.000 20 58700 -1.1541168 ra,dec,mjd,vkms= 324.000 20 58700 -9.3889258 ra,dec,mjd,vkms= 342.000 20 58700 -16.969031 ra,dec,mjd,vkms= 360.000 20 58700 -23.152438 ra,dec,mjd,vkms= 0.00000 20 58800 19.006564 ra,dec,mjd,vkms= 18.0000 20 58800 12.826196 ra,dec,mjd,vkms= 36.0000 20 58800 5.1371364 ra,dec,mjd,vkms= 54.0000 20 58800 -3.3079572 ra,dec,mjd,vkms= 72.0000 20 58800 -11.682420 ra,dec,mjd,vkms= 90.0000 20 58800 -19.166501 ra,dec,mjd,vkms= 108.000 20 58800 -25.027606 ra,dec,mjd,vkms= 126.000 20 58800 -28.692009 ra,dec,mjd,vkms= 144.000 20 58800 -29.801014 ra,dec,mjd,vkms= 162.000 20 58800 -28.246063 ra,dec,mjd,vkms= 180.000 20 58800 -24.179365 ra,dec,mjd,vkms= 198.000 20 58800 -17.998997 ra,dec,mjd,vkms= 216.000 20 58800 -10.309937 ra,dec,mjd,vkms= 234.000 20 58800 -1.8648438 ra,dec,mjd,vkms= 252.000 20 58800 6.5096188 ra,dec,mjd,vkms= 270.000 20 58800 13.993700 ra,dec,mjd,vkms= 288.000 20 58800 19.854805 ra,dec,mjd,vkms= 306.000 20 58800 23.519208 ra,dec,mjd,vkms= 324.000 20 58800 24.628213 ra,dec,mjd,vkms= 342.000 20 58800 23.073262 ra,dec,mjd,vkms= 360.000 20 58800 19.006564
desispec.image¶
Lightweight wrapper class for preprocessed image data.
desispec.image_model¶
Function to provide rapidly a model of a preprocessed image in order to have a pixel variance estimation that is almost not correlated with pixel value.
- desispec.image_model.compute_image_model(image, xytraceset, fiberflat=None, fibermap=None, with_spectral_smoothing=True, with_sky_model=True, spectral_smoothing_sigma_length=10.0, spectral_smoothing_nsig=4.0, psf=None, fit_x_shift=True)[source]¶
Returns a model of the input image, using a fast extraction, a processing of spectra with a common sky model and a smoothing, followed by a reprojection on the CCD image.
- Parameters:
image – a preprocessed image in the form of a desispec.image.Image object
xytraceset – a desispec.xytraceset.XYTraceSet object with trace coordinates
fiberflat – a desispec.fiberflat.FiberFlat object
optional – a desispec.fiberflat.FiberFlat object
with_spectral_smoothing – try and smooth the spectra to reduce noise (and eventualy reduce variance correlation)
optional – try and smooth the spectra to reduce noise (and eventualy reduce variance correlation)
with_sky_model – use a sky model as part of the spectral modeling to reduce the noise (requires a fiberflat)
optional – use a sky model as part of the spectral modeling to reduce the noise (requires a fiberflat)
spectral_smoothing_sigma_length – sigma of Gaussian smoothing along wavelength in A
optional – sigma of Gaussian smoothing along wavelength in A
spectral_smoothing_nsig – number of sigma rejection threshold to fall back to the original extracted spectrum instead of the smooth one
optional – number of sigma rejection threshold to fall back to the original extracted spectrum instead of the smooth one
psf (slow, by default=None, in which case a Gaussian profile is used) –
projection (optional specter.psf.GaussHermitePSF object to be used for the 1D) –
fit_x_shift – fit for an offset of the spectral traces.
optional – fit for an offset of the spectral traces.
- Returns:
a 2D np.array of same shape as image.pix
- desispec.image_model.numba_proj(image, x, sigma, flux)¶
Add a spectrum to a model of the pixel values in a CCD image assuming a Gaussian cross-dispersion profile.
- Parameters:
image – 2D numpy array , this array will be modified
x – 1D numpy array of size image.shape[0], coordinate of center of cross-dispersion profile of spectral trace, in pixel units, for each CCD row
sigma – 1D numpy array of size image.shape[0], sigma of cross-dispersion profile of spectral trace, in pixel units, for each CCD row
flux – 1D numpy array of size image.shape[0], quantity to project (for design usage: electrons per pixel) for each CCD row
desispec.interpolation¶
Utility functions for interpolation of spectra over different wavelength grids.
- desispec.interpolation._unweighted_resample(output_x, input_x, input_flux_density, extrapolate=False)[source]¶
Returns a flux conserving resampling of an input flux density. The total integrated flux is conserved.
- Parameters:
output_x – SORTED vector, not necessarily linearly spaced
input_x – SORTED vector, not necessarily linearly spaced
input_flux_density – input flux density dflux/dx sampled at x
both must represent the same quantity with the same unit input_flux_density = dflux/dx sampled at input_x
- Options:
- extrapolate: extrapolate using edge values of input array, default is False,
in which case values outside of input array are set to zero
- Returns:
returns output_flux
This interpolation conserves flux such that, on average, output_flux_density = input_flux_density
The input flux density outside of the range defined by the edges of the first and last bins is considered null. The bin size of bin ‘i’ is given by (x[i+1]-x[i-1])/2 except for the first and last bin where it is (x[1]-x[0]) and (x[-1]-x[-2]) so flux density is zero for x<x[0]-(x[1]-x[0])/2 and x>x[-1]+(x[-1]-x[-2])/2
The input is interpreted as the nodes positions and node values of a piece-wise linear function:
y(x) = sum_i y_i * f_i(x)
- with::
- f_i(x) = (x_{i-1}<x<=x_{i})*(x-x_{i-1})/(x_{i}-x_{i-1})
(x_{i}<x<=x_{i+1})*(x-x_{i+1})/(x_{i}-x_{i+1})
the output value is the average flux density in a bin flux_out(j) = int_{x>(x_{j-1}+x_j)/2}^{x<(x_j+x_{j+1})/2} y(x) dx / 0.5*(x_{j+1}+x_{j-1})
- desispec.interpolation.resample_flux(xout, x, flux, ivar=None, extrapolate=False)[source]¶
Returns a flux conserving resampling of an input flux density. The total integrated flux is conserved.
- Parameters:
xout (-) – output SORTED vector, not necessarily linearly spaced
x (-) – input SORTED vector, not necessarily linearly spaced
flux (-) – input flux density dflux/dx sampled at x
both x and xout must represent the same quantity with the same unit
- Options:
ivar: weights for flux; default is unweighted resampling
extrapolate: extrapolate using edge values of input array, default is False, in which case values outside of input array are set to zero.
Setting both ivar and extrapolate raises a ValueError because one cannot assign an ivar outside of the input data range.
- Returns:
if ivar is None, returns outflux if ivar is not None, returns outflux, outivar
This interpolation conserves flux such that, on average, output_flux_density = input_flux_density
The input flux density outside of the range defined by the edges of the first and last bins is considered null. The bin size of bin ‘i’ is given by (x[i+1]-x[i-1])/2 except for the first and last bin where it is (x[1]-x[0]) and (x[-1]-x[-2]) so flux density is zero for x<x[0]-(x[1]-x[0])/2 and x>x[-1]-(x[-1]-x[-2])/2
The input is interpreted as the nodes positions and node values of a piece-wise linear function:
y(x) = sum_i y_i * f_i(x)
with:
f_i(x) = (x_{i-1}<x<=x_{i})*(x-x_{i-1})/(x_{i}-x_{i-1}) + (x_{i}<x<=x_{i+1})*(x-x_{i+1})/(x_{i}-x_{i+1})
the output value is the average flux density in a bin:
flux_out(j) = int_{x>(x_{j-1}+x_j)/2}^{x<(x_j+x_{j+1})/2} y(x) dx / 0.5*(x_{j+1}+x_{j-1})
desispec.io¶
Tools for data and metadata I/O.
desispec.io.download¶
Download files from DESI repository.
- desispec.io.download._map_download(map_tuple)[source]¶
Wrapper function to pass to multiprocess.Pool.map().
- desispec.io.download.download(filenames, single_thread=False, workers=None, baseurl='https://data.desi.lbl.gov/desi')[source]¶
Download files from the DESI repository.
This function will try to create any directories that don’t already exist, using the exact paths specified in filenames.
- Parameters:
filenames – string or list-like object containing filenames.
single_thread – (optional) if
True
, do not use multiprocessing to download files.workers – (optional) integer indicating the number of worker processes to create.
baseurl – (optional) string containing the URL of the top-level DESI directory.
- Returns:
Full, local path to the file(s) downloaded.
desispec.io.emlinefit¶
IO routines for desi_emlinefit_afterburner.
- desispec.io.emlinefit.get_targetids(d, bitnames, log=None)[source]¶
Returns the TARGETIDs passing the bitnames for the CMX_TARGET, SV{1,2,3}_DESI_TARGET, or DESI_TARGET mask.
- Parameters:
d – structured array, typically a FIBERMAP catalog, with TARGETID and the CMX_TARGET, SV{1,2,3}_DESI_TARGET, or DESI_TARGET column.
bitnames – comma-separated list of target bitnames to fit from the
*DESI_TARGET
mask (string)log (optional, defaults to get_logger()) – Logger object
- Returns:
list of TARGETIDs passing the {CMX,*DESI}_TARGET mask (np.array)
- Return type:
targetids
Note
If several bitnames are provided, selects the union of those.
Safer if d is already trimmed on unique TARGETIDs (see FIBERMAP format with
zbest-*.fits
files)
- desispec.io.emlinefit.plot_emlines(outpdf, zspecs, emdict, emnames=['OII', 'HDELTA', 'HGAMMA', 'HBETA', 'OIII', 'HALPHA'], targetids=None, objtypes=None, spectypes=None, deltachi2s=None, ylim=(None, None), nrow=10, rowsort_byz=True)[source]¶
Plot the fitted data from get_emlines().
- Parameters:
outpdf – output pdf filename (string)
zspecs – list of redrock redshifts (numpy array)
emdict – a dictionary with various quantities, output by get_lines().
emnames (optional, defaults to ["OII", "HDELTA", "HGAMMA", "HBETA", "OIII", "HALPHA"]) – list of plotted lines (list of string)
targetids (optional, defaults to None) – list of TARGETIDs (numpy array)
objtypes (optional, defaults to None) – list of fibermap OBJTYPEs (numpy array)
spectypes (optional, defaults to None) – list of redrock SPECTYPEs (numpy array)
deltachi2s (optional, defaults to None) – list of redrock DELTACHI2s (numpy array)
ylim (optional, defaults to (None, None)) – ylim for plotting (float doublet)
nrow (optional, defaults to 10) – number of rows, i.e. galaxy, per pdf page (int)
rowsort_byz (optional, defaults to True) – the spectra are displayed by increasing redshifts (boolean)
Note
For emdict:
Each emission line has its own subdictionary.
Each emission line dictionary should contain these keys:
CONT : continuum in
1e-17 * erg/cm2/s/A
FLUX, FLUX_IVAR: flux in
1e-17 * erg/cm2/s/A
waves, fluxes, ivars, models: data used for fitting + fitted model
If some emname is not present in emdict, it will be discarded.
Each spectra will be one row of plot, each emission line corresponds to a column.
If rowsort_byz=False, then the spectra will be displayed by order of appearance.
- desispec.io.emlinefit.read_emlines_inputs(redrock, coadd, mwext_corr=True, rv=3.1, bitnames='ALL', targetids=None, rr_keys='TARGETID,Z,ZWARN,SPECTYPE,DELTACHI2', fm_keys='TARGET_RA,TARGET_DEC,OBJTYPE', log=None)[source]¶
Read the columns and spectra information (waves, fluxes, ivars) from the redrock and coadd files, which will go as input to get_emlines().
- Parameters:
redrock – full path to a redrock/zbest file
coadd – full path to a coadd file (everest-format)
mwext_corr (optional, defaults to True) – correct flux for foreground MW extinction? (boolean)
rv (optional, defaults to 3.1) – value of R_V, used if mwext_corr=True (float)
bitnames (optional, defaults to "ALL", meaning fitting all fibers) – comma-separated list of target bitnames to fit from the
*DESI_TARGET
mask (string)targetids (optional, defaults to None) – list of TARGETIDs to restrict to (int, list, or numpy array)
rr_keys (optional, defaults to "TARGETID,Z,ZWARN,SPECTYPE,DELTACHI2") – comma-separated list of columns from REDSHIFTS to propagate (string)
fm_keys (optional, defaults to "TARGET_RA,TARGET_DEC,OBJTYPE") – comma-separated list of columns from FIBERMAP to propagate (string)
log (optional, defaults to get_logger()) – Logger object
- Returns:
structured array with the desired redrock columns for the desired TARGETIDs fm: structured array with the desired fibermap columns for the desired TARGETIDs waves: wavelengths (numpy array of shape (Nwave)) fluxes: Galactic-extinction-corrected fluxes (numpy array of shape (Nspec, Nwave)) ivars: inverse variances (numpy array of shape (Nspec, Nwave))
- Return type:
rr
Note
We add TARGETID and Z to rr_keys if TARGETID not present in rr_keys nor in fm_keys.
If keys in rr_keys or fm_keys are not present in the redrock, those will be ignored.
If both bitnames and targetids are provided, we take the overlap of the two.
- desispec.io.emlinefit.write_emlines(output, emdict, rr=None, fm=None, redrock=None, coadd=None, rf_fit_hw=None, min_rf_fit_hw=None, rf_cont_w=None, rv=None, log=None)[source]¶
Writes the emission line fitting to a file.
- Parameters:
output – output fits file (string)
emdict – dictionary output by get_emlines()
rr (optional, defaults to None) – redrock REDSHIFTS/ZBEST extension
fm (optional, defaults to None) – redrock FIBERMAP extension
redrock (optional, defaults to None) – used redrock/zbest file
coadd (optional, defaults to None) – used coadd file (everest-format)
rf_fit_hw (optional, defaults to None) – rest-frame wavelength width (in A) used for fitting on each side of the line (float)
min_rf_fit_hw (optional, defaults to None) – minimum requested rest-frame width (in A) on each side of the line to consider the fitting (float)
rf_cont_w (optional, defaults to None) – rest-frame wavelength extent (in A) to fit the continuum (float)
rv (optional, defaults to None) – value of R_V (float)
log (optional, defaults to get_logger()) – Logger object
desispec.io.exposure_tile_qa¶
I/O routines for exposure and tile qa
- desispec.io.exposure_tile_qa.read_exposure_qa(filename)[source]¶
Reads an exposure-qa fits file.
- Parameters:
filename – full path to file
- Returnsfiber_qa_table, petal_qa_table
two astropy.table.Table objects
- desispec.io.exposure_tile_qa.read_tile_qa(filename)[source]¶
Reads an tile-qa fits file.
- Parameters:
filename – full path to file
- Returnsfiber_qa_table, petal_qa_table
two astropy.table.Table objects
- desispec.io.exposure_tile_qa.write_exposure_qa(filename, fiber_qa_table, petal_qa_table=None)[source]¶
Writes an exposure-qa fits file.
- Parameters:
filename – full path to file
fiber_qa_table – astropy.table.Table object with one row per target
petal_qa_table – astropy.table.Table object with one row per petal
- desispec.io.exposure_tile_qa.write_tile_qa(filename, fiber_qa_table, petal_qa_table=None)[source]¶
Writes an tile-qa fits file.
- Parameters:
filename – full path to file
fiber_qa_table – astropy.table.Table object with one row per target
petal_qa_table – astropy.table.Table object with one row per petal
desispec.io.fiberflat¶
IO routines for fiberflat.
- desispec.io.fiberflat.read_fiberflat(filename)[source]¶
Read fiberflat from filename
- Parameters:
filename (str) – Name of fiberflat file, or (night, expid, camera) tuple
- Returns:
- FiberFlat object with attributes
fiberflat, ivar, mask, meanspec, wave, header
Notes
fiberflat, ivar, mask are 2D [nspec, nwave] meanspec and wave are 1D [nwave]
- desispec.io.fiberflat.write_fiberflat(outfile, fiberflat, header=None, fibermap=None)[source]¶
Write fiberflat object to outfile
- Parameters:
outfile – filepath string or (night, expid, camera) tuple
fiberflat – FiberFlat object
- Optional:
header: dict or fits.Header object to use as HDU 0 header fibermap: table to store as FIBERMAP HDU
- Returns:
filepath of file that was written
desispec.io.fiberflat_vs_humidity¶
- desispec.io.fiberflat_vs_humidity.read_fiberflat_vs_humidity(filename)[source]¶
Read fiberflat vs humidity from filename
- Parameters:
filename (str) – path to fiberflat_vs_humidity file
- Returns: fiberflat , humidity , wave
fiberflat is 3D [nhumid, nspec, nwave] humidity is 1D [nhumid] (and in percent) wave is 1D [nwave] (and in Angstrom) header (fits header)
desispec.io.fibermap¶
IO routines for fibermap.
- desispec.io.fibermap._set_fibermap_columns()[source]¶
Prepare survey-specific list of columns.
- Returns:
As a convenience, return the full set of survey-specific columns.
- Return type:
- desispec.io.fibermap.assemble_fibermap(night, expid, badamps=None, badfibers_filename=None, force=False, allow_svn_override=True)[source]¶
Create a fibermap for a given night and expid.
- Parameters:
night (
int
) – YEARMMDD night of sunset.expid (
int
) – Exposure ID.badamps (
str
, optional) – Comma separated list of"{camera}{petal}{amp}"
, i.e."[brz][0-9][ABCD]"
. Example:'b7D,z8A'
.badfibers_filename (
str
, optional) – Filename with table of bad fibers with at least two columns: FIBER and FIBERSTATUSforce (
bool
, optional) – Create fibermap even if missing coordinates/guide files.allow_svn_override (
bool
, optional) – If True (default), allow fiberassign SVN to override raw data.
- Returns:
A representation of a fibermap FITS file.
- Return type:
- desispec.io.fibermap.compare_fiberassign(fa1, fa2, compare_all=False)[source]¶
Check whether two fiberassign tables agree for cols used by ICS/platemaker
- Parameters:
fa1 – fiberassign astropy Tables or numpy structured arrays
fa2 – fiberassign astropy Tables or numpy structured arrays
- Options:
compare_all: if True, compare all columns, not just those used by ops
Returns list of columns with mismatches; empty list if all agree
Note: if both are NaN, it is considered a match
- desispec.io.fibermap.empty_fibermap(nspec, specmin=0, survey='main')[source]¶
Return an empty fibermap Table to be filled in.
- desispec.io.fibermap.fibermap_new2old(fibermap)[source]¶
Converts new format fibermap into old format fibermap
- Parameters:
fibermap – new-format fibermap table (e.g. with FLUX_G column)
- Returns:
old format fibermap (e.g. with MAG column)
Note: this is a transitional convenience function to allow us to simulate new format fibermaps while still running code that expects the old format. After all code has been converted to use the new format, this will be removed.
- desispec.io.fibermap.find_fiberassign_file(night, expid, tileid=None, nightdir=None)[source]¶
Walk backwards in exposures to find matching fiberassign file
- Options:
tileid (int): tileid to look for nightdir (str): base directory for raw data on that night
Returns first fiberassign file found on or before expid on night.
Raises FileNotFoundError if no fiberassign file is found
- desispec.io.fibermap.read_fibermap(fp)[source]¶
Reads a fibermap file and returns its data as an astropy Table
- Parameters:
fp – input file name, or opened fitsio.FITS or astropy.io.fits.HDUList
- desispec.io.fibermap.update_survey_keywords(hdr)[source]¶
Standardize fiberassign header keywords SURVEY, FAPRGRM, FAFLAVOR
- Parameters:
hdr (dict-like) – HDU 0 header from a fiberassign file
- Returns:
value pairs added
- Return type:
dict of key
Note: updates hdr object in-place
- desispec.io.fibermap.write_fibermap(outfile, fibermap, header=None, clobber=True, extname='FIBERMAP')[source]¶
Write fibermap binary table to outfile.
- Parameters:
- Returns:
full path to filename of fibermap file written.
- Return type:
write_fibermap (str)
desispec.io.filters¶
- desispec.io.filters.load_filter(given_filter)[source]¶
Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object
- Parameters:
given_filter – given filter for which the qe is to be loaded. Desi templates/
uppercase (files have them in) –
SDSS (so it should be in upper case like) –
or (DECAM) –
here. (WISE. Speclite has lower case so are mapped) –
- desispec.io.filters.load_gaia_filter(band, dr=2)[source]¶
Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object
- Parameters:
band – filter pass-band in “G”,”BP”,”RP”
dr – 2 or 3
- desispec.io.filters.load_legacy_survey_filter(band, photsys)[source]¶
Uses speclite.filters to load the filter transmission Returns speclite.filters.FilterResponse object
- Parameters:
band – filter pass-band in “G”,”R”,”Z”,”W1”,”W2”
photsys – “N” or “S” for North (BASS+MzLS) or South (CTIO/DECam)
desispec.io.fluxcalibration¶
IO routines for flux calibration.
- desispec.io.fluxcalibration.read_average_flux_calibration(filename)[source]¶
Read average flux calibration file; returns an AverageFluxCalib object
- desispec.io.fluxcalibration.read_flux_calibration(filename)[source]¶
Read flux calibration file; returns a FluxCalib object
- desispec.io.fluxcalibration.read_stdstar_models(filename)[source]¶
Read stdstar models from filename.
- desispec.io.fluxcalibration.read_stdstar_templates(stellarmodelfile)[source]¶
Reads an input stellar model file
- Parameters:
stellarmodelfile – input filename
- Returns (wave, flux, templateid, teff, logg, feh) tuple:
wave : 1D[nwave] array of wavelengths [Angstroms] flux : 2D[nmodel, nwave] array of model fluxes templateid : 1D[nmodel] array of template IDs for each spectrum teff : 1D[nmodel] array of effective temperature for each model logg : 1D[nmodel] array of surface gravity for each model feh : 1D[nmodel] array of metallicity for each model
- desispec.io.fluxcalibration.write_average_flux_calibration(outfile, averagefluxcalib)[source]¶
Writes average flux calibration.
- Parameters:
outfile – output file name
averagefluxcalib – AverageFluxCalib object
- Options:
header : dict-like object of key/value pairs to include in header
- desispec.io.fluxcalibration.write_flux_calibration(outfile, fluxcalib, header=None)[source]¶
Writes flux calibration.
- Parameters:
outfile – output file name
fluxcalib – FluxCalib object
- Options:
header : dict-like object of key/value pairs to include in header
- desispec.io.fluxcalibration.write_stdstar_models(norm_modelfile, normalizedFlux, wave, fibers, data, fibermap, input_frames, header=None)[source]¶
Writes the normalized flux for the best models.
- Parameters:
norm_modelfile – output file path
normalizedFlux – 2D array of flux[nstdstars, nwave]
wave – 1D array of wavelengths[nwave] in Angstroms
fibers – 1D array of fiberids for these spectra
data – meta data table about which templates best fit
fibermap – fibermaps rows for the input standard stars
input_frames – Table with NIGHT, EXPID, CAMERA of input frames used
desispec.io.frame¶
I/O routines for Frame objects
- desispec.io.frame.read_frame(filename, nspec=None, skip_resolution=False)[source]¶
Reads a frame fits file and returns its data.
- Parameters:
filename – path to a file, or (night, expid, camera) tuple where night = string YEARMMDD expid = integer exposure ID camera = b0, r1, .. z9
skip_resolution – bool, option Speed up read time (>5x) by avoiding the Resolution matrix
- Returns:
desispec.Frame object with attributes wave, flux, ivar, etc.
- desispec.io.frame.read_meta_frame(filename, extname=0)[source]¶
Load the meta information of a Frame :param filename: path to a file :param extname: int, optional; Extension for grabbing header info
- Returns:
dict or astropy.fits.header
- Return type:
meta
- desispec.io.frame.search_for_framefile(frame_file, specprod_dir=None)[source]¶
Search for an input frame_file in the desispec redux hierarchy :param frame_file: str :param specprod_dir: str, optional
- Returns:
str, full path to frame_file if found else raise error
- Return type:
mfile
- desispec.io.frame.write_frame(outfile, frame, header=None, fibermap=None, units=None)[source]¶
Write a frame fits file and returns path to file written.
- Parameters:
outfile – full path to output file, or tuple (night, expid, channel)
frame – desispec.frame.Frame object with wave, flux, ivar…
- Optional:
header: astropy.io.fits.Header or dict to override frame.header fibermap: table to store as FIBERMAP HDU
- Returns:
full filepath of output file that was written
Note
to create a Frame object to pass into write_frame, frame = Frame(wave, flux, ivar, resolution_data)
desispec.io.image¶
I/O routines for Image objects
- desispec.io.image.write_image(outfile, image, meta=None)[source]¶
Writes image object to outfile
- Parameters:
outfile – output file string
image – desispec.image.Image object (or any object with 2D array attributes image, ivar, mask)
- Optional:
meta : dict-like object with metadata key/values (e.g. FITS header)
desispec.io.iotime¶
Utilities for parsing and plotting I/O timing from logfiles.
- desispec.io.iotime._ordered_unique_names(names)[source]¶
Return unique list of names, ordered by first appearance in list
Doesn’t scale well; intended for inputs <10000 long
- desispec.io.iotime.format(readwrite, filename, duration)[source]¶
Return standardized I/O timing message string for logging
- Parameters:
Returns: I/O timing message to log
Note: this function does not call log.info() itself so that the logging source file and line number can be associated with the I/O function itself instead of this utility formatting function.
- desispec.io.iotime.hist_iotimes(timing, tmax=10, plottitle=None)[source]¶
Histogram function timing from Table read with read_iotimes
- Parameters:
timing – Table with columns FUNC, DATETIME, IOTIME
- Options:
tmax (float): upper bound of histogram (overflows included in last bin)o plottitle (str): plot title
Returns matplotlib Figure
- desispec.io.iotime.parse(line)[source]¶
Parse a line for an iotime message produced by format
- Parameters:
line (str) – the line to parse
Returns None if no match, or dict with keys function,duration,readwrite,filename,timestamp
- desispec.io.iotime.parse_logfile(logfile)[source]¶
Read iotime log entries from logfile
Return Table with columns function duration readwrite filename timestamp datetime
- desispec.io.iotime.plot_iotimes(timing, plottitle=None, outfile=None)[source]¶
Plot I/O duration vs. time of I/O operation
- Parameters:
timing – Table with columns FUNC, DATETIME, IOTIME
- Options:
plottitle (str): Title to include for plot outfile (str): write plot to this file
Returns matplotlib figure; does not call plt.show()
desispec.io.meta¶
IO metadata functions.
- desispec.io.meta.faflavor2program(faflavor)[source]¶
Map FAFLAVOR keywords to what we wish we had set for FAPRGRM
- Parameters:
faflavor (str or array of str) – FAFLAVOR keywords from fiberassign
- Returns:
what FAPRGM would be if we had set it (dark, bright, backup, other)
- Return type:
Note: this was standardized by sv3 and main, but evolved during sv1 and sv2
- desispec.io.meta.find_exposure_night(expid, specprod_dir=None)[source]¶
Find the night that has the exposure :param expid: int :param specprod_dir: str, optional
- Returns:
str
- Return type:
night
- desispec.io.meta.findfile(filetype, night=None, expid=None, camera=None, tile=None, groupname=None, healpix=None, nside=64, band=None, spectrograph=None, survey=None, faprogram=None, rawdata_dir=None, specprod_dir=None, download=False, outdir=None, qaprod_dir=None, return_exists=False, readonly=False, logfile=False)[source]¶
Returns location where file should be
- Parameters:
filetype – file type, typically the prefix, e.g. “frame” or “psf”
- Args depending upon filetype:
night : YEARMMDD string expid : integer exposure id camera : ‘b0’ ‘r1’ .. ‘z9’ tile : integer tile (pointing) number groupname : spectral grouping name (e.g. “healpix”, “cumulative”, “pernight”) healpix : healpix pixel number nside : healpix nside band : one of ‘b’,’r’,’z’ identifying the camera band spectrograph : integer spectrograph number, 0-9 survey : e.g. sv1, sv3, main, special faprogram : fiberassign program, e.g. dark, bright
- Options:
rawdata_dir : overrides $DESI_SPECTRO_DATA specprod_dir : overrides $DESI_SPECTRO_REDUX/$SPECPROD/ qaprod_dir : defaults to $DESI_SPECTRO_REDUX/$SPECPROD/QA/ if not provided download : if not found locally, try to fetch remotely outdir : use this directory for output instead of canonical location return_exists: if True, also return whether the file exists readonly: if True, return read-only version of path if possible logfile: if True, returns the pathname of the log instead of the data product itself
Returns filename, or (filename, exists) if return_exists=True
- Raises:
ValueError – for invalid file types, and other invalid input
KeyError – for missing environment variables
Notes
The readonly option uses $DESI_ROOT_READONLY if it is set and exists; otherwise it returns the normal read/write path.
- desispec.io.meta.get_desi_root_readonly()[source]¶
Returns $DESI_ROOT_READONLY if set and path exists, otherwise $DESI_ROOT.
Caches answer upon first call, i.e. setting $DESI_ROOT_READONLY to a different value part way through running will use previously cached value. This prevents it from re-checking a non-existent path N>>1 times.
- desispec.io.meta.get_exposures(night, raw=False, rawdata_dir=None, specprod_dir=None)[source]¶
Get a list of available exposures for the specified night.
Exposures are identified as correctly formatted subdirectory names within the night directory, but no checks for valid contents of these exposure subdirectories are performed.
- Parameters:
night (str) – Date string for the requested night in the format YYYYMMDD.
raw (bool) – Returns raw exposures if set, otherwise returns processed exposures.
rawdata_dir (str) – [optional] overrides $DESI_SPECTRO_DATA
specprod_dir (str) – Path containing the exposures/ directory to use. If the value is None, then the value of
specprod_root()
is used instead. Ignored when raw is True.
- Returns:
- List of integer exposure numbers available for the specified night. The
list will be empty if no the night directory exists but does not contain any exposures.
- Return type:
- Raises:
ValueError – Badly formatted night date string
IOError – non-existent night.
- desispec.io.meta.get_files(filetype, night, expid, specprod_dir=None, qaprod_dir=None, **kwargs)[source]¶
Get files for a specified exposure.
Uses
findfile()
to determine the valid file names for the specified type. Any camera identifiers not matching the regular expression [brz][0-9] will be silently ignored.- Parameters:
filetype (str) – Type of files to get. Valid choices are ‘frame’, ‘cframe’, ‘psf’, etc.
night (str) – Date string for the requested night in the format YYYYMMDD.
expid (int) – Exposure number to get files for.
specprod_dir (str) – Path containing the exposures/ directory to use. If the value is None, then the value of
specprod_root()
is used instead. Ignored when raw is True.
- Returns:
- Dictionary of found file names using camera id strings as keys,
which are guaranteed to match the regular expression [brz][0-9].
- Return type:
- desispec.io.meta.get_nights(strip_path=True, specprod_dir=None, sub_folder='exposures')[source]¶
Generate a list of nights in a given folder (default is exposures/) Demands an 8 digit name beginning with 20
- Parameters:
strip_path – bool, optional; Strip the path to the nights folders
rawdata_dir –
specprod_dir –
sub_root – str, optional; ‘exposures’, ‘calib2d’
- Returns:
list of nights (without or with paths)
- Return type:
nights
- desispec.io.meta.get_nights_up_to_date(date, specprod_dir=None)[source]¶
Returns nights with an existing exposure_table in the current specprod
- desispec.io.meta.get_pipe_database()[source]¶
Get the production database location based on the environment.
- desispec.io.meta.get_pipe_logdir()[source]¶
Return the name of the subdirectory containing pipeline logs.
- Returns (str):
The name of the subdirectory.
- desispec.io.meta.get_pipe_nightdir()[source]¶
Return the name of the subdirectory containing per-night files.
- Returns (str):
The name of the subdirectory.
- desispec.io.meta.get_pipe_pixeldir()[source]¶
Return the name of the subdirectory containing per-pixel files.
- Returns (str):
The name of the subdirectory.
- desispec.io.meta.get_pipe_rundir(specprod_dir=None)[source]¶
Return the directory path for pipeline runtime files.
- Parameters:
specprod_dir (str) – Optional path to production directory. If None, the this is obtained from
specprod_root()
.
- Returns (str):
the directory path for pipeline runtime files.
- desispec.io.meta.get_pipe_scriptdir()[source]¶
Return the name of the subdirectory containing pipeline scripts.
- Returns (str):
The name of the subdirectory.
- desispec.io.meta.get_raw_files(filetype, night, expid, rawdata_dir=None)[source]¶
Get files for a specified exposure.
Uses
findfile()
to determine the valid file names for the specified type. Any camera identifiers not matching the regular expression [brz][0-9] will be silently ignored.- Parameters:
- Returns:
- Dictionary of found file names using camera id strings as keys,
which are guaranteed to match the regular expression [brz][0-9].
- Return type:
- desispec.io.meta.get_readonly_filepath(filepath)[source]¶
Generate optimized path for read-only usage of filepath
- Parameters:
filepath (str) – full path to input file
Returns: readonly_filepath using $DESI_ROOT_READONLY
If a readonly filepath can’t be derived, return original filepath
- desispec.io.meta.get_reduced_frames(channels=['b', 'r', 'z'], nights=None, ftype='cframe', **kwargs)[source]¶
Loops through a production to find all reduced frames (default is cframes) One can choose a subset of reduced frames by argument :param channels: list, optional :param nights: list, optional :param ftype: str, optional :param kwargs: passed to get_files()
- Returns:
list for frame filenames
- Return type:
all_frames
- desispec.io.meta.qaprod_root(specprod_dir=None)[source]¶
Return directory root for spectro production QA, i.e.
$DESI_SPECTRO_REDUX/$SPECPROD/QA
.- Raises:
KeyError – if these environment variables aren’t set.
- desispec.io.meta.rawdata_root()[source]¶
Returns directory root for raw data, i.e.
$DESI_SPECTRO_DATA
- Raises:
KeyError – if these environment variables aren’t set.
- desispec.io.meta.shorten_filename(filename)[source]¶
Attempt to shorten filename to fit in FITS header without CONTINUE
- Parameters:
filename (str) – input filename
Returns potentially shortened filename
- Replaces prefixes from environment variables:
$DESI_SPECTRO_CALIB -> SPCALIB
$DESI_SPECTRO_REDUX/$SPECPROD -> SPECPROD
- desispec.io.meta.specprod_root(specprod=None)[source]¶
Return directory root for spectro production, i.e.
$DESI_SPECTRO_REDUX/$SPECPROD
.- Options:
specprod (str): overrides $SPECPROD
- Raises:
KeyError – if these environment variables aren’t set.
- desispec.io.meta.validate_night(night)[source]¶
Validates a night string and converts to a date.
- Parameters:
night (str) – Date string for the requested night in the format YYYYMMDD.
- Returns:
Date object representing this night.
- Return type:
- Raises:
ValueError – Badly formatted night string.
desispec.io.params¶
IO routines for parameter values
desispec.io.photo¶
Simple methods for gathering photometric and targeting catalogs.
- desispec.io.photo._gather_tractorphot_onebrick(input_cat, legacysurveydir, radius_match, racolumn, deccolumn, datamodel)[source]¶
Support routine for gather_tractorphot.
- desispec.io.photo.gather_targetdirs(tileid, fiberassign_dir=None, verbose=False)[source]¶
Gather all the targeting directories used to build a given fiberassign catalog, including both primary and secondary targets and ToOs.
- Parameters:
- Returns:
list of targeting directories and files
- Return type:
targetdirs (numpy.ndarray)
- desispec.io.photo.gather_targetphot(input_cat, photocache=None, racolumn='TARGET_RA', deccolumn='TARGET_DEC', columns=None, fiberassign_dir=None, verbose=False)[source]¶
Find and stack the photometric targeting information given a set of targets.
- Parameters:
input_cat (astropy.table.Table) – input table with the following (required) columns: TARGETID, RACOLUMN, DECCOLUMN, TILEID.
photocache (dict, optional) – dictionary cache of targetids for large targeting catalogs.
racolumn (str) – name of the RA column in input_cat (defaults to RA_COLUMN)
deccolumn (str) – name of the RA column in input_cat (defaults to DEC_COLUMN)
columns (str array) – return this subset of columns
fiberassign_dir (str, optional) – top-level directory to fiberassign tables
Returns a table of targeting photometry using a consistent data model across primary (DR9) targets, secondary targets, and targets of opportunity. The data model is documented in targetphot_datamodel.
- desispec.io.photo.gather_tractorphot(input_cat, racolumn='TARGET_RA', deccolumn='TARGET_DEC', legacysurveydir=None, dr9dir=None, radius_match=1.0, columns=None, verbose=False)[source]¶
Retrieve the Tractor catalog for all the objects in this catalog (one brick).
- Parameters:
input_cat (astropy.table.Table) – input table with the following (required) columns: TARGETID, TARGET_RA, TARGET_DEC. Additional optional columns that will ensure proper matching are BRICKNAME, RELEASE, PHOTSYS, BRICKID, and BRICK_OBJID.
legacysurveydir (str) – full path to the location of the Tractor catalogs
dr9dir (str) – relegated keyword; please use legacysurveydir
radius_match (float, arcsec) – matching radius (default, 1 arcsec)
columns (str array) – return this subset of columns
Returns a table of Tractor photometry. Matches are identified either using BRICKID and BRICK_OBJID or using positional matching (1 arcsec radius).
- desispec.io.photo.targetphot_datamodel(from_file=False)[source]¶
Initialize the targetphot data model.
- Parameters:
from_file (bool, optional) – read the datamodel from a file on-disk.
Returns an astropy.table.Table with a consistent data model across cmx, sv[1-2], and main-survey observations.
- desispec.io.photo.tractorphot_datamodel(from_file=False, datarelease='dr9')[source]¶
Initialize the tractorphot data model for a given Legacy Surveys data release.
- Parameters:
Returns an astropy.table.Table which follows the Tractor catalog datamodel for the given data release.
desispec.io.qa¶
IO routines for QA.
- desispec.io.qa.load_qa_brick(filename)[source]¶
Load an existing QA_Brick or generate one, as needed :param filename: str
Returns: qa_brick: QA_Brick object
- desispec.io.qa.load_qa_frame(filename, frame_meta=None, flavor=None)[source]¶
Load an existing QA_Frame or generate one, as needed
- Parameters:
filename – str
frame_meta – dict like, optional
flavor – str, optional Type of QA_Frame
- Returns:
QA_Frame object
- Return type:
qa_frame
- desispec.io.qa.load_qa_multiexp(inroot)[source]¶
Load QA for a given production
- Parameters:
inroot – str base filename without format extension
- Returns:
dict
- Return type:
odict
- desispec.io.qa.qafile_from_framefile(frame_file, qaprod_dir=None, output_dir=None)[source]¶
Derive the QA filename from an input frame file :param frame_file: str :param output_dir: str, optional Over-ride default output path :param qa_dir: str, optional Over-ride default QA
Returns:
- desispec.io.qa.write_qa_brick(outfile, qabrick)[source]¶
Write QA for a given exposure
- Parameters:
outfile – filename
qabrick – QA_Brick object _data: dict of QA info
- desispec.io.qa.write_qa_exposure(outroot, qaexp, ret_dict=False)[source]¶
Write QA for a given exposure
- Parameters:
outroot – str filename without format extension
qa_exp – QA_Exposure object
ret_dict – bool, optional Return dict only? [for qa_prod, mainly]
- Returns:
str or dict
- Return type:
outfile or odict
- desispec.io.qa.write_qa_frame(outfile, qaframe, verbose=False)[source]¶
Write QA for a given frame
- Parameters:
outfile – str filename
qa_exp – QA_Frame object, with the following attributes qa_data: dict of QA info
- desispec.io.qa.write_qa_multiexp(outroot, mdict, indent=True)[source]¶
Write QA for a given production
- Parameters:
outroot – str filename without format extension
mdict – dict
- Returns:
- str
output filename
- Return type:
outfile
- desispec.io.qa.write_qa_ql(outfile, qaresult)[source]¶
Write QL output files
- Parameters:
outfile – str filename to be written (yaml)
qaresult – dict QAresults from run_qa()
- Returns:
str
- Return type:
outfile
desispec.io.raw¶
I/O for DESI raw data files
See DESI-1229 for format details TODO: move into datamodel after we have verified the format
- desispec.io.raw.read_raw(filename, camera, fibermapfile=None, fill_header=None, **kwargs)[source]¶
Returns preprocessed raw data from camera extension of filename.
- Parameters:
filename (
str
) – Input FITS filename with DESI raw data.camera (
str
) – Camera name (B0, R1, … Z9) or FITS extension name.fibermapfile (
str
, optional) – Read fibermap from this file; ifNone
create blank fibermap.fill_header (
list
, optional) – A list of HDU names or numbers. The header cards from these HDUs will be added to the header of the camera HDU read from filename.
- Returns:
Image object with member variables pix, ivar, mask, readnoise.
- Return type:
desispec.image.Image
- Raises:
IOError – If camera is not a HDU in filename.
KeyError – If
EXPTIME
is not present in any header in filename, or if bothNIGHT
andDATE-OBS
are missing from input headers.ValueError – If
NIGHT
in the primary header does not matchNIGHT
in the camera header, or if fill_header is not alist
.
Notes
Other keyword arguments are passed to
desispec.preproc.preproc()
, e.g. bias, pixflat, mask. Seepreproc()
documentation for details.
- desispec.io.raw.write_raw(filename, rawdata, header, camera=None, primary_header=None)[source]¶
Write raw pixel data to a DESI raw data file
- Parameters:
filename – file name to write data; if this exists, append a new HDU
rawdata – 2D ndarray of raw pixel data including overscans
header – dict-like object or fits.Header with keywords CCDSECx, BIASSECx, DATASECx where x=A,B,C,D
- Options:
camera : b0, r1 .. z9 - override value in header primary_header : header to write in HDU0 if filename doesn’t yet exist
The primary utility of this function over raw fits calls is to ensure that all necessary keywords are present before writing the file. CCDSECx, BIASSECx, DATASECx where x=A,B,C,D DATE-OBS will generate a non-fatal warning if missing
desispec.io.sky¶
IO routines for sky.
- desispec.io.sky.read_sky(filename)[source]¶
Read sky model and return SkyModel object with attributes wave, flux, ivar, mask, header.
skymodel.wave is 1D common wavelength grid, the others are 2D[nspec, nwave]
- desispec.io.sky.write_sky(outfile, skymodel, header=None)[source]¶
Write sky model.
- Parameters:
outfile – filename or (night, expid, camera) tuple
skymodel – SkyModel object, with the following attributes
wave – 1D wavelength in vacuum Angstroms
flux – 2D[nspec, nwave] sky flux
ivar – 2D inverse variance of sky flux
mask – 2D mask for sky flux
stat_ivar – 2D inverse variance of sky flux (statistical only)
dwavecoeff – 1D[ncoeff] array of PCA dwavelength coefficients (optional)
dlsfcoeff – 1D[ncoeff] array of PCA dlsf coefficients (optional)
header – optional fits header data (fits.Header, dict, or list)
desispec.io.skycorr¶
IO routines for sky.
- desispec.io.skycorr.read_skycorr(filename)[source]¶
Read sky correction and return SkyCorr object with attributes wave, flux, ivar, mask, header.
skymodel.wave is 1D common wavelength grid, the others are 2D[nspec, nwave]
- desispec.io.skycorr.read_skycorr_pca(filename)[source]¶
Read sky correction pca file and return SkyCorrPCA object.
- desispec.io.skycorr.write_skycorr(outfile, skycorr)[source]¶
Write sky model.
- Parameters:
outfile – filename or (night, expid, camera) tuple
skycorr – SkyCorr object
- desispec.io.skycorr.write_skycorr_pca(outfile, skycorrpca)[source]¶
Write sky model.
- Parameters:
outfile – filename or (night, expid, camera) tuple
skycorrpca – SkyCorrPCA object
desispec.io.skygradpca¶
IO routines for sky gradient pca.
- desispec.io.skygradpca.read_skygradpca(filename)[source]¶
Read sky grad pca file and return SkyGradPCA object.
- desispec.io.skygradpca.write_skygradpca(outfile, skygradpca)[source]¶
Write sky model.
- Parameters:
outfile – filename or (night, expid, camera) tuple
skygradpca – SkyGradPCA object
desispec.io.spectra¶
I/O routines for working with spectral grouping files.
- desispec.io.spectra._read_image(hdus, extname, dtype, rows=None)[source]¶
Helper function to read extname from fitsio.FITS hdus, filter by rows, convert to native endian, and cast to dtype. Returns image.
- desispec.io.spectra.read_frame_as_spectra(filename, night=None, expid=None, band=None, single=False)[source]¶
Read a FITS file containing a Frame and return a Spectra.
A Frame file is very close to a Spectra object (by design), and only differs by missing the NIGHT and EXPID in the fibermap, as well as containing only one band of data.
- Parameters:
infile (str) – path to read
- Options:
night (int): the night value to use for all rows of the fibermap. expid (int): the expid value to use for all rows of the fibermap. band (str): the name of this band. single (bool): if True, keep spectra as single precision in memory.
- Returns (Spectra):
The object containing the data read from disk.
- desispec.io.spectra.read_spectra(infile, single=False, targetids=None, rows=None, skip_hdus=None, select_columns={'EXP_FIBERMAP': None, 'EXTRA_CATALOG': None, 'FIBERMAP': None, 'SCORES': None})[source]¶
Read Spectra object from FITS file.
This reads data written by the write_spectra function. A new Spectra object is instantiated and returned.
- Parameters:
infile (str) – path to read
single (bool) – if True, keep spectra as single precision in memory.
targetids (list) – Optional, list of targetids to read from file, if present.
rows (list) – Optional, list of rows to read from file
skip_hdus (list) – Optional, list/set/tuple of HDUs to skip
select_columns (dict) – Optional, dictionary to select column names to be read. Default, all columns are read.
- Returns (Spectra):
The object containing the data read from disk.
skip_hdus options are FIBERMAP, EXP_FIBERMAP, SCORES, EXTRA_CATALOG, MASK, RESOLUTION; where MASK and RESOLUTION mean to skip those for all cameras. Note that WAVE, FLUX, and IVAR are always required.
If a table HDU is not listed in select_columns, all of its columns will be read
User can optionally specify targetids OR rows, but not both
- desispec.io.spectra.read_tile_spectra(tileid, night=None, specprod=None, reduxdir=None, coadd=False, single=False, targets=None, fibers=None, redrock=True, group='cumulative')[source]¶
Read and return combined spectra for a tile/night
- Parameters:
tileid (int) – Tile ID
- Options:
night (int or str) : YEARMMDD night specprod (str) : overrides $SPECPROD reduxdir (str) : overrides $DESI_SPECTRO_REDUX/$SPECPROD coadd (bool) : if True, read coadds instead of per-exp spectra single (bool) : if True, use float32 instead of double precision targets (array-like) : filter by TARGETID fibers (array-like) : filter by FIBER redrock (bool) : if True, also return row-matched redrock redshift catalog group (str) : reads spectra in group (pernight, cumulative, …)
- Returns: spectra or (spectra, redrock)
combined Spectra obj for all matching targets/fibers filter row-matched redrock catalog (if redrock=True)
- Raises:
ValueError if no files or matching spectra are found –
Note: the returned spectra are not necessarily in the same order as the targets or fibers input filters
- desispec.io.spectra.write_spectra(outfile, spec, units=None)[source]¶
Write Spectra object to FITS file.
This places the metadata into the header of the (empty) primary HDU. The first extension contains the fibermap, and then HDUs are created for the different data arrays for each band.
Floating point data is converted to 32 bits before writing.
desispec.io.table¶
Utility functions for reading FITS tables
- desispec.io.table.read_table(filename, ext=None)[source]¶
Reads a FITS table into an astropy Table, avoiding masked columns
- Parameters:
filename (str) – full path to input fits file
- Options:
ext (str or int): EXTNAME or extension number to read
Context: astropy 5.0 Table.read converts NaN and blank strings to masked values, which is a pain. This function reads the file with fitsio and then converts to a Table.
desispec.io.tpcorrparam¶
desispec.io.util¶
Utility functions for desispec IO.
- desispec.io.util._dict2ndarray(data, columns=None)[source]¶
Convert a dictionary of ndarrays into a structured ndarray
Also works if DATA is an AstroPy Table.
- Parameters:
data – input dictionary, each value is an ndarray
columns – optional list of column names
- Returns:
structured numpy.ndarray with named columns from input data dictionary
Notes
data[key].shape[0] must be the same for every key every entry in columns must be a key of data
- Example
d = dict(x=np.arange(10), y=np.arange(10)/2) nddata = _dict2ndarray(d, columns=[‘x’, ‘y’])
- desispec.io.util._supports_memmap(filename)[source]¶
Returns
True
if the filesystem containing filename supports opening memory-mapped files in update mode.
- desispec.io.util.add_columns(data, colnames, colvals)[source]¶
Adds extra columns to a data table
- Parameters:
data – astropy Table or numpy structured array
colnames – list of new column names to add
colvals – list of column values to add; each element can be scalar or vector
- Returns:
new table with extra columns added
Example
- fibermap = add_columns(fibermap,
[‘NIGHT’, ‘EXPID’], [20102020, np.arange(len(fibermap))])
Notes
This is similar to numpy.lib.recfunctions.append_fields, but it also accepts astropy Tables as the data input, and accepts scalar values to expand as entries in colvals.
- desispec.io.util.addkeys(hdr1, hdr2, skipkeys=None)[source]¶
Add new header keys from hdr2 to hdr1, skipping skipkeys
- Parameters:
hdr1 (dict-like) – destination header for keywords
hdr2 (dict-like) – source header for keywords
Modifies hdr1 in place
- desispec.io.util.backup_filename(filename)[source]¶
rename filename to next available filename.N
- Parameters:
filename (str) – full path to filename
- Returns:
New filename.N, or filename if original file didn’t already exist
- desispec.io.util.camword_intersection(camwords, full_spectros_only=False)[source]¶
Return the camword intersection of cameras in a list of camwords
- Parameters:
camwords – list of str camwords
- Options:
full_spectros_only: if True, only include spectrographs that have all 3 brz cameras
- Returns:
intersection of input camwords
- Return type:
final_camword (str)
- desispec.io.util.camword_to_spectros(camword, full_spectros_only=False)[source]¶
Takes a camword as input and returns any spectrograph represented within that camword. By default this includes partial spectrographs (with one or two cameras represented). But if full_spectros_only is set to True, only spectrographs with all cameras represented are given.
- Parameters:
camword –
cameras. (str. The camword of all) –
full_spectros_only – in the camword (the default) or if you only want fully populated spectrographs.
existing (bool. Default is False. Flag to specify if you want all spectrographs with any cameras) – in the camword (the default) or if you only want fully populated spectrographs.
- Returns:
spectros, list. A list of integer spectrograph numbers represented in the camword input.
- desispec.io.util.camword_union(camwords, full_spectros_only=False)[source]¶
Returns the union of a list of camwords. Optionally can return only those spectros with complete b, r, and z cameras. Note this intentionally does the union before truncating spectrographs, so two partial camwords can lead to an entire spectrograph,
- e.g. [a0b1z1, a3r1z2] -> [a013z2] if full_spectros_only=False
[a0b1z1, a3r1z2] -> [a013] if full_spectros_only=True
even through no camword has a complete set of camera 1, a complete set is represented in the union.
- Parameters:
camwords –
camwords. (list or array of strings. List of) –
full_spectros_only – b, r, and z cameras in the funal union should be returned.
with (bool. True if only complete spectrographs) – b, r, and z cameras in the funal union should be returned.
- Returns:
- final_camword, str. The final union of all input camwords, where
truncation of incomplete spectrographs may or may not be performed based on full_spectros_only.
- desispec.io.util.checkgzip(filename)[source]¶
Check for existence of filename, with or without .gz extension
- Parameters:
filename (str) – filename to check for
Returns path of existing file without or without .gz, or raises FileNotFoundError if neither exists
- desispec.io.util.columns_to_goodcamword(camword, badcamword, badamps=None, obstype=None, suppress_logging=False)[source]¶
Takes a camera information and obstype and returns the correct camword of good cameras for that type of observation.
- Parameters:
- Returns:
- Camword for that observation given the obstype and
input camera information.
- Return type:
goodcamword (str)
- desispec.io.util.create_camword(cameras)[source]¶
Function that takes in a list of cameras and creates a succinct listing of all spectrographs in the list with cameras. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.
- Parameters:
cameras (1-d array or list) – iterable containing strings of cameras, e.g. ‘b0’,’r1’,…
- Returns (str):
A string representing all information about the spectrographs/cameras given in the input iterable, e.g. a01234678b59z9
- desispec.io.util.decode_camword(camword)[source]¶
Function that takes in a succinct listing of all spectrographs and outputs a 1-d numpy array with a list of all spectrograph/camera pairs. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.
- Parameters:
camword (str) – A string representing all information about the spectrographs/cameras e.g. a01234678b59z9
- Returns (np.ndarray, 1d): an array containing strings of
cameras, e.g. ‘b0’,’r1’,…
- desispec.io.util.difference_camwords(fullcamword, badcamword, suppress_logging=False)[source]¶
Returns the difference of two camwords. The second argument cameras are removed from the first argument and the remainer is returned. Smart enough to ignore bad cameras if they don’t exist in full camword list.
- Parameters:
fullcamword (including the bad ones to be removed) –
cameras (str. The camword of all) –
badcamword –
output (str. A camword defining the bad cameras you don't want to include in the final camword that is) –
- Returns:
str. A camword of cameras in fullcamword that are not in badcamword.
- desispec.io.util.erow_to_goodcamword(erow, suppress_logging=False)[source]¶
Takes a dictionary-like object with at minimum keys for OBSTYPE (type str), CAMWORD (type str), BADCAMWORD (type str), BADAMPS (str).
- desispec.io.util.fitsheader(header)[source]¶
Convert header into astropy.io.fits.Header object.
- header can be:
None: return a blank Header
list of (key, value) or (key, (value,comment)) entries
dict d[key] -> value or (value, comment)
Header: just return it unchanged
Returns fits.Header object
- desispec.io.util.get_log_pathname(data_pathname)[source]¶
Returns unique log pathname for a given data product.
- Parameters:
data_pathname (str) – input filename of output data
- Returns:
pathname to the log
- Return type:
log_pathname (str)
Returns unique filename in same directory
- desispec.io.util.get_speclog(nights, rawdir=None)[source]¶
Scans raw data headers to return speclog of observations. Slow.
- Parameters:
nights – list of YEARMMDD nights to scan
- Options:
rawdir (str): overrides $DESI_SPECTRO_DATA
- Returns:
Table with columns NIGHT,EXPID,MJD,FLAVOR,OBSTYPE,EXPTIME
Scans headers of rawdir/NIGHT/EXPID/desi-EXPID.fits.fz
- desispec.io.util.get_tempfilename(filename)[source]¶
Returns unique tempfile in same directory as filename with same extension
- Parameters:
filename (str) – input filename
Returns unique filename in same directory
Example intended usage:
tmpfile = get_tempfile(filename) table.write(tmpfile) os.rename(tmpfile, filename)
By keeping the same extension as the input file, this preserves the ability of table.write to derive the format to use, and if something goes wrong with the I/O it doesn’t leave a corrupted partially written file with the final name. The tempfile includes the PID to provide some race condition protection (the last one do do os.rename wins, but at least different processes won’t corrupt each other’s files).
- desispec.io.util.header2wave(header)[source]¶
Converts header keywords into a wavelength grid.
returns wave = CRVAL1 + range(NAXIS1)*CDELT1
if LOGLAM keyword is present and true/non-zero, returns 10**wave
- desispec.io.util.healpix_subdirectory(nside, pixel)[source]¶
Return a fixed directory path for healpix grouped files.
Given an NSIDE and NESTED pixel index, return a directory named after a degraded NSIDE and pixel followed by the original nside and pixel. This function is just to ensure that this subdirectory path is always created by the same code.
- Returns (str):
a path containing the low and high resolution directories.
- desispec.io.util.is_svn_current(dirname)[source]¶
Return True/False for if svn checkout dirname is up-to-date with server
Raises ValueError if unable to determine (e.g. dirname isn’t svn checkout)
- desispec.io.util.iterfiles(root, prefix, suffix=None)[source]¶
Returns iterator over files starting with prefix found under root dir Optionally also check if filename ends with suffix
- desispec.io.util.makepath(outfile, filetype=None)[source]¶
Create path to outfile if needed.
If outfile isn’t a string and filetype is set, interpret outfile as a tuple of parameters to locate the file via findfile().
Returns /path/to/outfile
TODO: maybe a different name?
- desispec.io.util.native_endian(data)[source]¶
Convert numpy array data to native endianness if needed.
Returns new array if endianness is swapped, otherwise returns input data
Context: By default, FITS data from astropy.io.fits.getdata() are not Intel native endianness and scipy 0.14 sparse matrices have a bug with non-native endian data.
- desispec.io.util.parse_badamps(badamps, joinsymb=',')[source]¶
Parses badamps string from an exposure or processing table into the (camera,petal,amplifier) sets, with appropriate checking of those values to make sure they’re valid. Returns empty list if badamps is None.
- Parameters:
badamps – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.
(comma (str. A string of {camera}{petal}{amp} entries separated by symbol given with joinsymb) – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.
joinsymb –
badamps. (str. The symbol separating entries in the str list given by) –
- Returns:
- cam_petal_amps, list. A list where each entry is a length 3 tuple of (camera,petal,amplifier).
Camera is a lowercase string in [b, r, z]. Petal is an int from 0 to 9. Amplifier is an upper case string in [A, B, C, D].
- desispec.io.util.parse_cameras(cameras, loglevel='INFO')[source]¶
Function that takes in a representation of all spectrographs and outputs a string that succinctly lists all spectrograph/camera pairs. It uses “a” followed by numbers to mean that “all” (b,r,z) cameras are accounted for for those numbers. b, r, and z represent the camera of the same name. All trailing numbers represent the spectrographs for which that camera exists in the list.
- Parameters:
cameras – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().
array (str. 1-d) – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().
list – Either a str that is a comma separated list or a series of spectrographs. Also accepts a list or iterable that is processed with create_camword().
- Options:
loglevel, str: use e.g. “WARNING” to avoid INFO-level log messages for just this call
- Returns (str):
- camword, str. A string representing all information about the spectrographs/cameras
given in the input iterable, e.g. a01234678b59z9
- desispec.io.util.relsymlink(src, dst, pathonly=False)[source]¶
Create a relative symlink from dst -> src, while also handling $DESI_ROOT vs $DESI_ROOT_READONLY
- Options:
pathonly (bool): return the path, but don’t make the link
- Returns:
the releative path from dst -> src
- desispec.io.util.replace_prefix(filepath, oldprefix, newprefix)[source]¶
Replace filename prefix even if prefix is elsewhere in path or filename
- Parameters:
filepath – filename, optionally including path
oldprefix – original prefix to filename part of filepath
newprefix – new prefix to use for filename
- Returns:
new filepath with replaced prefix
e.g. replace_prefix(‘/blat/foo/blat-bar-blat.fits’, ‘blat’, ‘quat’) returns ‘/blat/foo/quat-bar-blat.fits’
- desispec.io.util.spectros_to_camword(spectros)[source]¶
Converts a specification of spectrographs to the equivalent camword
- Parameters:
spectros – str (e.g. “0-7,9”) or list of ints spectrographs
Returns: camword str (e.g. a012345679)
- desispec.io.util.validate_badamps(badamps, joinsymb=',')[source]¶
Checks (and transforms) badamps string for consistency with the for need in an exposure or processing table for use in the Pipeline. Specifically ensure they come in (camera,petal,amplifier) sets, with appropriate checking of those values to make sure they’re valid. Returns the input string except removing whitespace and replacing potential character separaters with joinsymb (default ‘,’). Returns None if None is given.
- Parameters:
badamps – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.
(comma (str. A string of {camera}{petal}{amp} entries separated by symbol given with joinsymb) – by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’.
joinsymb –
badamps. (str. The symbol separating entries in the str list given by) –
- Returns:
- newbadamps, str. Input badamps string of {camera}{petal}{amp} entries separated by symbol given with
joinsymb (comma by default). I.e. [brz][0-9][ABCD]. Example: ‘b7D,z8A’. Differs from input in that other symbols used to separate terms are replaced by joinsymb and whitespace is removed.
- desispec.io.util.write_bintable(filename, data, header=None, comments=None, units=None, extname=None, clobber=False, primary_extname='PRIMARY')[source]¶
Utility function to write a fits binary table complete with comments and units in the FITS header too. DATA can either be dictionary, an Astropy Table, a numpy.recarray or a numpy.ndarray.
- Parameters:
filename – full path to filename to write
data – dict or table-like data to write
- Options:
header: dict-like header key/value pairs to propagate comments (dict): comments[COLNAME] per column units (dict): units[COLNAME] per column extname (str): extension name for EXTNAME header clobber (bool): if True, overwrite pre-existing file primary_extname (str): EXTNAME to use for primary HDU=0
desispec.io.xytraceset¶
I/O routines for XYTraceSet objects.
- desispec.io.xytraceset.read_xytraceset(filename)[source]¶
Reads traces in PSF fits file
- Parameters:
filename – Path to input fits file which has to contain XTRACE and YTRACE HDUs
- Returns:
XYTraceSet object
- desispec.io.xytraceset.write_xytraceset(outfile, xytraceset)[source]¶
Write a traceset fits file and returns path to file written.
- Parameters:
outfile – full path to output file
xytraceset – desispec.xytraceset.XYTraceSet object
- Returns:
full filepath of output file that was written
desispec.joincosmics¶
Utility functions to join detected cosmic rays
- class desispec.joincosmics.LinearSelectionElement(n, m, angle)[source]¶
Define a selection element for morphological binary image processing.
- desispec.joincosmics.downsample_image(image, n)[source]¶
Downsample 2D input image n x n.
- Parameters:
image (ndarray) – 2D input image.
n (int) – Downsampling factor, applied to both image dimensions.
- Returns:
result – Resampled image with shape = image.shape//n.
- Return type:
ndarray
desispec.linalg¶
Some linear algebra functions.
- desispec.linalg.cholesky_invert(A)[source]¶
returns the inverse of a positive definite matrix
- Args :
A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray)
- Returns:
2D positive definite matrix, inverse of A (numpy.ndarray)
- Return type:
cov
- desispec.linalg.cholesky_solve(A, B, overwrite=False, lower=False)[source]¶
Returns the solution X of the linear system A.X=B assuming A is a positive definite matrix
- Args :
A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray) B : 1D vector, must have dimension n (numpy.ndarray)
- Options :
overwrite: replace A data by cholesky decomposition (faster) lower: cholesky decomposition triangular matrix is lower instead of upper
- Returns :
X : 1D vector, same dimension as B (numpy.ndarray)
- desispec.linalg.cholesky_solve_and_invert(A, B, overwrite=False, lower=False)[source]¶
returns the solution X of the linear system A.X=B assuming A is a positive definite matrix
- Args :
A : 2D (real symmetric) (nxn) positive definite matrix (numpy.ndarray) B : 1D vector, must have dimension n (numpy.ndarray)
- Options :
overwrite: replace A data by cholesky decomposition (faster) lower: cholesky decomposition triangular matrix is lower instead of upper
- Returns:
X,cov, where X : 1D vector, same dimension n as B (numpy.ndarray) cov : 2D positive definite matrix, inverse of A (numpy.ndarray)
- desispec.linalg.spline_fit(output_wave, input_wave, input_flux, required_resolution, input_ivar=None, order=3, max_resolution=None)[source]¶
Performs spline fit of input_flux vs. input_wave and resamples at output_wave
- Parameters:
output_wave – 1D array of output wavelength samples
input_wave – 1D array of input wavelengths
input_flux – 1D array of input flux density
required_resolution (float) – resolution for spline knot placement (same unit as wavelength)
- Options:
input_ivar : 1D array of weights for input_flux order (int) : spline order max_resolution (float) : if not None and first fit fails, try once this resolution
- Returns:
1D array of flux sampled at output_wave
- Return type:
output_flux
desispec.log¶
This is a transitional dummy wrapper on desiutil.log
.
- desispec.log.get_logger(*args, **kwargs)[source]¶
Transitional dummy wrapper on
desiutil.log.get_logger()
.
desispec.magnitude¶
Broadband flux and magnitudes.
- desispec.magnitude.ab_flux_in_ergs_s_cm2_A(wave)[source]¶
- Parameters:
wave – 1D numpy array (Angstrom)
- Returns:
ab flux in units of ergs/s/cm2/A
- desispec.magnitude.compute_ab_mag(spectrum_wave, spectrum_flux, transmission_wave, transmission_value)[source]¶
Computes ab mag
- Parameters:
spectrum_wave – 1D numpy array (Angstrom)
spectrum_flux – 1D numpy array (in units of 1e-17 ergs/s/cm2/A), same size as spectrum_wave
transmission_wave – 1D numpy array (Angstrom)
transmission_value – 1D numpy array , dimensionless, same size as transmission_wave
- Returns:
mag (float scalar)
- desispec.magnitude.compute_broadband_flux(spectrum_wave, spectrum_flux, transmission_wave, transmission_value)[source]¶
Computes broadband flux
- Parameters:
spectrum_wave – 1D numpy array (Angstrom)
spectrum_flux – 1D numpy array is some input density unit, same size as spectrum_wave
transmission_wave – 1D numpy array (Angstrom)
transmission_value – 1D numpy array , dimensionless, same size as transmission_wave
- Returns:
integrated flux (unit= A x (input density unit)) , scalar
desispec.maskbits¶
Mask bits for the spectro pipeline.
Stephen Bailey, LBNL, January 2015
Example:
from desispec.maskbits import ccdmask
#- bit operations
mask |= ccdmask.COSMIC #- set ccdmask.COSMIC in integer/array `mask`
mask & ccdmask.COSMIC #- get ccdmask.COSMIC from integer/array `mask`
(mask & ccdmask.COSMIC) != 0 #- test boolean status of ccdmask.COSMIC in integer/array `mask`
ccdmask.COSMIC | specmask.SATURATED #- Combine two bitmasks.
#- bit attributes
ccdmask.mask('COSMIC') #- 2**0, same as ccdmask.COSMIC
ccdmask.mask(0) #- 2**0, same as ccdmask.COSMIC
ccdmask.COSMIC #- 2**0, same as ccdmask.mask('COSMIC')
ccdmask.bitnum('COSMIC') #- 0
ccdmask.bitname(0) #- 'COSMIC'
ccdmask.names() #- ['COSMIC', 'HOT', 'DEAD', 'SATURATED', ...]
ccdmask.names(3) #- ['COSMIC', 'HOT']
ccdmask.comment(0) #- "Cosmic ray"
ccdmask.comment('BADPIX') #- "Cosmic ray"
desispec.maskedmedian¶
Utility function to perform a median of images with masks.
- desispec.maskedmedian.masked_median(images, masks=None)[source]¶
Perfomes a median of an list of input images. If a list of mask is provided, the median is performed only on unmasked pixels.
- Parameters:
images – 3D numpy array : list of images of same shape
- Options:
masks : list of mask images of same shape as the images. Only pixels with mask==0 are considered in the median.
Returns : median image
desispec.mgii_afterburner¶
Authors: Edmond CHAUSSIDON (CEA Saclay), Corentin RAVOUX (CEA Saclay) Contact: edmond.chaussidon@cea.fr, corentin.ravoux@cea.fr
Note: Please see https://desi.lbl.gov/trac/attachment/wiki/TargetSelectionWG/TSTeleconMinutes/DESITS210113/QSO_completeness_Ravoux_13_01_2021_v2.pdf for details about the MgII fitter
the “optimal”/chosen parameters are (can be modify when desi_qso_mgii_afterburner is called): lambda_width = 250 max_sigma = 200 min_sigma = 10 min_deltachi2 = 16 min_signifiance_A = 3 min_A = 0
- desispec.mgii_afterburner.create_mask_fit(fit_results, max_sigma=None, min_sigma=None, min_deltachi2=None, min_A=None, min_signifiance_A=None)[source]¶
Create a mask based on the results of the MgII fit and some parameters which constraints these parameters
- Parameters:
fit_results (numpy array) – Array containing the parameters of the fit
max_sigma (float) – Maximal value for the standard deviation of the fitted Gaussian peak
min_sigma (float) – Minimal value for the standard deviation of the fitted Gaussian peak
min_deltachi2 (float) – Minimal value for the difference of chi2 between redrock fit and MgII fitter over the lambda_width interval considered
min_A (float) – Minimal value for the amplitude of the fitted Gaussian peak
min_signifiance_A (float) – Minimal signifiance for the amplitude of the fitted Gaussian peak. The signifiance is here define as the ratio between peak amplitude and error on the peak amplitude.
- Returns:
- mask with the same length than
fit_results which indicates the objects validating parameter constraints
- Return type:
mask (boolean numpy array)
- desispec.mgii_afterburner.create_model(spectra, redshifts, archetype_fit=False, archetypes_dir=None, template_dir=None)[source]¶
< COPY from prospect.plotframes to avoid to load prospect in desispec >
Returns model_wave[nwave], model_flux[nspec, nwave], row matched to redshifts, which can be in a different order than spectra. - redshifts must be entry-matched to spectra.
- desispec.mgii_afterburner.fit_mgii_line(target_id, redshift_redrock, flux, ivar_flux, model_flux, wavelength, lambda_width, add_linear_term=False, gaussian_smoothing_fit=None, mask_mgii=None)[source]¶
Fitting routine. Fit a Gaussian peak on preselected spectra and return the main parameters of the fit including parameter errors.
- Parameters:
target_id (numpy array) – Array containing target id of the the object to fit
redshift_redrock (numpy array) – Array containing the redshift of the the object to fit
flux (numpy array) – Array containing the full flux arrays of every object to fit
ivar_flux (numpy array) – Array containing the inverse variance arrays of every object to fit
model_flux (numpy array) – Array containing the best fit redrock model for every object to fit
wavelength (numpy array) – Array containing the wavelength
lambda_width (float) – The width in wavelength (in Angstrom) considered for the fitting arount the MgII peak
add_linear_term (boolean) – Add a linear term to the Gaussian peak term to fit the continuum.
gaussian_smoothing_fit (float) – If not None, the spectra is smoothed by the given value before the fit
mask_mgii (float) – If not None, mask a region of near the MgII peak with the given witdh to fit double MgII peak (in progress)
- Returns:
Array containing the parameters of the fit
- Return type:
fit_results (numpy array)
- desispec.mgii_afterburner.get_spectra(spectra_name, redrock_name, lambda_width, index_to_fit, template_dir=None, archetypes_dir=None)[source]¶
Get the spectra and the best fit model from a given spectra and redrock file.
- Parameters:
spectra_name (str) – The name of the spectra file.
redrock_name (str) – The name of the redrock file associated to the spectra file.
lambda_width (float) – The width in wavelength (in Angstrom) considered for the fitting arount the MgII peak
index_to_fit (boolean numpy array) – boolean array of size 500 specifing which spectra have to be used
add_linear_term (boolean) – Add a linear term to the Gaussian peak term to fit the continuum.
template_dir (str) – If the redrock template variable is not loaded by the desi environment, specify the template path
archetypes_dir (str) – If not None, use the archetypes templates in the path specified
- Returns:
Array containing target id of the the object to fit
redshift_redrock (numpy array): Array containing the redshift of the the object to fit
flux (numpy array): Array containing the full flux arrays of every object to fit
ivar_flux (numpy array): Array containing the inverse variance arrays of every object to fit
model_flux (numpy array): Array containing the best fit redrock model for every object to fit
wavelength (numpy array): Array containing the wavelength
index_with_fit (boolean numpy array): boolean array of index_to_fit size masking index where mgII fitter is not apply
- Return type:
target_id (numpy array)
- desispec.mgii_afterburner.load_redrock_templates(template_dir=None)[source]¶
< COPY from prospect.plotframes to avoid to load prospect in desispec >
Load redrock templates; redirect stdout because redrock is chatty
- desispec.mgii_afterburner.mgii_fitter(spectra_name, redrock_name, index_to_fit, lambda_width, add_linear_term=False, gaussian_smoothing_fit=None, template_dir=None, archetypes_dir=None, max_sigma=None, min_sigma=None, min_deltachi2=None, min_A=None, min_signifiance_A=None)[source]¶
MgII fitter afterburner main function. For a given spectra file and its associated redrock file (redrock output), returns a numpy mask which indicates the objects fitted by redrock as galaxies and considered as QSO by the MgII fitter. The mask is determined following input parameters.
- Parameters:
spectra_name (str) – The name of the spectra file.
redrock_name (str) – The name of the redrock file associated to the spectra file.
lambda_width (float) – The width in wavelength (in Angstrom) considered for the fitting arount the MgII peak
index_to_fit (boolean numpy array) – boolean array of size 500 specifing which spectra have to be used
add_linear_term (boolean) – Add a linear term to the Gaussian peak term to fit the continuum.
template_dir (str) – If the redrock template variable is not loaded by the desi environment, specify the template path
archetypes_dir (str) – If not None, use the archetypes templates in the path specified
max_sigma (float) – Maximal value for the standard deviation of the fitted Gaussian peak
min_sigma (float) – Minimal value for the standard deviation of the fitted Gaussian peak
min_deltachi2 (float) – Minimal value for the difference of chi2 between redrock fit and MgII fitter over the lambda_width interval considered
min_A (float) – Minimal value for the amplitude of the fitted Gaussian peak
min_signifiance_A (float) – Minimal signifiance for the amplitude of the fitted Gaussian peak. The signifiance is here define as the ratio between peak amplitude and error on the peak amplitude.
- Returns:
mask with the same length than index_with_fit which indicates object considered by MgII fitter as QSO
fit_results (numpy array): Array containing the parameters of the fit to save them
index_with_fit (boolean numpy array): boolean array of index_to_fit size masking index where mgII fitter is not apply
- Return type:
mask_fit (boolean numpy array)
desispec.night_qa¶
- desispec.night_qa._javastring()[source]¶
Return a string that embeds a date in a webpage.
Note
Credits to ADM (desitarget/QA.py).
- desispec.night_qa._read_ctedet(night, prod, ctedet_expid, petal, camera)[source]¶
Internal function used by create_ctedet_pdf(), reading the ctedet_expid preproc info.
- Parameters:
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
ctedet_expid – EXPID for the CTE diagnosis (1s FLAT, or darker science exposure) (int)
petal – 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 (int)
camera – “b”, “r”, or “z” (string)
- Returns:
a dictionary with the IMAGE data, plus various infos, assuming a preproc file is available, otherwise
None
.- Return type:
mydict
- desispec.night_qa._read_dark(fn, night, prod, dark_expid, petal, camera, binning=4)[source]¶
Internal function used by create_dark_pdf(), to read and bin the 300s dark.
- Parameters:
fn – full path to the dark image (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
dark_expid – EXPID of the 300s DARK exposure to display (int)
petal – 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 (int)
camera – “b”, “r”, or “z” (string)
binning (optional, defaults to 4) – binning of the image (which will be beforehand trimmed) (int)
- Returns:
a dictionary with the binned+masked image, plus various infos, assuming a preproc file is available. Otherwise
None
.- Return type:
mydict
- desispec.night_qa._read_sframesky(night, prod, expid)[source]¶
Internal function called by create_sframesky_pdf, which generates the per-expid sframesky plot.
- Parameters:
outpng – output png file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
expid – expid to display (int)
- Returns:
mydict, dictionary with per-camera wave, flux, and various infos, assuming sframe files were available, otherwise
None
.
- desispec.night_qa.create_badcol_png(outpng, night, prod, n_previous_nights=10)[source]¶
For a given night, create a png file with displaying the number of bad columns per {camera}{petal}.
- Parameters:
outpng – output png file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
n_previous_nights (optional, defaults to 10) – number of previous nights to plot (int)
- desispec.night_qa.create_ctedet_pdf(outpdf, night, prod, ctedet_expid, nproc, nrow=21, xmin=None, xmax=None, ylim=(-5, 10), yoffcols={'A': -2.5, 'B': -3.5, 'C': -2.5, 'D': -3.5})[source]¶
For a given night, create a pdf with a CTE diagnosis (from preproc files).
- Parameters:
outpdf – output pdf file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
ctedet_expid – EXPID for the CTE diagnosis (1s FLAT, or darker science exposure) (int)
nproc – number of processes running at the same time (int)
nrow (optional, defaults to 21) – number of rows to include in median (int)
xmin (optional, defaults to None) – minimum column to display (float)
xmax (optional, defaults to None) – maximum column to display (float)
ylim (optional, default to (-5, 10)) – ylim for the median plot (duplet)
(optional (yoffcols) – -2.5, “B” : -3.5, “C” : -2.5, “D” : -3.5}): y-values to report the per-amplifier OFFCOLS info, if any (dictionnary of floats)
{"A" (defaults to) – -2.5, “B” : -3.5, “C” : -2.5, “D” : -3.5}): y-values to report the per-amplifier OFFCOLS info, if any (dictionnary of floats)
Note
Credits to S. Bailey.
Copied-pasted-adapted from /global/homes/s/sjbailey/desi/dev/ccd/plot-amp-cte.py
- desispec.night_qa.create_dark_pdf(outpdf, night, prod, dark_expid, nproc, binning=4, bkgsub_science_cameras=None, run_preproc=None)[source]¶
For a given night, create a pdf with the 300s binned dark.
- Parameters:
outpdf – output pdf file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
dark_expid – EXPID of the 300s or 1200s DARK exposure to display (int)
nproc – number of processes running at the same time (int)
binning (optional, defaults to 4) – binning of the image (which will be beforehand trimmed) (int)
bkgsub_science_cameras (optional, defaults to None) – comma-separated list of the cameras to be additionally processed with the –bkgsub-for-science argument (e.g. “b”) (string)
run_preproc (optional, defaults to None) – if None, autoderive if preproc should be run, otherwise override if it should be run or not (None or bool)
Note
If the identified dark image is not processed and if the raw data is there, we do process it (in a temporary folder), so that we can generate the plots.
- desispec.night_qa.create_mp4(fns, outmp4, duration=15)[source]¶
Create an animated .mp4 from a set of input files (usually pngs).
- Parameters:
fns – list of input filenames, in the correct order (list of strings)
outmp4 – output .mp4 filename
duration (optional, defaults to 15) – video duration in seconds (float)
Note
Requires ffmpeg to be installed.
At NERSC, run in the bash command line: “module load ffmpeg”.
The movie uses fns in the provided order.
- desispec.night_qa.create_petalnz_pdf(outpdf, night, prod, tileids, surveys, dchi2_threshold=25, group='cumulative', newlya_ecsv=None)[source]¶
For a given night, create a per-petal, per-tracer n(z) pdf file.
- Parameters:
outpdf – output pdf file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
tileids – list of tileids to consider (list or numpy array)
surveys – list of the surveys for each tileid of tileids (list or numpy array)
dchi2_threshold (optional, defaults to 25) – DELTACHI2 value to split the sample (float)
group (optional, str) – tile group “cumulative” or “pernight”
newlya_ecsv (optional, defaults to None) – if set, table saving the per-tile number of newly identified Ly-a and the tile coverage (if no dark tiles, no file will be saved).
Note
Only displays:
sv1, sv2, sv3, main, as otherwise the plotted tracers are not relevant;
FAPRGRM=”bright” or “dark” tileids;
for the main survey, tiles with EFFTIME > MINTFRAC * GOALTIME.
If the tile-qa-TILEID-thruNIGHT.fits file is missing, that tileid is skipped.
For the Lya, work from the zmtl*fits files, trying to mimick what is done in desitarget.mtl.make_mtl().
The LRG, ELG, QSO, BGS_BRIGHT, BGS_FAINT bit are the same for sv1, sv2, sv3, main, so ok to simply use the bit mask values from the main.
TBD : we query the FAPRGRM of the
tile-qa-*.fits
header, not sure that properly works for surveys other than main..
- desispec.night_qa.create_sframesky_pdf(outpdf, night, prod, expids, nproc)[source]¶
For a given night, create a pdf from per-expid sframe for the sky fibers only.
- Parameters:
outpdf – output pdf file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
expids – expids to display (list or np.array)
nproc – number of processes running at the same time (int)
- desispec.night_qa.create_skyzfiber_png(outpng, night, prod, tileids, dchi2_threshold=9, group='cumulative')[source]¶
- For a given night, create a Z vs. FIBER plot for all SKY fibers, and one for
each of the main backup/bright/dark programs
- Parameters:
outpdf – output pdf file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
tileids – list of tileids to consider (list or numpy array)
dchi2_threshold (optional, defaults to 9) – DELTACHI2 value to split the sample (float)
group (str, optional) – tile group “cumulative” or “pernight”
Note
Work from the
redrock-*.fits
files.
- desispec.night_qa.create_tileqa_pdf(outpdf, night, prod, expids, tileids, group='cumulative')[source]¶
For a given night, create a pdf from the tile-qa*png files, sorted by increasing EXPID.
- Parameters:
outpdf – output pdf file (string)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
expids – expids of the tiles to display (list or np.array)
tileids – tiles to display (list or np.array)
group (str, optional) – tile group “cumulative” or “pernight”
- desispec.night_qa.get_ctedet_night_expid(night, prod)[source]¶
Returns the EXPID of the 1s FLAT exposure for a given night. If not present, takes the science exposure with the lowest sky counts.
- Parameters:
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
- Returns:
EXPID (int)
- Return type:
expid
Note
If nothing found, returns None.
We look for preproc files.
As we are looking for a faint signal, we want the image with the less electrons, thus it could be picking a BRIGHT short exposure against a longer DARK exposure.
- desispec.night_qa.get_dark_night_expid(night, prod)[source]¶
Returns the EXPID of the 300s DARK exposure for a given night.
- Parameters:
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
- Returns:
EXPID (int)
- Return type:
expid
Note
If nothing found, returns None.
20220110 : new method, relying on processing_tables
- desispec.night_qa.get_morning_dark_night_expid(night, prod, exptime=1200)[source]¶
Returns the EXPID of the latest 1200s DARK exposure for a given night.
- Parameters:
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
exptime (optional, defaults to 1200) – exposure time we are looking for (float)
- Returns:
EXPID (int)
- Return type:
expid
Note
If nothing found, returns None.
As of now (20221220), those morning darks are not processed by the daily pipeline; hence we do not use the processing_tables, but the exposure_tables.
- desispec.night_qa.get_nightqa_outfns(outdir, night)[source]¶
Utility function to get nightqa file names.
- Parameters:
outdir – output folder name (string)
night – night (int)
- Returns:
dictionary with file names
- desispec.night_qa.get_surveys_night_expids(night, datadir=None)[source]¶
List the (EXPIDs, TILEIDs) from a given night for a given survey.
- Parameters:
night – night (int)
surveys – comma-separated list of surveys to consider, in lower-cases, e.g. “sv1,sv2,sv3,main” (str)
datadir (optional, defaults to $DESI_SPECTRO_DATA) – full path where the {NIGHT}/desi-{EXPID}.fits.fz files are (str)
- Returns:
list of the EXPIDs (np.array()) tileids: list of the TILEIDs (np.array()) surveys: list of the SURVEYs (np.array())
- Return type:
expids
Note
Based on: - parsing the OBSTYPE keywords from the SPEC extension header of the desi-{EXPID}.fits.fz files; - for OBSTYPE=”SCIENCE”, parsing the
fiberassign-TILEID.fits*
header
- desispec.night_qa.path_full2web(fn)[source]¶
Convert full path to web path (needs DESI_ROOT to be defined).
- Parameters:
fn – filename full path
- Returns:
Web path
- desispec.night_qa.plot_newlya(ax, ntilecovs, nnewlyas, tileids=None, title=None, xlabel='Number of observed tiles', ylabel='(N_newlya_obs / N_newlya_expect) - 1', xlim=(0.5, 6.5), ylim=(-2.5, 2.5), nvalifiber_norm=3900)[source]¶
- Plot the ratio of the number of newly identified Ly-a to the expected number,
vs. the tile observations coverage for a list of tiles.
- Parameters:
ax – pyplot Axes object
ntilecovs – average number of previously observed tiles for each number of pass (np.array(len(tiles), n_dark_passids))
nnewlyas – list of nb of newly identified Ly-a (with a valid fiber), normalized to nvalifiber_norm (list or np.array() of ints)
tileids (optional, defaults to None) – tileids (to report outliers) (list or np.array() of ints)
title (optional, defaults to None) – plot title (str)
xlabel (optional, defaults to "Number of observed tiles") – plot xlabel (str)
ylabel (optional, defaults to "Number of newly identified LYA candidates") – plot ylabel(str)
xlim (optional, defaults to (0.5, 6.5)) – plot xlim (tuple)
ylim (optional, defaults to (-2.5, 2.5) – plot ylim (tuple)
nvalifiber_norm (optional, defaults to 3900) – number of valid fibers to normalize to, for the expected regions (int)
Note
ntilecovs[:, 0] lists the fractions of the tiles covered by 1 tile, etc.
The plotted y-values are: (N_newlya_observed / N_newlya_expected) - 1.
The expected numbers are based on all main dark tiles (from daily) up to May 26th 2022.
The 1-2-3-sigma regions reflect the approximate scatter of those data.
- desispec.night_qa.write_html_collapse_script(html, classname)[source]¶
Write the required lines to have the collapsing sections working.
- Parameters:
html – an html file object.
classname – “collapsible” only for now (string)
- desispec.night_qa.write_html_today(html)[source]¶
Write in an html object today’s date.
- Parameters:
html – html file object.
- desispec.night_qa.write_nightqa_html(outfns, night, prod, css, expids, tileids, surveys)[source]¶
Write the nightqa-{NIGHT}.html page.
- Parameters:
outfns – dictionary with filenames generated by desi_night_qa (output from get_nightqa_outfns)
night – night (int)
prod – full path to prod folder, e.g. /global/cfs/cdirs/desi/spectro/redux/blanc (string)
css – path to the nightqa.css file
expids – EXPIDs for the considered science exposures (np.array() of ints)
tileids – TILEIDs for the considered science exposures (np.array() of ints)
surveys – SURVEYs for the considered science exposures (np.array() of strings)
Notes
expids, tileids, surveys: fiducial use is that those are the outputs of get_surveys_night_expids()
desispec.parallel¶
Helper functions and classes for dealing with parallelization and related topics.
- desispec.parallel.default_nproc = 1¶
Default number of multiprocessing processes. Set globally on first import.
- desispec.parallel.dist_balanced(nwork, maxworkers)[source]¶
Distribute items between a flexible number of workers.
This assumes that each item has equal weight, and that they should be divided into contiguous blocks of items and assigned to workers in rank order.
If the number of workers is less than roughly sqrt(nwork), then we do not reduce the number of workers and the result is the same as the dist_uniform function. If there are more workers than this, then the number of workers is reduced until all workers have close to the same number of tasks.
- Parameters:
- Returns:
A list of tuples, one for each worker. The first element of the tuple is the first item assigned to the worker, and the second element is the number of items assigned to the worker.
- desispec.parallel.dist_discrete(worksizes, nworkers, workerid, power=1.0)[source]¶
Distribute indivisible blocks of items between groups.
Given some contiguous blocks of items which cannot be subdivided, distribute these blocks to the specified number of groups in a way which minimizes the maximum total items given to any group. Optionally weight the blocks by a power of their size when computing the distribution.
This is effectively the “Painter”s Partition Problem”.
- Parameters:
- Returns:
A tuple. The first element of the tuple is the first block assigned to the worker ID, and the second element is the number of blocks assigned to the worker.
- desispec.parallel.dist_discrete_all(worksizes, nworkers, power=1.0)[source]¶
Distribute indivisible blocks of items between groups.
Given some contiguous blocks of items which cannot be subdivided, distribute these blocks to the specified number of groups in a way which minimizes the maximum total items given to any group. Optionally weight the blocks by a power of their size when computing the distribution.
This is effectively the “Painter”s Partition Problem”.
- desispec.parallel.dist_uniform(nwork, nworkers, id=None)[source]¶
Statically distribute some number of items among workers.
This assumes that each item has equal weight, and that they should be divided into contiguous blocks of items and assigned to workers in rank order.
This function returns the index of the first item and the number of items for the specified worker ID, or the information for all workers.
- Parameters:
- Returns (tuple):
A tuple of ints, containing the first item and number of items. If id=None, then return a list containing the tuple for all workers.
- desispec.parallel.stdouterr_redirected(to=None, comm=None, overwrite=False)[source]¶
Redirect stdout and stderr to a file.
The general technique is based on:
http://stackoverflow.com/questions/5081657 http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/
One difference here is that each process in the communicator redirects to a different temporary file, and the upon exit from the context the rank zero process concatenates these in order to the file result.
- desispec.parallel.take_turns(comm, at_a_time, func, *args, **kwargs)[source]¶
Processes call a function in groups.
Any extra positional and keyword arguments are passed to the function.
- Parameters:
comm – mpi4py.MPI.Comm or None.
at_a_time (int) – the maximum number of processes to run at a time.
func – the function to call.
- Returns:
The return value on each process is the return value of the function.
- desispec.parallel.weighted_partition(weights, n, groups_per_node=None)[source]¶
Partition weights into n groups with approximately same sum(weights)
- Parameters:
weights – array-like weights
n – number of groups
Returns list of lists of indices of weights for each group
Notes
compared to dist_discrete_all, this function allows non-contiguous items to be grouped together which allows better balancing.
desispec.pipeline¶
Tools for pipeline creation and running.
desispec.pipeline.control¶
Tools for controling pipeline production.
- desispec.pipeline.control.chain(tasktypes, nightstr=None, states=None, expid=None, spec=None, pack=False, nosubmitted=False, depjobs=None, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, dryrun=False)[source]¶
Run a chain of jobs for multiple pipeline steps.
For the list of task types, get all ready tasks meeting the selection criteria. Then either pack all tasks into one job or submit each task type as its own job. Input job dependencies can be specified, and dependencies are tracked between jobs in the chain.
- Parameters:
tasktypes (list) – list of valid task types.
nightstr (str) – Comma separated (YYYYMMDD) or regex pattern. Only nights matching these patterns will be considered.
states (list) – list of task states to select.
nights (list) – list of nights to select.
expid (int) – exposure ID to select.
pack (bool) – if True, pack all tasks into a single job.
nosubmitted (bool) – if True, do not run jobs that have already been submitted.
depjobs (list) – list of job ID dependencies.
nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
nersc_shifter (str) – the name of the shifter image to use.
mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
nodb (bool) – if True, do not use the production DB.
out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
debug (bool) – if True, enable DEBUG log level in generated scripts.
dryrun (bool) – if True, do not submit the jobs.
- Returns:
the job IDs from the final step in the chain.
- Return type:
- desispec.pipeline.control.check_tasks(tasks, db=None)[source]¶
Check the state of pipeline tasks.
If the database handle is given, use the DB for checking. Otherwise use the filesystem.
- desispec.pipeline.control.cleanup(db, tasktypes, failed=False, submitted=False, expid=None)[source]¶
Clean up stale tasks in the DB.
- desispec.pipeline.control.create(root=None, data=None, redux=None, prod=None, force=False, basis=None, calib=None, db_sqlite=False, db_sqlite_path=None, db_postgres=False, db_postgres_host='nerscdb03.nersc.gov', db_postgres_port=5432, db_postgres_name='desidev', db_postgres_user='desidev_admin', db_postgres_authorized='desidev_ro', nside=64)[source]¶
Create (or re-create) a production.
- Parameters:
root (str) – value to use for DESI_ROOT.
data (str) – value to use for DESI_SPECTRO_DATA.
redux (str) – value to use for DESI_SPECTRO_REDUX.
prod (str) – value to use for SPECPROD.
force (bool) – if True, overwrite existing production DB.
basis (str) – value to use for DESI_BASIS_TEMPLATES.
calib (str) – value to use for DESI_SPECTRO_CALIB.
db_sqlite (bool) – if True, use SQLite for the DB.
db_sqlite_path (str) – override path to SQLite DB.
db_postgres (bool) – if True, use PostgreSQL for the DB.
db_postgres_host (str) – PostgreSQL hostname.
db_postgres_port (int) – PostgreSQL connection port number.
db_postgres_name (str) – PostgreSQL DB name.
db_postgres_user (str) – PostgreSQL user name.
db_postgres_authorized (str) – Additional PostgreSQL users to authorize.
nside (int) – HEALPix nside value used for spectral grouping.
- desispec.pipeline.control.dryrun(tasks, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, db_postgres_user='desidev_ro', force=False)[source]¶
Print equivalent command line jobs.
For the specified tasks, print the equivalent stand-alone commands that would be run on each task. A pipeline job calls the internal desispec.scripts entrypoints directly.
- Parameters:
tasks (list) – list of tasks to run.
nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
nersc_shifter (str) – the name of the shifter image to use.
mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
nodb (bool) – if True, do not use the production DB.
db_postgres_user (str) – If using postgres, connect as this user for read-only access”
force (bool) – if True, print commands for all tasks, not just the ones in a ready state.
- desispec.pipeline.control.gen_scripts(tasks_by_type, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, db_postgres_user='desidev_ro')[source]¶
Generate scripts to run tasks of one or more types.
If multiple task type keys are contained in the dictionary, they will be packed into a single batch job.
- Parameters:
tasks_by_type (dict) – each key is the task type and the value is a list of tasks.
nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
nersc_shifter (str) – the name of the shifter image to use.
mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
nodb (bool) – if True, do not use the production DB.
out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
debug (bool) – if True, enable DEBUG log level in generated scripts.
db_postgres_user (str) – If using postgres, connect as this user for read-only access”
- Returns:
the generated script files
- Return type:
- desispec.pipeline.control.get_tasks(db, tasktypes, nights, states=None, expid=None, spec=None, nosubmitted=False, taskfile=None)[source]¶
Get tasks of multiple types that match certain criteria.
- Parameters:
db (DataBase) – the production DB.
tasktypes (list) – list of valid task types.
states (list) – list of task states to select.
nights (list) – list of nights to select.
expid (int) – exposure ID to select.
spec (int) – spectrograph to select.
nosubmitted (bool) – if True, ignore tasks that were already submitted.
- Returns:
all tasks of all types.
- Return type:
- desispec.pipeline.control.get_tasks_type(db, tasktype, states, nights, expid=None, spec=None)[source]¶
Get tasks of one type that match certain criteria.
- Parameters:
- Returns:
list of tasks meeting the criteria.
- Return type:
(list)
- desispec.pipeline.control.getready(db, nightstr=None)[source]¶
Update forward dependencies in the database.
Update database for one or more nights to ensure that forward dependencies know that they are ready to run.
- desispec.pipeline.control.run(taskfile, nosubmitted=False, depjobs=None, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False)[source]¶
Create job scripts and run them.
This gets tasks from the taskfile and sorts them by type. Then it generates the scripts. Finally, it runs or submits those scripts to the scheduler.
- Parameters:
taskfile (str) – read tasks from this file (if not specified, read from STDIN).
nosubmitted (bool) – if True, do not run jobs that have already been submitted.
depjobs (list) – list of job ID dependencies.
nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
nersc_shifter (str) – the name of the shifter image to use.
mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
nodb (bool) – if True, do not use the production DB.
out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
debug (bool) – if True, enable DEBUG log level in generated scripts.
- Returns:
the job IDs returned by the scheduler.
- Return type:
- desispec.pipeline.control.run_scripts(scripts, deps=None, slurm=False)[source]¶
Run job scripts with optional dependecies.
This either submits the jobs to the scheduler or simply runs them in order with subprocess.
- desispec.pipeline.control.script(taskfile, nersc=None, nersc_queue='regular', nersc_maxtime=0, nersc_maxnodes=0, nersc_shifter=None, mpi_procs=1, mpi_run='', procs_per_node=0, nodb=False, out=None, debug=False, db_postgres_user='desidev_ro')[source]¶
Generate pipeline scripts for a taskfile.
This gets tasks from the taskfile and sorts them by type. Then it generates the scripts.
- Parameters:
taskfile (str) – read tasks from this file (if not specified, read from STDIN).
nersc (str) – if not None, the name of the nersc machine to use (cori-haswell | cori-knl).
nersc_queue (str) – the name of the queue to use (regular | debug | realtime).
nersc_maxtime (int) – if specified, restrict the runtime to this number of minutes.
nersc_maxnodes (int) – if specified, restrict the job to use this number of nodes.
nersc_shifter (str) – the name of the shifter image to use.
mpi_run (str) – if specified, and if not using NERSC, use this command to launch MPI executables in the shell scripts. Default is to not use MPI.
mpi_procs (int) – if not using NERSC, the number of MPI processes to use in shell scripts.
procs_per_node (int) – if specified, use only this number of processes per node. Default runs one process per core.
nodb (bool) – if True, do not use the production DB.
out (str) – Put task scripts and logs in this directory relative to the production ‘scripts’ directory. Default puts task directory in the main scripts directory.
debug (bool) – if True, enable DEBUG log level in generated scripts.
db_postgres_user (str) – If using postgres, connect as this user for read-only access”
- Returns:
the generated script files
- Return type:
- desispec.pipeline.control.status(task=None, tasktypes=None, nightstr=None, states=None, expid=None, spec=None, db_postgres_user='desidev_ro')[source]¶
Check the status of pipeline tasks.
Args:
- Returns:
None
- desispec.pipeline.control.sync(db, nightstr=None, specdone=False)[source]¶
Synchronize DB state based on the filesystem.
This scans the filesystem for all tasks for the specified nights, and updates the states accordingly.
- desispec.pipeline.control.tasks(tasktypes, nightstr=None, states=None, expid=None, spec=None, nosubmitted=False, db_postgres_user='desidev_ro', taskfile=None)[source]¶
Get tasks of multiple types that match certain criteria.
- Parameters:
tasktypes (list) – list of valid task types.
nightstr (list) – comma separated (YYYYMMDD) or regex pattern.
states (list) – list of task states to select.
expid (int) – exposure ID to select.
spec (int) – spectrograph to select.
nosubmitted (bool) – if True, ignore tasks that were already submitted.
db_postgres_user (str) – If using postgres, connect as this user for read-only access”
taskfile (str) – if set write to this file, else write to STDOUT.
desispec.pipeline.db¶
Pipeline processing database
- class desispec.pipeline.db.DataBase[source]¶
Class for tracking pipeline processing objects and state.
- cleanup(tasktypes=None, expid=None, cleanfailed=False, cleansubmitted=False)[source]¶
Reset states of tasks.
Any tasks that are marked as “running” will have their state reset to “ready”. This can be called if a job dies before completing all tasks.
- Parameters:
tasktypes (list) – if not None, clean up only tasks of these types.
expid (int) – if not None, only clean tasks related to this exposure ID. Note that tasks which are independent of an expid (psfnight, fiberflatnight, spectra, redshift) will be ignored if this option is given.
cleanfailed (bool) – if True, also reset failed tasks to ready.
cleansubmitted (bool) – if True, set submitted flag to False.
- getready(night=None)[source]¶
Update DB, changing waiting to ready depending on status of dependencies .
- Parameters:
night (str) – The night to process.
- set_states(tasks)[source]¶
Efficiently set the state of many tasks at once.
- Parameters:
tasks (list) – list of tuples containing the task name and the state to set.
- Returns:
Nothing.
- set_states_type(tasktype, tasks, postprocessing=True)[source]¶
Efficiently get the state of many tasks of a single type.
- set_submitted_type(tasktype, tasks, unset=False)[source]¶
Flag a list of tasks of a single type as submitted.
- sync(night, specdone=False)[source]¶
Update states of tasks based on filesystem.
Go through all tasks in the DB for the given night and determine their state on the filesystem. Then update the DB state to match.
- Parameters:
night (str) – The night to scan for updates.
specdone – If true, set spectra to done if files exist.
- class desispec.pipeline.db.DataBasePostgres(host, port, dbname, user, schema=None, authorize=None)[source]¶
Pipeline database using PostgreSQL as the backend.
- Parameters:
host (str) – The database server.
port (int) – The connection port.
dbname (str) – The database to connect.
user (str) – The user name for the connection. The password should be stored in the ~/.pgpass file.
schema (str) – The schema within the database. If this is specified, then the database is assumed to exist. Otherwise the schema is computed from a hash of the production location and will be created.
authorize (str) – If creating the schema, this is the list of additional roles that should be granted access.
- class desispec.pipeline.db.DataBaseSqlite(path, mode)[source]¶
Pipeline database using sqlite3 as the backend.
- Parameters:
- desispec.pipeline.db.all_task_types()[source]¶
Get the list of possible task types that are supported.
- Returns:
The list of supported task types.
- Return type:
- desispec.pipeline.db.all_tasks(night, nside, expid=None)[source]¶
Get all possible tasks for a single night.
This uses the filesystem to query the raw data for a particular night and return a dictionary containing all possible tasks for each task type. For objects which span multiple nights (e.g. spectra, redrock), this returns the tasks which are touched by the given night.
- desispec.pipeline.db.check_tasks(tasklist, db=None, inputs=None)[source]¶
Check a list of tasks and return their state.
If the database is specified, it is used to check the state of the tasks and their dependencies. Otherwise the filesystem is checked.
- desispec.pipeline.db.load_db(dbstring, mode='w', user=None)[source]¶
Load a database from a connection string.
This instantiates either an sqlite or postgresql database using a string. If this string begins with “postgresql:”, then it is taken to be the information needed to connect to a postgres server. Otherwise it is assumed to be a filesystem path to use with sqlite. The mode is only meaningful when using sqlite. Postgres permissions are controlled through the user permissions.
- Parameters:
dbstring (str) – either a filesystem path (sqlite) or a colon-separated string of connection properties in the form “postresql:<host>:<port>:<dbname>:<user>:<schema>”.
mode (str) – for sqlite, the mode.
user (str) – for postgresql, an alternate user name for opening the DB. This can be used to connect as a user with read-only access.
- Returns:
a derived database class of the appropriate type.
- Return type:
- desispec.pipeline.db.task_sort(tasks)[source]¶
Sort a list of tasks by type.
This takes a list of arbitrary tasks and sorts them by type. The result is placed in an ordered dictionary of lists in run order.
- Parameters:
tasks (list) – the list of input tasks.
- Returns:
ordered dictionary of tasks sorted by type.
- Return type:
(OrderedDict)
desispec.pipeline.defs¶
Common definitions needed by pipeline modules.
- desispec.pipeline.defs.prod_options_name = 'options.yaml'¶
The name of the options file inside the run directory.
- desispec.pipeline.defs.state_colors = {'done': '#00ff00', 'failed': '#ff0000', 'ready': '#0000ff', 'running': '#ffff00', 'waiting': '#000000'}¶
State colors used for visualization.
- desispec.pipeline.defs.task_name_sep = '_'¶
The separator string used for building object names.
- desispec.pipeline.defs.task_states = ['waiting', 'ready', 'running', 'done', 'failed']¶
The valid states of each pipeline task.
desispec.pipeline.plan¶
Tools for planning pipeline runs.
- desispec.pipeline.plan.compute_nodes(nworker, taskproc, nodeprocs)[source]¶
Compute number of nodes for the number of workers.
- desispec.pipeline.plan.compute_worker_tasks(tasktype, tasklist, tfactor, nworker, workersize, startup=0.0, db=None, num_nodes=None)[source]¶
Compute the distribution of tasks for specified workers.
- Parameters:
tasktype (str) – The task type.
tasklist (list) – List of tasks, all of type tasktype.
tfactor (float) – Additional runtime scaling factor.
nworker (int) – The number of workers.
workersize (int) – The number of processes in each worker.
startup (float, optional) – Startup overhead in minutes for each worker.
db (DataBase, optional) – the database to pass to the task runtime calculation.
num_nodes (int, optional) – number of nodes over which the workers are distributed
- Returns:
The (sorted tasks, sorted runtime weights, dist) results where dist is the a list of tuples (one per worker) indicating the indices of tasks for that worker in the returned sorted list of tasks.
- Return type:
(tuple)
- desispec.pipeline.plan.nersc_job_size(tasktype, tasklist, machine, queue, maxtime, maxnodes, nodeprocs=None, db=None, balance=False)[source]¶
Compute the NERSC job parameters based on constraints.
Given the list of tasks, query their estimated runtimes and determine the “best” job size to use. If the job is too big to fit the constraints then split up the tasks into multiple jobs.
If maxtime or maxnodes is zero, then the defaults for the queue are used.
The default behavior is to create jobs that are as large as possible- i.e. to run all tasks simultaneously in parallel. In general, larger jobs with a shorter run time will move through the queue faster. If the job size exceeds the maximum number of nodes, then the job size is fixed to this maximum and the runtime is extended. If the runtime exceeds maxtime, then the job is split.
- Parameters:
tasktype (str) – the type of these tasks.
tasklist (list) – the list of tasks.
machine (str) – the nersc machine name, e.g. cori-haswell, cori-knl
queue (str) – the nersc queue name, e.g. regular or debug
maxtime (int) – the maximum run time in minutes.
maxnodes (int) – the maximum number of nodes.
nodeprocs (int) – the number of processes per node. If None, estimate this based on the per-process memory needs of the task and the machine properties.
db (DataBase) – the database to pass to the task runtime calculation.
balance (bool) – if True, change the number of workers to load balance the job.
- Returns:
- List of tuples (nodes, nodeprocs, runtime, nworker, workersize,
tasks) containing one entry per job. Each entry specifies the number of nodes to use, the expected total runtime, number of workers, and the list of tasks for that job.
- Return type:
- desispec.pipeline.plan.nersc_machine(name, queue)[source]¶
Return the properties of the specified NERSC host.
- desispec.pipeline.plan.worker_times(tasktimes, workerdist, startup=0.0)[source]¶
Compute the time needed for each worker.
- Parameters:
- Returns:
The (worker times, min, max).
- Return type:
(tuple)
- Notes / Examples:
len(tasktimes) = number of tasks len(workerdist) = number of workers workerdist[i] = tuple of tasktime indices assigned to worker i sum(tasktimes[workerdist[i]]) = expected total time for worker i
desispec.pipeline.prod¶
Functions for updating and loading a production.
- desispec.pipeline.prod.load_prod(mode='w', user=None)[source]¶
Load the database and options for a production.
This loads the database from the production location defined by the usual DESI environment variables. It also loads the global options file for the production.
- Parameters:
- Returns:
- (pipeline.db.DataBase, dict) The database for the production
and the global options dictionary.
- Return type:
- desispec.pipeline.prod.select_nights(allnights, nightstr)[source]¶
Select nights based on regex matches.
Given a list of nights, select all nights matching the specified patterns and return this subset.
- desispec.pipeline.prod.task_read(path)[source]¶
Read a task list from a text file or STDIN.
Lines that begin with ‘#’ are ignored as comments. If the path is None, lines are read from STDIN until an EOF marker is received.
- desispec.pipeline.prod.task_write(path, tasklist)[source]¶
Write a task list to a text file or STDOUT.
If the path is None, write lines to STDOUT. In all cases, write a special termination line so that this stream or file can be passed into the task_read function.
- desispec.pipeline.prod.update_prod(nightstr=None, hpxnside=64, expid=None)[source]¶
Create or update a production directory tree.
For a given production, create the directory hierarchy and the starting default options.yaml file if it does not exist. Also initialize the production DB if it does not exist. Then update the DB with one or more nights from the raw data. Nights to update may be specified by a list of simple regex matches.
desispec.pipeline.run¶
Tools for running the pipeline.
- desispec.pipeline.run.dry_run(tasktype, tasklist, opts, procs, procs_per_node, db=None, launch='mpirun -np', force=False)[source]¶
Compute the distribution of tasks and equivalent commands.
This function takes similar arguments as run_task_list() except simulates the data distribution and commands that would be run if given the specified number of processes and processes per node.
This can be used to debug issues with the runtime concurrency or the actual options that will be passed to the underying main() entry points for each task.
This function requires that the DESI environment variables are set to point to the current production directory.
Only tasks that are ready to run (based on the filesystem checks or the database) will actually be attempted.
NOTE: Since this function is just informative and for interactive use, we print information directly to STDOUT rather than logging.
- Parameters:
tasktype (str) – the pipeline step to process.
tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.
opts (dict) – the global options (for example, as read from the production options.yaml file).
procs (int) – the number of processes to simulate.
procs_per_node (int) – the number of processes per node to simulate.
db (pipeline.db.DB) – The optional database to update.
launch (str) – The launching command for a job. This is just a convenience and prepended to each command before the number of processes.
force (bool) – If True, ignore database and filesystem state and just run the tasks regardless.
- Returns:
Nothing.
- desispec.pipeline.run.run_dist(tasktype, tasklist, db, nproc, procs_per_node, force=False)[source]¶
Compute the runtime distribution of tasks.
For a given number of processes, parse job environment variables and compute the number of workers to use and the remaining tasks to process. Divide the processes into groups, and associate some (or all) of those groups to workers. Assign tasks to these groups of processes. Some groups may have zero tasks if there are more groups than workers needed.
- Returns:
- The (groupsize, groups, tasks, dist) information. Groupsize
is the processes per group. Groups is a list of tuples (one per process) giving the group number and rank within the group. The tasks are a sorted list of tasks containing the subset of the inputs that needs to be run. The dist is a list of tuples (one per group) containing the indices of tasks assigned to each group.
- Return type:
- desispec.pipeline.run.run_task(name, opts, comm=None, logfile=None, db=None)[source]¶
Run a single task.
Based on the name of the task, call the appropriate run function for that task. Log output to the specified file. Run using the specified MPI communicator and optionally update state to the specified database.
Note: This function DOES NOT check the database or filesystem to see if the task has been completed or if its dependencies exist. It assumes that some higher-level code has done that if necessary.
- Parameters:
- Returns:
the total number of processes that failed.
- Return type:
- desispec.pipeline.run.run_task_list(tasktype, tasklist, opts, comm=None, db=None, force=False)[source]¶
Run a collection of tasks of the same type.
This function requires that the DESI environment variables are set to point to the current production directory.
This function first takes the communicator and uses the maximum processes per task to split the communicator and form groups of processes of the desired size. It then takes the list of tasks and uses their relative run time estimates to assign tasks to the process groups. Each process group loops over its assigned tasks.
If the database is not specified, no state tracking will be done and the filesystem will be checked as needed to determine the current state.
Only tasks that are ready to run (based on the filesystem checks or the database) will actually be attempted.
- Parameters:
tasktype (str) – the pipeline step to process.
tasklist (list) – the list of tasks. All tasks should be of type “tasktype” above.
opts (dict) – the global options (for example, as read from the production options.yaml file).
comm (mpi4py.Comm) – the full communicator to use for whole set of tasks.
db (pipeline.db.DB) – The optional database to update.
force (bool) – If True, ignore database and filesystem state and just run the tasks regardless.
- Returns:
- the number of ready tasks, number that are done, and the number
that failed.
- Return type:
- desispec.pipeline.run.run_task_list_db(tasktype, tasklist, comm=None)[source]¶
Run a list of tasks using the pipeline DB and options.
This is a wrapper around run_task_list which uses the production database and global options file.
- desispec.pipeline.run.run_task_simple(name, opts, comm=None)[source]¶
Run a single task with no DB or log redirection.
This a wrapper around run_task() for use without a database and with no log redirection. See documentation for that function.
desispec.pipeline.scriptgen¶
Tools for generating shell and slurm scripts.
- desispec.pipeline.scriptgen.batch_nersc(tasks_by_type, outroot, logroot, jobname, machine, queue, maxtime, maxnodes, nodeprocs=None, openmp=False, multiproc=False, db=None, shifterimg=None, debug=False)[source]¶
Generate slurm script(s) to process lists of tasks.
Given sets of task lists and constraints about the machine, generate slurm scripts for use at NERSC.
- Parameters:
tasks_by_type (OrderedDict) – Ordered dictionary of the tasks for each type to be written to a single job script.
outroot (str) – root output script name.
logroot (str) – root output log name.
jobname (str) – the name of the job.
machine (str) – the NERSC machine name.
queue (str) – the name of the queue
maxtime (int) – the maximum run time in minutes.
maxnodes (int) – the maximum number of nodes to use.
nodeprocs (int) – the number of processes to use per node.
openmp (bool) – if True, set OMP_NUM_THREADS to the correct value.
multiproc (bool) – if True, use OMP_NUM_THREADS=1 and disable core binding of processes.
db (DataBase) – the pipeline database handle.
shifter (str) – the name of the shifter image to use.
debug (bool) – if True, set DESI log level to DEBUG in the script.
- Returns:
list of generated slurm files.
- Return type:
(list)
- desispec.pipeline.scriptgen.batch_shell(tasks_by_type, outroot, logroot, mpirun='', mpiprocs=1, openmp=1, db=None)[source]¶
Generate bash script(s) to process lists of tasks.
Given sets of task lists, generate a script that processes each in order.
- Parameters:
tasks_by_type (OrderedDict) – Ordered dictionary of the tasks for each type to be written to a single job script.
outroot (str) – root output script name.
logroot (str) – root output log name.
mpirun (str) – optional command to use for launching MPI programs.
mpiprocs (int) – if mpirun is specified, use this number of processes.
openmp (int) – value to set for OMP_NUM_THREADS.
db (DataBase) – the pipeline database handle.
- Returns:
list of generated script files.
- Return type:
(list)
- desispec.pipeline.scriptgen.dump_job_env(fh, tfactor, startup, nworker, workersize)[source]¶
Write parameters needed at runtime to an open filehandle.
- desispec.pipeline.scriptgen.nersc_job(jobname, path, logroot, desisetup, commands, machine, queue, nodes, cnodes, ppns, minutes, nworker, workersize, multisrun=False, openmp=False, multiproc=False, shifterimg=None, debug=False)[source]¶
Create a SLURM script for use at NERSC.
Args:
desispec.pipeline.tasks¶
Classes that describe pipeline tasks.
desispec.pipeline.tasks.base¶
Common operations for pipeline tasks.
- class desispec.pipeline.tasks.base.BaseTask[source]¶
Base class for tasks.
This defines the interfaces for the classes representing pipeline tasks. This class should not be instantiated directly.
- create(db)[source]¶
Initialize a database for this task type.
This may include creating one or more tables.
- Parameters:
db (pipeline.DB) – the database instance.
- deps(name, db=None, inputs=None)[source]¶
Get the dependencies for a task.
This gets a list of other tasks which are required.
- Parameters:
- Returns:
- a dictionary of dependencies. The keys are arbitrary and
the values can be either scalar task names or lists of tasks.
- Return type:
- insert(cursor, props)[source]¶
Insert a task into a database.
This uses the name and extra keywords to update one or more task-specific tables.
- Parameters:
cursor (DB cursor) – the database cursor of an open connection.
props (dict) – dictionary of properties for the task.
- retrieve(db, name)[source]¶
Retrieve all task information from the DB.
This may include additional information beyond the contents of the task name (e.g. from other tables).
- run_and_update(db, name, opts, comm=None)[source]¶
Run the task and update DB state.
The state of the task is marked as “done” if the command completes without raising an exception and if the output files exist.
- run_cli(name, opts, procs, launch=None, log=None, db=None)[source]¶
Return the equivalent command-line interface.
- Parameters:
- Returns:
a command line.
- Return type:
- run_defaults()[source]¶
Default options.
This dictionary of default options will be written to the options.yaml file in a production directory. The options will then be loaded from that file at run time.
Changes to this function will only impact newly-created productions, and these options will be overridden by any changes to the options.yaml file.
- Returns:
dictionary of default options.
- Return type:
- run_max_mem_proc(name, db=None)[source]¶
Maximum memory in GB per process required.
If zero is returned, it indicates that the memory requirement is so small that the code can run fully-packed on any system.
- run_max_mem_task(name, db=None)[source]¶
Maximum memory in GB per task required.
If zero is returned, it indicates that the memory requirement is so small that the code can run on a single node.
- desispec.pipeline.tasks.base.task_type(name)[source]¶
Given a task name, find the type from the list of available ones.
desispec.pipeline.tasks.cframe¶
- class desispec.pipeline.tasks.cframe.TaskCFrame[source]¶
Class containing the properties of a sky fit task.
desispec.pipeline.tasks.extract¶
- class desispec.pipeline.tasks.extract.TaskExtract[source]¶
Class containing the properties of one extraction task.
desispec.pipeline.tasks.fiberflat¶
- class desispec.pipeline.tasks.fiberflat.TaskFiberflat[source]¶
Class containing the properties of one extraction task.
desispec.pipeline.tasks.fiberflatnight¶
- class desispec.pipeline.tasks.fiberflatnight.TaskFiberflatNight[source]¶
Class containing the properties of one fiberflat combined night task.
desispec.pipeline.tasks.fibermap¶
- class desispec.pipeline.tasks.fibermap.TaskFibermap[source]¶
Class containing the properties of one fibermap.
Since fibermaps have no dependencies and are not created by the pipeline, this class is just used to specify names, etc.
desispec.pipeline.tasks.fluxcalib¶
- class desispec.pipeline.tasks.fluxcalib.TaskFluxCalib[source]¶
Class containing the properties of a sky fit task.
desispec.pipeline.tasks.preproc¶
- class desispec.pipeline.tasks.preproc.TaskPreproc[source]¶
Class containing the properties of one preprocessed pixel file.
desispec.pipeline.tasks.psf¶
- class desispec.pipeline.tasks.psf.TaskPSF[source]¶
Class containing the properties of one PSF task.
desispec.pipeline.tasks.psfnight¶
- class desispec.pipeline.tasks.psfnight.TaskPSFNight[source]¶
Class containing the properties of one PSF combined night task.
- _option_dict(name, opts)[source]¶
Build the full list of options.
This includes appending the filenames and incorporating runtime options.
desispec.pipeline.tasks.qadata¶
- class desispec.pipeline.tasks.qadata.TaskQAData[source]¶
Class containing the properties of a sky fit task.
desispec.pipeline.tasks.rawdata¶
- class desispec.pipeline.tasks.rawdata.TaskRawdata[source]¶
Class containing the properties of one rawdata.
Since rawdatas have no dependencies and are not created by the pipeline, this class is just used to specify names, etc.
desispec.pipeline.tasks.redshift¶
- class desispec.pipeline.tasks.redshift.TaskRedshift[source]¶
Class containing the properties of one spectra task.
- _option_list(name, opts)[source]¶
Build the full list of options.
This includes appending the filenames and incorporating runtime options.
desispec.pipeline.tasks.sky¶
- class desispec.pipeline.tasks.sky.TaskSky[source]¶
Class containing the properties of a sky fit task.
desispec.pipeline.tasks.spectra¶
- class desispec.pipeline.tasks.spectra.TaskSpectra[source]¶
Class containing the properties of one spectra task.
desispec.pipeline.tasks.starfit¶
- class desispec.pipeline.tasks.starfit.TaskStarFit[source]¶
Class containing the properties of one extraction task.
desispec.pipeline.tasks.traceshift¶
- class desispec.pipeline.tasks.traceshift.TaskTraceShift[source]¶
Class containing the properties of one trace shift task.
desispec.pixflat¶
Routines for pixel flat fielding.
- desispec.pixflat.convolve2d(image, k, weight=None)[source]¶
Return a 2D convolution of image with kernel k, optionally with a weight image
- Parameters:
image – 2D np.array image
k – 2D np.array kernel, each dimension must be odd and greater than 1
weight – 2D np.array of same shape as image
optional – 2D np.array of same shape as image
- Returns:
2D np.array convolved image of same shape as input image
- Return type:
cimage
desispec.pixgroup¶
Tools to regroup spectra in individual exposures by healpix on the sky.
- class desispec.pixgroup.FrameLite(wave, flux, ivar, mask, resolution_data, fibermap, header, scores=None)[source]¶
Lightweight Frame object for regrouping
This is intended for I/O without the overheads of float32 -> float64 conversion, correcting endianness, etc.
- class desispec.pixgroup.SpectraLite(bands, wave, flux, ivar, mask, resolution_data, fibermap, exp_fibermap=None, scores=None)[source]¶
Lightweight spectra I/O object for regrouping
- num_spectra()[source]¶
Get the number of spectra contained in this group.
- Returns (int):
Number of spectra contained in this group.
- num_targets()[source]¶
Get the number of distinct targets.
- Returns (int):
Number of unique targets with spectra in this object.
- desispec.pixgroup.add_missing_frames(frames)[source]¶
Adds any missing frames with ivar=0 FrameLite objects with correct shape to match those that do exist.
- Parameters:
frames – dict of FrameLite objects, keyed by (night,expid,camera)
Modifies frames in-place.
Example: if frames has keys (2020,1,’b0’) and (2020,1,’r0’) but not (2020,1,’z0’), this will add a blank FrameLite object for z0.
The purpose of this is to facilitate frames2spectra, which needs something for every spectro camera for every exposure that is included.
- desispec.pixgroup.fibermap2tilepix(fibermap, nside=64)[source]¶
Maps fibermap to which healpix are covered by which petals
- Parameters:
fibermap – table with columns TARGET_RA, TARGET_DEC, PETAL_LOC
- Options:
nside (int): nested healpix nside (must be power of 2)
Returns dict petalpix[petal] = list(healpix covered by that petal)
- desispec.pixgroup.frames2spectra(frames, pix=None, nside=64)[source]¶
Combine a dict of FrameLite into a SpectraLite for healpix pix
- Parameters:
frames – dict of FrameLight, keyed by (night, expid, camera)
- Options:
pix: only include targets in this NESTED healpix pixel number nside: Healpix nside, must be power of 2
- Returns:
SpectraLite object with subset of spectra from frames that are in the requested healpix pixel pix
- desispec.pixgroup.get_exp2healpix_map(expfile, survey=None, program=None, specprod_dir=None, nights=None, expids=None)[source]¶
Maps exposures to healpixels using preproc/NIGHT/EXPID/tilepix*.json files
- Parameters:
expfile (str) – filepath to production exposure file with EXPOSURES and FRAMES HDUs
- Options:
survey (str): filter by this survey (main, sv3, sv1, …) program (str): filter by this FAPRGRM (dark, bright, backup, other) specprod_dir (str): override $DESI_SPECTRO_REDUX/$SPECPROD nights (array-like): filter to only these nights expids (array-like): filter to only these exposure IDs
Returns table with columns NIGHT EXPID TILEID SURVEY PROGRAM SPECTRO HEALPIX
- desispec.pixgroup.update_frame_cache(frames, framekeys, specprod_dir=None)[source]¶
Update a cache of FrameLite objects to match requested frameskeys
- Parameters:
frames – dict of FrameLite objects, keyed by (night, expid, camera)
framekeys – list of desired (night, expid, camera)
Updates frames in-place
Notes
frames is dictionary, framekeys is list. When finished, the keys of frames match the entries in framekeys
desispec.preproc¶
Preprocess raw DESI exposures.
- desispec.preproc._background(image, header, patch_width=200, stitch_width=10, stitch=False)[source]¶
determine background using a 2D median with square patches of width = width that are interpolated and optionnally try and match the level of amplifiers (does not subtract the background)
- Parameters:
image (ndarray) –
= (( image) –
CCDSEC (header is used to read) –
- Options:
patch_width (integer) size in pixels of the median square patches stitch_width (integer) width in pixels of amplifier edges to match level of amplifiers stitch : do match level of amplifiers
Returns background image with same shape as input image
- desispec.preproc._clipped_std_bias(nsigma)[source]¶
Returns the bias on the standard deviation of a sigma-clipped dataset
Divide by the returned bias to get a corrected value:
a = nsigma bias = sqrt((integrate x^2 exp(-x^2/2), x=-a..a) / (integrate exp(-x^2/2), x=-a..a)) = sqrt(1 - 2a exp(-a^2/2) / (sqrt(2pi) erf(a/sqrt(2))))
- desispec.preproc._global_background(image, patch_width=200)[source]¶
determine background using a 2D median with square patches of width = width that are interpolated (does not subtract the background)
- Parameters:
image (ndarray) –
= (( image) –
- Options:
patch_width (integer) size in pixels of the median square patches
Returns background image with same shape as input image
- desispec.preproc._savgol_clipped(data, window=15, polyorder=5, niter=0, threshold=3.0)[source]¶
Simple method to iteratively do a SavGol filter with rejection and replacing rejected pixels by nearest neighbors
- Parameters:
Returns:
- desispec.preproc.calc_overscan(pix, nsigma=5, niter=3)[source]¶
Calculates overscan, readnoise from overscan image pixels
- Parameters:
pix (ndarray) – overscan pixels from CCD image
- Optional:
nsigma (float) : number of standard deviations for sigma clipping niter (int) : number of iterative refits
- Returns:
Mean, sigma-clipped value readnoise (float):
- Return type:
overscan (float)
- desispec.preproc.compute_background_between_fiber_blocks(image, xyset)[source]¶
Computes CCD background between blocks of fibers
- Parameters:
image – desispec.image.Image object
xyset – desispec.xytraceset.XYTraceSet object
- Returns (model, qadict):
model: np.array of same shape as image.pix qadict: dictionary of keywords for QA with min/max per amplifier
Notes
Has hardcoded number of blocks and fibers and typical spacing between blocked tuned to DESI spectrographs.
- desispec.preproc.compute_overscan_step(overscan_col, median_size=7, edge_margin=50)[source]¶
Compute the overscan step score ‘OSTEP’ from an array of overscan values averaged per CCD row
- Parameters:
overscan_col – 1D numpy.array
- Options:
median_size (int): window size for median pre-filter of overscan_col edge_margin (int): ignore this number of rows at the CCD edges
- Returns:
OSTEP value (float scalar)
- desispec.preproc.find_overscan_cosmic_trails(rawimage, ov_col, overscan_values, col_width=300, threshold=25000.0, smooth=100)[source]¶
Find overscan columns that might be impacted by a trail from bright cosmic
- Parameters:
rawimage – numpy 2D array of raw image
ov_col – tuple(yslice, xslice) from parse_sec_keyword(‘BIASSECx’) defining overscan region
- Options:
col_width: number of pixels from overscan region to consider threshold: ADU threshold for what might cause a problematic trail smooth: median filter smoothing scale
Returns (badrows, active_col_val) where badrows is a boolean array of whether each row is bad or not, and active_col_val is an array of column-summed and row median-filtered from the active region of the CCD next to the overscan region.
- desispec.preproc.get_calibration_image(cfinder, keyword, entry, header=None)[source]¶
Reads a calibration file
- Parameters:
cfinder – None or CalibFinder object
keyword – BIAS, MASK, or PIXFLAT
entry – boolean or filename or image if entry==False return False if entry==True use calibration filename from calib. config and read it if entry==str use this for the filename if entry==image return input
- Options:
header : if not None, update header[‘CAL…’] = calib provenance
- Returns:
2D numpy array with calibration image
For the case of keyword=’BIAS’, check for nightly bias before using default bias in $DESI_SPECTRO_CALIB
- desispec.preproc.get_readout_mode(header)[source]¶
Derive CCD readout mode from CCD header
- Parameters:
header – dict-like FITS header object with BIASSEC keywrds
Returns “4Amp”, “2AmpLeftRight”, or “2AmpUpDown”
“4Amp” means all 4 amps (ABCD) were used for CCD readout; “2AmpLeftRight” means 1 left amp (AC) and 1 right amp (BD) were used; “2AmpUpDown” means 1 upper amp (CD) and one lower (AB) were used.
- desispec.preproc.numba_mean(image_flux, image_ivar, x, hw=3)¶
Returns mean of pixels vs. row about x+-hw
- Parameters:
image_flux – 2D array of CCD image pixels
image_ivar – 2D array of inverse variance of image_flux
x – 1D array of x location per row, len(x) = image_flux.shape[0]
- Options:
hw (int): halfwidth over which to average
Returns (flux, ivar) 1D arrays with weighted mean and inverse variance of pixels[i, int(x-hw):int(x+hw)+1] per row i
- desispec.preproc.parse_sec_keyword(value)[source]¶
parse keywords like BIASSECB=’[7:56,51:4146]’ into python slices
- python and FITS have almost opposite conventions,
FITS 1-indexed vs. python 0-indexed
FITS upperlimit-inclusive vs. python upperlimit-exclusive
FITS[x,y] vs. python[y,x]
i.e. BIASSEC2=’[7:56,51:4146]’ -> (slice(50,4146), slice(6,56))
- desispec.preproc.preproc(rawimage, header, primary_header, bias=True, dark=True, pixflat=True, mask=True, bkgsub_dark=False, nocosmic=False, cosmics_nsig=6, cosmics_cfudge=3.0, cosmics_c2fudge=None, ccd_calibration_filename=None, nocrosstalk=False, nogain=False, overscan_per_row=False, use_overscan_row=False, use_savgol=None, nodarktrail=False, remove_scattered_light=False, psf_filename=None, bias_img=None, model_variance=False, no_traceshift=False, bkgsub_science=False, keep_overscan_cols=False, no_overscan_per_row=False, no_ccd_region_mask=False)[source]¶
preprocess image using metadata in header
image = ((rawimage-bias-overscan)*gain)/pixflat
- Parameters:
rawimage – 2D numpy array directly from raw data file
header – dict-like metadata, e.g. from FITS header, with keywords CAMERA, BIASSECx, DATASECx, CCDSECx where x = A, B, C, D for each of the 4 amplifiers (also supports old naming convention 1, 2, 3, 4).
primary_header – dict-like metadata fit keywords EXPTIME, DOSVER DATE-OBS is also required if bias, pixflat, or mask=True
- Optional bias, pixflat, and mask can each be:
False: don’t apply that step True: use default calibration data for that night ndarray: use that array filename (str or unicode): read HDU 0 and use that
- Optional overscan features:
- overscan_per_rowbool, Subtract the overscan_col values
row by row from the data.
- use_overscan_rowbool, Subtract off the overscan_row
from the data (default: False). Requires ORSEC in the Header
- use_savgolbool, Specify whether to use Savitsky-Golay filter for
the overscan. (default: False). Requires use_overscan_row=True to have any effect.
Optional variance model if model_variance=True Optional background subtraction with median filtering accross the whole CCD if bkgsub_dark=True Optional background subtraction with median filtering between groups of fiber traces if bkgsub_science=True
Optional disabling of cosmic ray rejection if nocosmic=True Optional disabling of dark trail correction if nodarktrail=True
Optional bias image (testing only) may be provided by bias_img=
- Optional tuning of cosmic ray rejection parameters:
cosmics_nsig: number of sigma above background required cosmics_cfudge: number of sigma inconsistent with PSF required cosmics_c2fudge: fudge factor applied to PSF (default is 0.5, set in code)
Optional fit and subtraction of scattered light
Optional disabling of overscan subtraction per row if no_overscan_per_row=True
- Returns Image object with member variables:
pix : 2D preprocessed image in units of electrons per pixel ivar : 2D inverse variance of image mask : 2D mask of image (0=good) readnoise : 2D per-pixel readnoise of image meta : metadata dictionary TODO: define what keywords are included
- preprocessing includes the following steps:
bias image subtraction
overscan subtraction (from BIASSEC* keyword defined regions)
readnoise estimation (from BIASSEC* keyword defined regions)
gain correction (from GAIN* keywords)
pixel flat correction
cosmic ray masking
propagation of input known bad pixel mask
inverse variance estimation
Notes:
The bias image is subtracted before any other calculation to remove any non-uniformities in the overscan regions prior to calculating overscan levels and readnoise.
The readnoise is an image not just one number per amp, because the pixflat image also affects the interpreted readnoise.
The inverse variance is estimated from the readnoise and the image itself, and thus is biased.
- desispec.preproc.read_bias(filename=None, camera=None, dateobs=None)[source]¶
Return calibration bias filename for camera on dateobs or night
- Options:
filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’
Notes
must provide filename, or both camera and dateobs
- desispec.preproc.read_dark(filename=None, camera=None, dateobs=None, exptime=None)[source]¶
Return dark current 2D image ( accounting for exposure time)
- Parameters:
filename – input filename to read
camera – e.g. ‘b0’, ‘r1’, ‘z9’
dateobs – DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’
exptime – the exposure time of the image in seconds
Notes
must provide filename
- desispec.preproc.read_mask(filename=None, camera=None, dateobs=None)[source]¶
Read bad pixel mask image for camera on dateobs.
- Options:
filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’
Notes
must provide filename, or both camera and dateobs
- desispec.preproc.read_pixflat(filename=None, camera=None, dateobs=None)[source]¶
Read calibration pixflat image for camera on dateobs.
- Options:
filename : input filename to read camera : e.g. ‘b0’, ‘r1’, ‘z9’ dateobs : DATE-OBS string, e.g. ‘2018-09-23T08:17:03.988’
Notes
must provide filename, or both camera and dateobs
- desispec.preproc.subtract_peramp_overscan(image, hdr)[source]¶
Subtract per-amp overscan using BIASSEC* keywords
- Parameters:
image – 2D image array, modified in-place
hdr – FITS header with BIASSEC[ABCD] or BIASSEC[1234] keywords
Note: currently used in desispec.ccdcalib.compute_bias_file to model bias image, but not preproc itself (which subtracts that bias, and has more complex support for row-by-row, col-overscan, etc.)
desispec.qa¶
desispec.qa.html¶
Module for generating QA HTML.
- desispec.qa.html.calib(qaprod_dir=None, specprod_dir=None)[source]¶
Generate HTML to orgainze calib HTML
- desispec.qa.html.calib_exp(night, expid, qaprod_dir=None)[source]¶
Geneate HTML for calib exposure PNGs :param night: :param expid:
Returns:
- desispec.qa.html.finish(f, body, links=None)[source]¶
Fill in the HTML file and end it :param f: :type f: file :param body: :type body: str :param links: :type links: str, optional
- desispec.qa.html.make_exposure(night, expid, qaprod_dir=None)[source]¶
Generate HTML for exposure PNGs
- desispec.qa.html.make_exposures(qaprod_dir=None)[source]¶
Generate HTML to organize exposure HTML
- Returns:
links (str)
body (str)
- desispec.qa.html.toplevel(qaprod_dir=None)[source]¶
Generate HTML to top level QA Mainly generates the highest level HTML file which has links to the Exposure and Calib QA.
This also slurps any .png files in the top-level
desispec.qa.qa_brick¶
Classes to organize and execute QA for a DESI exposure.
desispec.qa.qa_exposure¶
Classes to organize and execute QA for a DESI exposure.
desispec.qa.qa_frame¶
Classes to organize and execute QA for a DESI exposure.
- desispec.qa.qa_frame.qaframe_from_frame(frame_file, specprod_dir=None, make_plots=False, qaprod_dir=None, output_dir=None, clobber=True)[source]¶
Generate a qaframe object from an input frame_file name (and night)
Write QA to disk Will also make plots if directed :param frame_file: str :param specprod_dir: str, optional :param qa_dir: str, optional – Location of QA :param make_plots: bool, optional :param output_dir: str, optional
Returns:
desispec.qa.qa_multiexp¶
Class to organize QA for multiple exposures. Likely to only be used as parent of QA_Night or QA_Prod.
desispec.qa.qa_night¶
Class to organize QA for one night of DESI exposures.
desispec.qa.qa_plots¶
Module for QA plots.
- desispec.qa.qa_plots.brick_redrock(outfil, zf, qabrick)[source]¶
QA plots for redrock fits
- Parameters:
outfil –
qabrick –
zf – ZfindBase object
- Returns:
Stuff?
- desispec.qa.qa_plots.exposure_fiberflat(channel, expid, metric, outfile=None)[source]¶
Generate an Exposure level plot of a FiberFlat metric :param channel: str, e.g. ‘b’, ‘r’, ‘z’ :param expid: int :param metric: str, allowed entires are: [‘meanflux’]
Returns:
- desispec.qa.qa_plots.exposure_fluxcalib(outfil, qa_data)[source]¶
QA plots for Flux calibration in an Exposure
- Parameters:
outfil – str – Name of PDF file
qa_data – dict – QA data, including that of the individual frames
- desispec.qa.qa_plots.exposure_map(x, y, metric, mlbl=None, outfile=None, title=None, ax=None, fig=None, psz=9.0, cmap=None, vmnx=None)[source]¶
Generic method used to generated Exposure level QA One channel at a time
- Parameters:
x – list or ndarray
y – list or ndarray
metric – list or ndarray
mlbl – str, optional
outfile – str, optional
title – str, optional
- desispec.qa.qa_plots.exposure_s2n(qa_exp, metric, outfile='exposure_s2n.png', verbose=True, specprod_dir=None)[source]¶
Generate an Exposure level plot of a S/N metric :param qa_exp: QA_Exposure :param metric: str, allowed entires are: [‘resid’] :param specprod_dir: str, optional
Returns:
- desispec.qa.qa_plots.frame_fiberflat(outfil, qaframe, frame, fiberflat)[source]¶
QA plots for fiber flat
- Parameters:
outfil –
qaframe –
frame –
fiberflat –
clobber – bool, optional
- Returns:
Stuff?
- desispec.qa.qa_plots.frame_fluxcalib(outfil, qaframe, frame, fluxcalib)[source]¶
QA plots for Flux calibration in a Frame
- Parameters:
outfil – str, name of output file
qaframe – dict containing QA info
frame – frame object containing extraction of standard stars
fluxcalib – fluxcalib object containing flux calibration
Returns:
- desispec.qa.qa_plots.frame_s2n(s2n_dict, outfile, rescut=0.2, verbose=True)[source]¶
Plot S/N diagnostics for a given frame Replaces a previous-QL script
- desispec.qa.qa_plots.frame_skyres(outfil, frame, skymodel, qaframe, quick_look=False)[source]¶
Generate QA plots and files for sky residuals of a given frame
- Parameters:
outfil (str) – Name of output file
frame (Frame object) –
skymodel (SkyModel object) –
qaframe (QAFrame object) –
- desispec.qa.qa_plots.get_channel_clrs()[source]¶
Simple dict to organize styles for channels :returns: dict :rtype: channel_dict
- desispec.qa.qa_plots.prod_ZP(qa_prod, outfile=None, channels=('b', 'r', 'z'), xaxis='MJD')[source]¶
Generate a plot summarizing the ZP for a production
- Parameters:
qa_prod – QA_Prod object
outfile – str, optional Output file name
channels – tuple, optional List of channels to show
xaxis – str, optional Designate x-axis. Options are: MJD, expid, texp
Returns:
- desispec.qa.qa_plots.prod_avg_s2n(qa_prod, outfile=None, optypes=['ELG'], xaxis='MJD', fiducials=None)[source]¶
Generate a plot summarizing average S/N in a production for a few object types in a few cameras
- Parameters:
qa_prod – QA_Prod object
outfile – str, optional Output file name
optypes – list, optional List of fiducial objects to show Options are: ELG, LRG, QSO
xaxis – str, optional Designate x-axis. Options are: MJD, expid, texp
fiducials –
Returns:
- desispec.qa.qa_plots.prod_channel_hist(qa_prod, qatype, metric, xlim=None, outfile=None, pp=None, close=True)[source]¶
Generate a series of histrograms (one per channel)
- Parameters:
qa_prod – QA_Prod class
qatype – str
metric – str
xlim – tuple, optional
outfile – str, optional
pp – PdfPages, optional
close – bool, optional
Returns:
- desispec.qa.qa_plots.prod_time_series(qa_multi, qatype, metric, outfile=None, close=True, pp=None, bright_dark=0, exposures=False, night=None, horiz_line=None)[source]¶
Generate a time series plot for a production Can be MJD or Exposure number
- Parameters:
qa_multi – QA_Prod or QA_Night
qatype – str
metric – str
outfile – str, optional
close – bool, optional
pp –
bright_dark – int, optional; (flag: 0=all; 1=bright; 2=dark)
night – str, optional Only used for the Title
horiz_line – float, optional Draw a horizontal line at input value
- desispec.qa.qa_plots.show_meta(ax, qaframe, qaflavor, outfil)[source]¶
Show meta data on the figure
- Parameters:
ax – matplotlib.ax
qaframe – QA_Frame
qaflavor – str
Returns:
- desispec.qa.qa_plots.skyline_resid(channel, sky_wave, sky_flux, sky_res, sky_ivar, outfile=None, pp=None, close=True, dpi=700)[source]¶
QA plot for residuals on sky lines ala Julien Guy :param sky_wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close: :param nslices: :param dpi:
Returns:
- desispec.qa.qa_plots.skysub_gauss(sky_wave, sky_flux, sky_res, sky_ivar, outfile=None, pp=None, close=True, binsz=0.1, dpi=700, nfbin=4)[source]¶
Generate a plot examining the Gaussianity of the residuals Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param sky_ivar: :param outfile: :param pp: :param close:
Returns:
- desispec.qa.qa_plots.skysub_resid_dual(sky_wave, sky_flux, sky_res, outfile=None, pp=None, close=True, nslices=20, dpi=700)[source]¶
Generate a plot of sky subtraction residuals Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close:
Returns:
- desispec.qa.qa_plots.skysub_resid_series(sky_dict, xtype, outfile=None, pp=None, close=True, nslices=20, dpi=700)[source]¶
Generate a plot of sky subtraction residuals for a series of inputs Typically for a given channel :param wave: :param sky_flux: :param sky_res: :param outfile: :param pp: :param close:
Returns:
desispec.qa.qa_plots_ql¶
This includes routines to make pdf plots on the qa outputs from quicklook.
For information on QA dictionaries used here as input, visit wiki page: https://desi.lbl.gov/trac/wiki/Pipeline/QuickLook/QuicklookQAOutputs/Science
- desispec.qa.qa_plots_ql.plot_RMS(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot RMS
- Parameters:
qa_dict – dictionary of qa outputs from running qa_quicklook.Get_RMS
outfile – Name of plot output file
- desispec.qa.qa_plots_ql.plot_SNR(qa_dict, outfile, objlist, fitsnr, rescut=0.2, sigmacut=2.0, plotconf=None, hardplots=False)[source]¶
Plot SNR
- Parameters:
qa_dict – dictionary of qa outputs from running qa_quicklook.Calculate_SNR
outfile – output png file
objlist – list of objtype for log(snr**2) vs. mag plots
badfibs – list of fibers with infs or nans to remove for plotting
fitsnr – list of snr vs. mag fitting coefficients # JXP – THIS IS NOT TRUE!!
rescut – only plot residuals (+/-) less than rescut (default 0.2)
sigmacut – only plot residuals (+/-) less than sigma cut (default 2.0)
NOTE – rescut taken as default cut parameter
- desispec.qa.qa_plots_ql.plot_XWSigma(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot XWSigma
- Parameters:
qa_dict – qa dictionary from countpix qa
outfile – file of the plot
- desispec.qa.qa_plots_ql.plot_bias_overscan(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Map of bias from overscan from 4 regions of CCD
- Parameters:
qa_dict – qa dictionary from bias_from_overscan qa
outfile – pdf file of the plot
- desispec.qa.qa_plots_ql.plot_countpix(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot pixel counts above some threshold
- Parameters:
qa_dict – qa dictionary from countpix qa
outfile – pdf file of the plot
- desispec.qa.qa_plots_ql.plot_countspectralbins(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot count spectral bins.
- Parameters:
qa_dict – dictionary of qa outputs from running qa_quicklook.CountSpectralBins
outfile – Name of figure.
- desispec.qa.qa_plots_ql.plot_lpolyhist(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot histogram for each legendre polynomial coefficient in WSIGMA array.
- Parameters:
qa_dict – Dictionary of qa outputs from running qa_quicklook.Check_Resolution
outfile – Name of figure.
- desispec.qa.qa_plots_ql.plot_sky_continuum(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot mean sky continuum from lower and higher wavelength range for each fiber and accross amps.
- Parameters:
qa_dict – dictionary from sky continuum QA
outfile – pdf file to save the plot
- desispec.qa.qa_plots_ql.plot_sky_peaks(qa_dict, outfile, plotconf=None, hardplots=False)[source]¶
Plot rms of sky peaks for smy fibers across amps
- Parameters:
qa_dict – dictionary from sky peaks QA
outfile – pdf file to save the plot
desispec.qa.qa_prod¶
Class to organize QA for a full DESI production run.
desispec.qa.qa_quicklook¶
Monitoring algorithms for Quicklook pipeline.
- class desispec.qa.qa_quicklook.Sky_Residual(name, config, logger=None)[source]¶
Use offline sky_residual function to calculate sky residuals
- desispec.qa.qa_quicklook.get_frame(filetype, night, expid, camera, specdir)[source]¶
Make frame object from file if in development mode
- desispec.qa.qa_quicklook.get_image(filetype, night, expid, camera, specdir)[source]¶
Make image object from file if in development mode
desispec.qa.qalib¶
Simple low level library functions for offline and online QAs.
- desispec.qa.qalib.SN_ratio(flux, ivar)[source]¶
SN Ratio median snr for the spectra, flux should be sky subtracted.
- Parameters:
flux (array) – 2d [nspec,nwave] the signal (typically for spectra, this comes from frame object
ivar (array) – 2d [nspec,nwave] corresponding inverse variance
- Returns:
1d [nspec]
- Return type:
medsnr (array)
- desispec.qa.qalib.SignalVsNoise(frame, params, fidboundary=None)[source]¶
Signal vs. Noise
Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera.
- Parameters:
flux (array) – 2d [nspec,nwave] the signal (typically for spectra, this comes from frame object
ivar (array) – 2d [nspec,nwave] corresponding inverse variance
fidboundary – list of slices indicating where to select in fiber and wavelength directions for each amp (output of slice_fidboundary function)
- desispec.qa.qalib._get_mags(frame)[source]¶
Extract frame.fibermap fluxes into mags depending upon camera
- Parameters:
frame – Frame object
Returns array of magnitudes, using 99.0 when flux<0
b camera frames return g-band magnitudes; r camera -> r-mags; z camera -> z-mags
- desispec.qa.qalib.ampregion(image)[source]¶
Get the pixel boundary regions for amps
- Parameters:
image – desispec.image.Image object
- desispec.qa.qalib.continuum(wave, flux, wmin=None, wmax=None)[source]¶
Find the median continuum of the spectrum inside a wavelength region.
- Parameters:
wave – 1d wavelength array
flux – 1d counts/flux array
wmax (wmin and) – region to consider for the continuum
- desispec.qa.qalib.countbins(flux, threshold=0)[source]¶
Count the number of bins above a given threshold on each fiber
- Parameters:
flux – 2d (nspec,nwave)
threshold – threshold counts
- desispec.qa.qalib.countpix(image, nsig=None)[source]¶
Count the pixels above a given threshold in units of sigma.
- Parameters:
image – 2d image array
nsig – threshold in units of sigma, e.g 2 for 2 sigma
- desispec.qa.qalib.fiducialregion(frame, psf)[source]¶
Get the fiducial amplifier regions on the CCD pixel to fiber by wavelength space
- Parameters:
frame – desispec.frame.Frame object
psf – desispec.psf.PSF like object
- desispec.qa.qalib.getrms(image)[source]¶
Calculate the rms of the pixel values)
- Parameters:
image – 2d array
- desispec.qa.qalib.integrate_spec(wave, flux)[source]¶
Calculate the integral of the spectrum in the given range using trapezoidal integration
Note: limits of integration are min and max values of wavelength
- Parameters:
wave – 1d wavelength array
flux – 1d flux array
- desispec.qa.qalib.orig_SNRFit(frame, night, camera, expid, params, fidboundary=None, offline=False)[source]¶
Signal vs. Noise With fitting
Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera. then fit the log(snr)=a+b*mag or log(snr)=poly(mag)
see http://arXiv.org/abs/0706.1062v2 for proper fitting of power-law distributions it is not implemented here!
- qadict has the following data model
“MAGNITUDES” : ndarray - Depends on camera (DECAM_G, DECAM_R, DECAM_Z) “MEDIAN_SNR” : ndarray (nfiber) “NUM_NEGATIVE_SNR” : int “SNR_MAG_TGT” “FITCOEFF_TGT” : list “SNR_RESID” : list, can be trimmed down during the fitting “FIDSNR_TGT” “RA” : ndarray (nfiber) “DEC” : ndarray (nfiber) “OBJLIST” : list - Save a copy to make sense of the list order later “EXPTIME” : float “FIT_FILTER” : str “r2” : float - Fitting parameter
- Parameters:
frame – desispec.Frame object
night –
camera –
expid – int
params – parameters dictionary
{ – “Func”: “linear”, # Fit function type one of [“linear”,”poly”,”astro”] “FIDMAG”: 22.0, # magnitude to evaluate the fit “Filter”:”DECAM_R”, #filter name
} –
fidboundary – list of slices indicating where to select in fiber and wavelength directions for each amp (output of slice_fidboundary function)
offline – bool, optional If True, save things differently for offline
- Returns:
dict
- Return type:
qadict
- desispec.qa.qalib.s2n_flux_astro(flux, A, B)[source]¶
Function for a normalized (by texp**1/2) curve to flux vs S/N
- desispec.qa.qalib.s2n_funcs(exptime=None)[source]¶
Functions for fitting S/N
- Parameters:
exptime – float, optional
- Returns:
dict
- Return type:
funcMap
- desispec.qa.qalib.s2nfit(frame, camera, params)[source]¶
Signal vs. Noise With fitting
Take flux and inverse variance arrays and calculate S/N for individual targets (ELG, LRG, QSO, STD) and for each amplifier of the camera. then fit snr=A*mag/sqrt(A*mag+B)
see http://arXiv.org/abs/0706.1062v2 for proper fitting of power-law distributions it is not implemented here!
Instead we use scipy.optimize.curve_fit
- Parameters:
frame – desispec.Frame object
camera – str, name of the camera
params – parameters dictionary for S/N
- Returns:
- dict
MEDIAN_SNR (ndarray, nfiber): Median S/N of light in each fiber FIT_FILTER (str): Filter used for the fluxes EXPTIME (float): Exposure time XXX_FIBERID (list): Fibers matching ELG, LRG, BGS, etc. SNR_MAG_TGT (list): List of lists with S/N and mag of ELG, LRG, BGS, etc. FITCOEFF_TGT (list): List of fitted coefficients. Junk fits have np.nan OBJLIST (list): List of object types analyzed (1 or more fiber)
- Return type:
qadict
- desispec.qa.qalib.sky_continuum(frame, wrange1, wrange2)[source]¶
QA Algorithm for sky continuum.
To be called from desispec.sky.qa_skysub and desispec.qa.qa_quicklook.Sky_Continuum.run_qa
- Parameters:
frame –
wrange1 –
wrange2 –
- Returns:
skyfiber, contfiberlow, contfiberhigh, meancontfiber, skycont
- desispec.qa.qalib.sky_resid(param, frame, skymodel, quick_look=False)[source]¶
QA Algorithm for sky residual To be called from desispec.sky.qa_skysub and desispec.qa.qa_quicklook.Sky_residual.run_qa :param param: dict of QA parameters :param frame: desispec.Frame object after sky subtraction :param skymodel: desispec.SkyModel object
Returns a qa dictionary for sky resid
- desispec.qa.qalib.slice_fidboundary(frame, leftmax, rightmin, bottommax, topmin)[source]¶
leftmax,rightmin,bottommax,topmin - Indices in spec-wavelength space for different amps (e.g output from fiducialregion function) #- This could be merged to fiducialregion function
- Returns (list):
list of tuples of slices for spec- wavelength boundary for the amps.
desispec.qa.utils¶
Module for QA support.
- desispec.qa.utils.get_skyres(cframes, sub_sky=False, flatten=True)[source]¶
- Parameters:
cframes – str or list Single cframe or a list of them
sub_sky – bool, optional Subtract the sky? This should probably not be done
flatten – bool, optional Return a flat, 1D array for each variable
combine – bool, optional combine the individual sky fibers? Median ‘smash’
- Returns:
ndarray flux : ndarray res : ndarray ivar : ndarray
- Return type:
wave
desispec.qproc¶
desispec.qproc.flavor¶
- desispec.qproc.flavor.check_qframe_flavor(qframe, input_flavor=None)[source]¶
Tool to check the flavor of a qframe
- Parameters:
qframe – DESI QFrame object
- Optional:
input_flavor
- Returns:
flavor string
desispec.qproc.io¶
I/O routines for qproc objects
- desispec.qproc.io.read_qframe(filename, nspec=None, skip_resolution=False)[source]¶
Reads a frame fits file and returns its data.
- Parameters:
filename – path to a file
skip_resolution – bool, option Speed up read time (>5x) by avoiding the Resolution matrix
- Returns:
desispec.Frame object with attributes wave, flux, ivar, etc.
- desispec.qproc.io.write_qframe(outfile, qframe, header=None, fibermap=None, units=None)[source]¶
Write a frame fits file and returns path to file written.
- Parameters:
outfile – full path to output file, or tuple (night, expid, channel)
qframe – desispec.qproc.QFrame object with wave, flux, ivar…
- Optional:
header: astropy.io.fits.Header or dict to override frame.header fibermap: table to store as FIBERMAP HDU
- Returns:
full filepath of output file that was written
Note
to create a QFrame object to pass into write_qframe, qframe = QFrame(wave, flux, ivar)
desispec.qproc.qarc¶
- desispec.qproc.qarc.process_arc(qframe, xytraceset, linelist=None, npoly=2, nbins=2)[source]¶
qframe: desispec.qframe.QFrame object xytraceset : desispec.xytraceset.XYTraceSet object linelist: line list to fit npoly: polynomial order for sigma expansion nbins: no of bins for the half of the fitting window return: xytraceset (with ysig vs wave)
- desispec.qproc.qarc.sigmas_from_arc(wave, flux, ivar, linelist, n=2)[source]¶
Gaussian fitting of listed arc lines and return corresponding sigmas in pixel units Args: linelist: list of lines (A) for which fit is to be done n: fit region half width (in bin units): n=2 bins => (2*n+1)=5 bins fitting window.
desispec.qproc.qextract¶
- desispec.qproc.qextract.qproc_boxcar_extraction(xytraceset, image, fibers=None, width=7, fibermap=None, save_sigma=True)[source]¶
Fast boxcar extraction of spectra from a preprocessed image and a trace set
- Parameters:
xytraceset – DESI XYTraceSet object
image – DESI preprocessed Image object
- Optional:
fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : extraction boxcar width, default is 7 fibermap : table
- Returns:
QFrame object
desispec.qproc.qfiberflat¶
- desispec.qproc.qfiberflat.qproc_apply_fiberflat(qframe, fiberflat, return_flat=False)[source]¶
Apply a fiber flat to a qframe.
- Inputs:
qframe: desispec.qproc.qframe.QFrame object which will be modified fiberflat: desispec.fiberflat.FiberFlat object with the flat to apply
- Optional:
return_flat : if True, returns the flat field that has been applied
Returns nothing or the flat that has been applied.
- desispec.qproc.qfiberflat.qproc_compute_fiberflat(qframe, niter_meanspec=4, nsig_clipping=3.0, spline_res_clipping=20.0, spline_res_flat=5.0)[source]¶
Fast estimation of fiberflat
desispec.qproc.qframe¶
Lightweight wrapper class for spectra extract with qextract, row-by-row extraction (boxcar or profile)
desispec.qproc.qsky¶
- desispec.qproc.qsky.qproc_sky_subtraction(qframe, return_skymodel=False)[source]¶
Fast sky subtraction directly applied to the input qframe.
- Args:
qframe : DESI QFrame object
- Optional:
return_skymodel returns the skymodel as an array of same shape as qframe.flux
desispec.qproc.rowbyrowextract¶
Row-by-row extraction routines (Horne 1986).
- desispec.qproc.rowbyrowextract.build_atcinva(atcinva, xstart, xend, profiles, ivar)¶
Build A^T C^-1 A for the row-by-row extraction.
Modifies input atcinva in place. Assumes spectra are ordered, so xstart[i+1] > xstart[i].
- desispec.qproc.rowbyrowextract.build_atcinvb(atcinvb, xstart, profiles, data)¶
Build A^T C^-1 b for the row-by-row extraction.
Modifies input atcinvb in place.
- Parameters:
atcinvb (np.ndarray) – matrix to fill in (modified in place)
xstart (int) – x coordinate of start of each profile
profiles (np.ndarray) – cross-dispersion profile for each trace
data (np.ndarray) – flux * ivar for appropriate row of image
- desispec.qproc.rowbyrowextract.extract(image, psf, blocksize=25, fibermap=None, nspec=500, return_model=False, tails=False)[source]¶
Extract spectra from image using an optimal row-by-row extraction.
The algorithm extracts the spectra row-by-row, finding the fluxes associated with the 1D PSF that best explain the data.
This only works for Gauss-Hermite PSFs at present, since it does some trickery to efficiently get 1D PSFs for large groups of wavelengths and spectra.
- Parameters:
image (Image) – image from which to extract the spectra.
psf (specter.psf.gausshermite.GaussHermitePSF) – PSF to use
blocksize (int) – number of spectra to fit simultaneously. Spectra in different blocks will be fit independently.
fibermap (astropy.Table) – fibermap for image
nspec (int) – total number of spectra to extract
return_model (bool) – if True, return also the model of the image, the profiles, and the locations in the image corresponding to the profiles.
tails (bool) – include PSF tails in the modeling. These are usually identically zero for DESI and so default to off.
- Returns:
frame – The 1D extracted spectra as a QFrame
- Return type:
desispec.qproc.qframe.QFrame
- desispec.qproc.rowbyrowextract.model(frame, profile, profilepix, shape)[source]¶
Create model image for frame.
- desispec.qproc.rowbyrowextract.onedprofile_gaussian(psf, ypix, ispec, nsig=4)[source]¶
Compute the cross-dispersion profile for a Gaussian.
This was warm-up code for the Gauss-Hermite implementation below, but may have independent value? It will work with any specter PSF, albeit using a Gaussian approximation to the cross-dispersion profile.
- Parameters:
psf (specter.psf.PSF) – psf
ypix (np.ndarray) – rows for which to compute profile
ispec (np.ndarray) – indices of spectra for which to compute profiles
nsig (np.ndarray) – how many sigma out to compute the Gaussian; profile is treated as zero beyond roughly here.
- Returns:
profilepix, profile, lambda
profilepix (np.ndarray) – the x-coordinates corresponding to profile on the detector
profile (np.ndarray) – the cross dispersion profiles
lambda (np.ndarray) – the wavelengths of the spectra at the requested locations
- desispec.qproc.rowbyrowextract.onedprofile_gh(psf, ypix, ispec, tails=True)[source]¶
Compute cross-dispersion profile for Gauss-Hermite PSFs
- Parameters:
psf (specter.psf.gausshermite.GaussHermitePSF) – psf
ypix (np.ndarray) – rows for which to compute profile
ispec (np.ndarray) – indices of spectra for which to compute profiles
- Returns:
profilepix, profile, lambda
profilepix (np.ndarray) – the x-coordinates corresponding to profile on the detector
profile (np.ndarray) – the cross dispersion profiles
lambda (np.ndarray) – the wavelengths of the spectra at the requested locations
- desispec.qproc.rowbyrowextract.pgh(x, m=0, xc=0.0, sigma=1.0)¶
Pixel-integrated (probabilist) Gauss-Hermite function. :param x: pixel-center baseline array :param m: order of Hermite polynomial multiplying Gaussian core :param xc: sub-pixel position of Gaussian centroid relative to x baseline :param sigma: sigma parameter of Gaussian core in units of pixels
Uses the relationship Integral{ H_k(x) exp(-0.5 x^2) dx} = -H_{k-1}(x) exp(-0.5 x^2) + const Written: Adam S. Bolton, U. of Utah, fall 2010 Adapted for efficiency by S. Bailey while dropping generality modified from the orig _pgh to enable jit-compiling –> no longer passing in self, calling custom numba functions in util E. Schlafly added some broadcasting bits to make it play more nicely with large arrays. This should probably go to specter instead.
- desispec.qproc.rowbyrowextract.tail1d(core, xx, index)[source]¶
Compute the cross-dispersion profile of the tail model.
- Parameters:
core (np.ndarray) – core scale (pixels)
xx (np.ndarray) – distance in x from center
index (np.ndarray) – power law index of tail model
- Returns:
np.ndarray
tail amplitude at given location
desispec.quicklook¶
desispec.quicklook.arcprocess¶
- desispec.quicklook.arcprocess.process_arc(frame, linelist=None, npoly=2, nbins=2, domain=None)[source]¶
frame: desispec.frame.Frame object, preumably resolution not evaluated. linelist: line list to fit npoly: polynomial order for sigma expansion nbins: no of bins for the half of the fitting window return: coefficients of the polynomial expansion
- desispec.quicklook.arcprocess.sigmas_from_arc(wave, flux, ivar, linelist, n=2)[source]¶
Gaussian fitting of listed arc lines and return corresponding sigmas in pixel units Args: linelist: list of lines (A) for which fit is to be done n: fit region half width (in bin units): n=2 bins => (2*n+1)=5 bins fitting window.
- desispec.quicklook.arcprocess.write_psffile(infile, wcoeffs, wcoeffs_wavemin, wcoeffs_wavemax, outfile, wavestepsize=None)[source]¶
extract psf file, add wcoeffs, and make a new psf file preserving the traces etc. psf module will load this
desispec.quicklook.merger¶
A class to merge quicklook qa outputs.
desispec.quicklook.palib¶
Low level functions to be from top level PAs.
- desispec.quicklook.palib.apply_flux_calibration(frame, fluxcalib)[source]¶
Apply flux calibration to sky subtracted qframe Use offline algorithm, but assume qframe object is input and that it is on native ccd wavelength grid Calibration vector is resampled to frame wavelength grid
frame: QFrame object fluxcalib: FluxCalib object
Modifies frame.flux and frame.ivar
- desispec.quicklook.palib.get_resolution(wave, nspec, tset, usesigma=False)[source]¶
Calculates approximate resolution values at given wavelengths in the format that can directly feed resolution data of desispec.frame.Frame object.
wave: wavelength array nsepc: no of spectra (int) tset: desispec.xytraceset like object usesigma: allows to use sigma from psf file for resolution computation.
returns : resolution data (nspec,nband,nwave); nband = 1 for usesigma = False, otherwise nband=21
- desispec.quicklook.palib.project(x1, x2)[source]¶
return a projection matrix so that arrays are related by linear interpolation x1: Array with one binning x2: new binning
Return Pr: x1= Pr.dot(x2) in the overlap region
- desispec.quicklook.palib.resample_spec(wave, flux, outwave, ivar=None)[source]¶
rebinning conserving S/N Algorithm is based on http://www.ast.cam.ac.uk/%7Erfc/vpfit10.2.pdf Appendix: B.1
Args: wave : original wavelength array (expected (but not limited) to be native CCD pixel wavelength grid outwave: new wavelength array: expected (but not limited) to be uniform binning flux : df/dx (Flux per A) sampled at x ivar : ivar in original binning. If not None, ivar in new binning is returned.
Note: Full resolution computation for resampling is expensive for quicklook.
desispec.interpolation.resample_flux using weights by ivar does not conserve total S/N. Tests with arc lines show much narrow spectral profile, thus not giving realistic psf resolutions This algorithm gives the same resolution as obtained for native CCD binning, i.e, resampling has insignificant effect. Details,plots in the arc processing note.
desispec.quicklook.pas¶
- class desispec.quicklook.pas.PipelineAlg(name, inptype, outtype, config, logger=None)[source]¶
Simple base class for Pipeline algorithms
desispec.quicklook.procalgs¶
Pipeline Preprocessing algorithms for Quicklook.
- class desispec.quicklook.procalgs.ApplyFiberFlat(name, config, logger=None)[source]¶
PA to Apply the fiberflat field to the given frame
- class desispec.quicklook.procalgs.ApplyFiberFlat_QL(name, config, logger=None)[source]¶
PA to Apply the fiberflat field (QL) to the given frame
- class desispec.quicklook.procalgs.ApplyFiberFlat_QP(name, config, logger=None)[source]¶
PA to Apply the fiberflat field (QP) to the given qframe
- class desispec.quicklook.procalgs.ApplyFluxCalibration(name, config, logger=None)[source]¶
PA to apply flux calibration to the given sframe
- class desispec.quicklook.procalgs.BoxcarExtract(name, config, logger=None)[source]¶
- do_boxcar(tset, outwave, boxwidth=2.5, nspec=500, maskFile=None, usesigma=False, quick_resolution=False)¶
Extracts spectra row by row, given the centroids
- Parameters:
image – desispec.image object
tset – desispec.xytraceset like object
outwave – wavelength array for the final spectra output
boxwidth – HW box size in pixels
usesigma – if True, use sigma from psf file (ysigma) to calculate resolution data.
quick_resolution – whether to calculate the resolution matrix or use QuickResolution object
Returns flux, ivar, resolution
- class desispec.quicklook.procalgs.ComputeFiberflat(name, config, logger=None)[source]¶
PA to compute fiberflat field correction from a DESI continuum lamp frame
- class desispec.quicklook.procalgs.ComputeFiberflat_QL(name, config, logger=None)[source]¶
PA to compute fiberflat field correction from a DESI continuum lamp frame
- class desispec.quicklook.procalgs.ComputeSky(name, config, logger=None)[source]¶
PA to compute sky model from a DESI frame
- class desispec.quicklook.procalgs.ComputeSky_QL(name, config, logger=None)[source]¶
PA to compute sky model from a DESI frame
- class desispec.quicklook.procalgs.Extraction_2d(name, config, logger=None)[source]¶
Offline 2D extraction for offline QuickLook
- class desispec.quicklook.procalgs.Flexure(name, config, logger=None)[source]¶
Use desi_compute_trace_shifts to output modified psf file
- class desispec.quicklook.procalgs.Initialize(name, config, logger=None)[source]¶
This PA takes information from the fibermap and raw header and adds it to the general info section of the merged dictionary
- class desispec.quicklook.procalgs.ResolutionFit(name, config, logger=None)[source]¶
Fitting of Arc lines on extracted arc spectra, polynomial expansion of the fitted sigmas, and updating the coefficients to the new traceset file
- class desispec.quicklook.procalgs.SkySub_QL(name, config, logger=None)[source]¶
This is for QL Sky subtraction. The input frame object should be fiber flat corrected. Unlike offline, if no skymodel file is given as input, a sky compute method is called to create a skymodel object and then subtraction is performed. Outputing that skymodel to a file is optional and can be configured.
- class desispec.quicklook.procalgs.SkySub_QP(name, config, logger=None)[source]¶
Sky subtraction. The input frame object should be fiber flat corrected. No sky model is saved for now
desispec.quicklook.qas¶
- class desispec.quicklook.qas.MonitoringAlg(name, inptype, config, logger=None)[source]¶
Simple base class for monitoring algorithms
desispec.quicklook.ql_plotlib¶
Generic plotting algorithms for QuickLook QAs.
- desispec.quicklook.ql_plotlib.ql_2dplot(ax, xvals, yvals, plottitle, xtitle, ytitle, xlim=None, ylim=None)[source]¶
Make 2d plot of specific metrics provided in configuration file
- Parameters:
ax – matplotlib subplot
xvals – QA metric to be plotted along the xaxis
yvals – QA metric to be plotted along the yaxis
plottitle – plot title from configuration file
xtitle – x axis label
ytitle – y axis label
- Optional:
xlim: list containing x range (i.e. [x_lo,x_hi]) ylim: list containing y range (i.e. [y_lo,y_hi])
- Returns:
matplotlib sublot containing plotted metrics
- desispec.quicklook.ql_plotlib.ql_3dplot(ax, xvals, yvals, zvals, plottitle, xtitle, ytitle, zlim=None, heatmap=None)[source]¶
Make 3d scatter plot of specific metrics provided in configuration file
- Parameters:
ax – matplotlib subplot
xvals – QA metric to be plotted along the xaxis
yvals – QA metric to be plotted along the yaxis
zvals – QA metric to be plotted
plottitle – plot title from configuration file
xtitle – x axis label
ytitle – y axis label
- Optional:
zlim: list containing scatter plot range (i.e. [z_lo,z_hi])
- Returns:
matplotlib sublot containing plotted metrics
- desispec.quicklook.ql_plotlib.ql_patchplot(ax, vals, plottitle, grid, heatmap=None)[source]¶
Make patch plot of specific metrics provided in configuration file
- Parameters:
ax – matplotlib subplot
vals – QA metric to be plotted
plottitle – plot title from configuration file
grid – shape of patch plot
- Optional:
heat: specify color of heatmap (must conform to matplotlib)
- Returns:
matplotlib sublot containing plotted metrics
- desispec.quicklook.ql_plotlib.ql_qaplot(fig, plotconf, qadict, camera, expid, outfile)[source]¶
Get plotting configuration info and setup plots
- Parameters:
fig – matplotlib figure
plotconf – list of config info for each plot
qadict – QA metrics dictionary
camera – to be used in output png title
expid – to be used in output png title
outfile – output png file
- Returns:
png file containing all desired plots
desispec.quicklook.qlboxcar¶
Boxcar extraction for Spectra from Desi Image.
- desispec.quicklook.qlboxcar.do_boxcar(image, tset, outwave, boxwidth=2.5, nspec=500, maskFile=None, usesigma=False, quick_resolution=False)[source]¶
Extracts spectra row by row, given the centroids
- Parameters:
image – desispec.image object
tset – desispec.xytraceset like object
outwave – wavelength array for the final spectra output
boxwidth – HW box size in pixels
usesigma – if True, use sigma from psf file (ysigma) to calculate resolution data.
quick_resolution – whether to calculate the resolution matrix or use QuickResolution object
Returns flux, ivar, resolution
desispec.quicklook.qlconfig¶
- class desispec.quicklook.qlconfig.Config(configfile, night, camera, expid, singqa, amps=True, rawdata_dir=None, specprod_dir=None, outdir=None, qlf=False, psfid=None, flatid=None, templateid=None, templatenight=None, qlplots=False, store_res=None)[source]¶
A class to generate Quicklook configurations for a given desi exposure. expand_config will expand out to full format as needed by quicklook.setup
- dump_pa(paname)[source]¶
dump the PA outputs to respective files. This has to be updated for fframe and sframe files as QL anticipates for dumpintermediate case.
- dump_qa()[source]¶
yaml outputfile for the set of qas for a given pa Name and default locations of files are handled by desispec.io.meta.findfile
- property paargs¶
Many arguments for the PAs are taken default. Some of these may need to be variable psfspfile is for offline extraction case
- property palist¶
palist for this config see :class: Palist for details.
- property qalist¶
qalist for the given palist
- class desispec.quicklook.qlconfig.Palist(thislist=None, algorithms=None)[source]¶
Generate PA list and QA list for the Quicklook Pipeline for the given exposure
- desispec.quicklook.qlconfig.check_config(outconfig, singqa)[source]¶
Given the expanded config, check for all possible file existence etc….
desispec.quicklook.qlexceptions¶
Exception classes for Quicklook.
desispec.quicklook.qlheartbeat¶
desispec.quicklook.qllogger¶
- class desispec.quicklook.qllogger.QLLogger(name=None, loglevel=20)[source]¶
Simple logger class using logging
desispec.quicklook.qlpsf¶
Given a psf output file e.g. output from bootcalib.write_psf or desimodel/data/specpsf/PSF files this defines an interface that other codes can use the trace and wavelength solutions.
Mostly making parallel to specter.psf.PSF baseclass and inheriting as needed, but only xtrace, ytrace and wavelength solution available for this case. No resolution information yet.
- class desispec.quicklook.qlpsf.PSF(filename)[source]¶
Base class for 2D psf
- angstroms_per_pixel(ispec, wavelength)[source]¶
Return CCD pixel width in Angstroms for spectrum ispec at given wavlength(s). Wavelength may be scalar or array.
desispec.quicklook.qlresolution¶
Quicklook version of resolution object that can calculate resolution efficiently from psf information
Author: Sami Kama
- class desispec.quicklook.qlresolution.QuickResolution(*args: Any, **kwargs: Any)[source]¶
Quicklook version of the resolution mimicking desispec.resolution.Resolution with some reduction in dimentionality. Contains code from Resolution implementation Note that this is similar to desispec.resolution.Resolution, though faster and differing in implementation details that should be cross checked before merging these or replacing one with the other
desispec.quicklook.quickfiberflat¶
Here will be the fiberflat routines specific to quicklook.
Dhungana, 2016
- desispec.quicklook.quickfiberflat.apply_fiberflat(frame, fiberflat)[source]¶
- Args: frame: desispec.frame.Frame object
fiberflat: desispec.fiberflat.Fiberflat object
- desispec.quicklook.quickfiberflat.compute_fiberflat()[source]¶
computes fiberflat: A boss like algorithm writing in progress and will fit in here.
Args:
desispec.quicklook.quicklook¶
- desispec.quicklook.quicklook.get_chan_spec_exp(inpname, camera=None)[source]¶
Get channel, spectrograph and expid from the filename itself
- Parameters:
inpname – can be raw or pix, or frame etc filename
camera – is required for raw case, eg, r0, b5, z8 irrelevant for others
- desispec.quicklook.quicklook.mapkeywords(kw, kwmap)[source]¶
Maps the keyword in the configuration to the corresponding object returned by the desispec.io module. e.g Bias Image file is mapped to biasimage object… for the same keyword “BiasImage”
- desispec.quicklook.quicklook.runpipeline(pl, convdict, conf)[source]¶
Runs the quicklook pipeline as configured
- Parameters:
pl – is a list of [pa,qas] where pa is a pipeline step and qas the corresponding qas for that pa
convdict – converted dictionary e.g : conf[“IMAGE”] is the real psf file but convdict[“IMAGE”] is like desispec.image.Image object and so on. details in setup_pipeline method below for examples.
conf – a configured dictionary, read from the configuration yaml file. e.g: conf=configdict=yaml.safe_load(open(‘configfile.yaml’,’rb’))
- desispec.quicklook.quicklook.setup_pipeline(config)[source]¶
Given a configuration from QLF, this sets up a pipeline [pa,qa] and also returns a conversion dictionary from the configuration dictionary so that Pipeline steps (PA) can take them. This is required for runpipeline.
desispec.quicklook.quicksky¶
Here will be the sky computing and sky subtraction routines for QL.
- desispec.quicklook.quicksky.compute_sky(fframe, fibermap=None, nsig_clipping=4.0, apply_resolution=False)[source]¶
Adding in the offline algorithm here to be able to apply resolution for sky compute. We will update this here as needed for quicklook. The original weighted sky compute still is the default.
- Args: fframe: fiberflat fielded frame object
fibermap: fibermap object apply_resolution: if True, uses the resolution in the frame object to evaluate sky allowing fiber to fiber variation of resolution.
- desispec.quicklook.quicksky.subtract_sky(fframe, skymodel)[source]¶
skymodel: skymodel object. fframe: frame object to do the sky subtraction, should be already fiber flat fielded need same number of fibers and same wavelength grid
desispec.resolution¶
Standardized handling of sparse wavelength resolution matrices.
Use python -m desispec.resolution
to run unit tests.
- class desispec.resolution.Resolution(*args: Any, **kwargs: Any)[source]¶
Canonical representation of a resolution matrix.
Inherits all of the method of scipy.sparse.dia_matrix, including todense() for converting to a dense 2D numpy array of matrix elements, most of which will be zero (so you generally want to avoid this).
- Parameters:
data – Must be in one of the following formats listed below.
- Options:
- offsets: list of diagonals that the data represents. Only used if
data is a 2D dense array.
- Raises:
ValueError – Invalid input for initializing a sparse resolution matrix.
Data formats:
a scipy.sparse matrix in DIA format with the required diagonals (but not necessarily in the canoncial order);
a 2D square numpy arrray (i.e., a dense matrix) whose non-zero values beyond default_ndiag will be silently dropped; or
a 2D numpy array[ndiag, nwave] that encodes the sparse diagonal values in the same format as scipy.sparse.dia_matrix.data .
The last format is the one used to store resolution matrices in FITS files.
- to_fits_array()[source]¶
Convert to an array of sparse diagonal values.
This is the format used to store resolution matrices in FITS files. Note that some values in the returned rectangular array do not correspond to actual matrix elements since the diagonals get smaller as you move away from the central diagonal. As long as you treat this array as an opaque representation for FITS I/O, you don’t care about this. To actually use the matrix, create a Resolution object from the fits array first.
- Returns:
- An array of (num_diagonals,nbins) sparse matrix
element values close to the diagonal.
- Return type:
- desispec.resolution._gauss_pix(x, mean=0.0, sigma=1.0)[source]¶
Utility function to integrate Gaussian density within pixels
- Parameters:
- Returns:
array of integals of the Gaussian density in the pixels.
Note
All pixels must be the same size
- desispec.resolution._sort_and_symmeterize(data, offsets)[source]¶
Sort data,offsets and pad to ensure equal number of upper/lower diagonals
- Parameters:
data – 2D array of diagonals, following scipy.sparse.dia_matrix.data ordering
offsets – 1D array of offsets; must be complete from min to max but doesn’t have to be sorted
- Returns:
fulldata, fulloffsets
desispec.scatteredlight¶
Try to model and remove the scattered light.
- desispec.scatteredlight.model_scattered_light(image, xyset)[source]¶
Model the scattered light in a preprocessed image. The method consist in convolving the “direct” light image (image * mask along spectral traces) and calibrating this convolved image using the data between the fiber bundles.
- Parameters:
image – desispec.image.Image object
xyset – desispec.xytraceset.XYTraceSet object
- Returns:
np.array of same shape as image.pix
- Return type:
model
desispec.scripts¶
Main functions and commandline parsing.
Utility script to remove archive tiles for the merged target list (MTL).
- desispec.scripts.archive_tilenight.archivetile(tiledir, archivedir, badpetals=[], dryrun=False)[source]¶
Archive tiledir to archivedir, leaving symlink behind
- Parameters:
tiledir – full path to tiles/cumulative/TILEID/LASTNIGHT
archivedir – full path to tiles/archive/TILEID/ARCHIVEDATE
- Options:
dryrun: if True, print messages but don’t move directories.
Returns int error code, non-zero if there was a problem
- desispec.scripts.archive_tilenight.check_missing_zmtl(tiledir, archivetileroot)[source]¶
Check for zmtl file that exist in previous archives but not latest prod
- Parameters:
tiledir – full path to tiles/cumulative/TILEID/LASTNIGHT
archivetileroot – full path to tiles/archive/TILEID
Returns: dict missing[petal] /path/to/zmtl/in/archive/not/in/prod
- desispec.scripts.archive_tilenight.create_badpetal_zmtl(inzmtl, outzmtl)[source]¶
Using inzmtl as template, create outzmtl with ZWARN BAD_PETALQA set
- Parameters:
inzmtl – full path to input zmtl file
outzmtl – full path to output zmtl file
- desispec.scripts.archive_tilenight.freezedir(path, dryrun=False, force=False)[source]¶
Remove write permission from path unless dryrun
- Parameters:
path (str) – path to directory to remove
- Options:
dryrun: if True, print info but don’t actually remove write access force: if True, rerun chmod even if directory already appears frozen
Returns non-zero error code upon failure (not an exception)
Note: dryrun overrides force
- desispec.scripts.archive_tilenight.move_and_link_directory(src, dst)[source]¶
Move src to dst, and create link from src -> dst. If src is already a link, recursively copy instead.
- Parameters:
src – full path to a source directory
dst – full path to a destination directory that doesn’t yet exist
After this is run:
dst will have the contents originally in src
src will be a link to dst
desispec.scripts.assemble_fibermap¶
desispec.scripts.autocalib_fiberflat¶
desispec.scripts.average_fiberflat¶
desispec.scripts.badcolumn_mask¶
desispec.scripts.bootcalib¶
Utility functions to perform a quick calibration of DESI data
desispec.scripts.calibrate_tsnr_ensemble¶
Generate Master TSNR ensemble DFLUX files. See doc. 4723. Note: in this instance, ensemble avg. of flux is written, in order to efficiently generate tile depths.
- desispec.scripts.calibrate_tsnr_ensemble.tsnr_efftime(exposures_table_filename, tsnr_table_filename, tracer, plot=True, exclude=None, use_sv1=False)[source]¶
Given an external calibration, e.g. /global/cfs/cdirs/desi/survey/observations/SV1/sv1-exposures.fits
with e.g. EFFTIME_DARK and
a tsnr afterburner run, e.g. /global/cfs/cdirs/desi/spectro/redux/cascades/tsnr-cascades.fits
Compute linear coefficient to convert TSNR2_TRACER_BRZ to EFFTIME_DARK or EFFTIME_BRIGHT.
desispec.spectra.coadd_frames¶
Coadd spectra.
desispec.scripts.coadd_preproc¶
desispec.scripts.coadd_spectra¶
Coadd spectra
desispec.scripts.compute.tsnr_ensemble¶
Generate Master TSNR ensemble DFLUX files. See doc. 4723. Note: in this instance, ensemble avg. of flux is written, in order to efficiently generate tile depths.
desispec.scripts.daily_processing¶
- desispec.scripts.daily_processing.daily_processing_manager(specprod=None, exp_table_path=None, proc_table_path=None, path_to_data=None, expobstypes=None, procobstypes=None, z_submit_types=None, camword=None, badcamword=None, badamps=None, override_night=None, tab_filetype='csv', queue='realtime', exps_to_ignore=None, data_cadence_time=300, queue_cadence_time=1800, exp_cadence_time=2, dry_run_level=0, dry_run=False, no_redshifts=False, continue_looping_debug=False, dont_check_job_outputs=False, dont_resubmit_partial_jobs=False, verbose=False, use_specter=False, use_tilenight=False)[source]¶
Generates processing tables for the nights requested. Requires exposure tables to exist on disk.
- Parameters:
specprod – str. The name of the current production. If used, this will overwrite the SPECPROD environment variable.
exp_table_path – str. Full path to where to exposure tables are stored, WITHOUT the monthly directory included.
proc_table_path – str. Full path to where to processing tables to be written.
path_to_data – str. Path to the raw data.
expobstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the exposure table.
procobstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the processing table.
z_submit_types – list of str’s or comma separated list of string. The “group” types of redshifts that should be submitted with each exposure. If not specified, default for daily processing is [‘cumulative’, ‘pernight-v0’]. If false, ‘false’, or [], then no redshifts are submitted.
camword – str. Camword that, if set, alters the set of cameras that will be set for processing. Examples: a0123456789, a1, a2b3r3, a2b3r4z3.
badcamword – str. Camword that, if set, will be removed from the camword defined in camword if given, or the camword inferred from the data if camword is not given.
badamps – str. Comma seperated list of bad amplifiers that should not be processed. Should be of the form “{camera}{petal}{amp}”, i.e. “[brz][0-9][ABCD]”. Example: ‘b7D,z8A’
override_night – str or int. 8 digit night, e.g. 20200314, of data to run on. If None, it runs on the current night.
tab_filetype – str. The file extension (without the ‘.’) of the exposure and processing tables.
queue – str. The name of the queue to submit the jobs to. Default is “realtime”.
exps_to_ignore – list. A list of exposure id’s that should not be processed. Each should be an integer.
data_cadence_time – int. Wait time in seconds between loops in looking for new data. Default is 30 seconds.
queue_cadence_time – int. Wait time in seconds between loops in checking queue statuses and resubmitting failures. Default is 1800s.
exp_cadence_time – int. Wait time in seconds between loops over each science exposure. Default 2.
dry_run_level – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
int – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
nonzero (If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
dry_run – this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.
defined (bool. When to run without submitting scripts or not. If dry_run_level is) – this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.
over-rides (then it) – this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.
no_redshifts –
True (optional. Default is False. If) –
submitted. (redshifts are not) –
continue_looping_debug – bool. FOR DEBUG purposes only. Will continue looping in search of new data until the process is terminated. Default is False.
dont_check_job_outputs – data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.
False (bool. Default is False. If) – data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.
final (the code checks for the existence of the expected) – data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.
dont_resubmit_partial_jobs – False, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
is (bool. Default is False. Must be used with dont_check_job_outputs=False. If this flag) – False, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
verbose – bool. True if you want more verbose output, false otherwise. Current not propagated to lower code, so it is only used in the main daily_processing script itself.
use_specter –
bool –
True –
specter (use) –
default. (otherwise use gpu_specter by) –
use_tilenight (bool, optional) – Default is False. If True, use desi_proc_tilenight for prestdstar, stdstar, and poststdstar steps for science exposures.
Returns: Nothing
Notes
Generates both exposure table and processing tables ‘on the fly’ and saves them at various checkpoints. These should be capable of being reloaded in case of interuption or accidental termination of the manager’s process.
desispec.scripts.editexptable¶
- desispec.scripts.editexptable.change_exposure_table_rows(exptable, exp_str, colname, value, include_comment='', append_string=False, overwrite_value=False, joinsymb=',')[source]¶
Changes the column named colname to value of value for rows of exposure table in exptable that correspond to the exposures defined in exp_str.
Note: This edits and returns the exptable given in the inputs.
- Parameters:
exptable –
exposure. (astropy.table.Table. An exposure table defined in desispec.workflow.exptable. Each column is an) –
exp_str – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.
value. (bool. Default is False. Must be set to True if you want to overwrite a non-default) – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.
colname –
values. (str. The column name in the exptable where you want to change) –
value –
to. (any scalar type. The value you want to change the column value of each exp_str exposure row) –
include_comment – value for the given exp_str exposures.
to (str. A user specified comment to be added to the COMMENTS column after setting colname) – value for the given exp_str exposures.
append_string –
string. (bool. True if you want to append your input value to the end of an existing) –
overwrite_value – If current value is a default value for that column for that row, this doesn’t need to be set.
value. – If current value is a default value for that column for that row, this doesn’t need to be set.
joinsymb – Default is ‘,’.
'|'. (str. The symbol used to separate string elements that are being appended. Shouldn't be) – Default is ‘,’.
- Returns:
- exptable, astropy.table.Table. The exposure table given in the input, with edits made to the column colname
for the rows corresponding to the exposure ID’s in exp_str.
- desispec.scripts.editexptable.columns_not_to_append()[source]¶
Defines column names that shouldn’t be edited.
- desispec.scripts.editexptable.columns_not_to_edit()[source]¶
Defines column names that shouldn’t be edited.
- desispec.scripts.editexptable.columns_not_to_report()[source]¶
Returns list of column names that shouldn’t have reporting information saved because they are user-defined values.
- desispec.scripts.editexptable.document_in_comments(tablerow, colname, value, comment_col='HEADERERR')[source]¶
Places “reporting” string in the appropriate comment column of the exposure table to document the edits being made.
Note: This alters and returns the input tablerow. How astropy handles this may vary. As of Jan 2021, I believe a copy is made in memory upon altering of a tablerow object. So the output here should be returned and assigned to overwrite the old value in the input table.
- Parameters:
tablerow –
array. (astropy.table.Row. A table row with columns colname and comment_col. Comment_col must be a numpy) –
colname –
edited. (str. The name of the column that is being) –
value –
to. (any scalar type. The value that the column's current value should be changed) –
comment_col –
HEADERERR. (str. The name of the comment column where the change reporting should be placed. Default is) –
- Returns:
- tablerow, astropy.table.Row. A table row with columns colname and comment_col. Comment_col is a numpy array
with the new reporting string included.
- desispec.scripts.editexptable.edit_exposure_table(exp_str, colname, value, night=None, include_comment='', tablepath=None, append_string=False, overwrite_value=False, use_spec_prod=True, read_user_version=False, write_user_version=False, overwrite_file=True, joinsymb=',')[source]¶
Edits the exposure table on disk to change the column named colname to value of value for rows of exposure table that correspond to the exposures defined in exp_str. The table on disk can be defined using night given directly with tablepath.
Note: This overwrites an exposure table file on disk by default.
- Parameters:
exp_str – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.
value. (bool. Default is False. Must be set to True if you want to overwrite a non-default) – The string can be any combination of integer ranges, single integers, or ‘all’. Each range or integer is separated by a comma. ‘all’ implies all exposures. Ranges can be given using ‘:’, ‘-’, or ‘..’. All ranges are assumed to be inclusive.
colname –
values. (str. The column name in the exptable where you want to change) –
value –
to. (any scalar type. The value you want to change the column value of each exp_str exposure row) –
night –
table. (str or int. The night the exposures were acquired on. This uniquely defines the exposure) –
include_comment – value for the given exp_str exposures.
to (str. A user specified comment to be added to the COMMENTS column after setting colname) – value for the given exp_str exposures.
tablepath – in desispec.workflow.exptable.
file (str. A relative or absolute path to the exposure table) – in desispec.workflow.exptable.
default (if named differently from the) – in desispec.workflow.exptable.
append_string –
string. (bool. True if you want to append your input value to the end of an existing) –
overwrite_value – If current value is a default value for that column for that row, this doesn’t need to be set.
value. – If current value is a default value for that column for that row, this doesn’t need to be set.
use_spec_prod – defined SPECPROD as opposed to the exposure table repository location. Default is True.
currently (bool. True if you want to read in the exposure table defined by night from the) – defined SPECPROD as opposed to the exposure table repository location. Default is True.
read_user_version – USER name. Meant for test editing of a file multiple times. If the file doesn’t exist, the non-user value is loaded. Default is False.
user's (bool. True if you want to write in an exposure table saved including the current) – USER name. Meant for test editing of a file multiple times. If the file doesn’t exist, the non-user value is loaded. Default is False.
write_user_version – USER name. Meant for test editing of a file without overwriting the true exposure table. Default is False.
user's – USER name. Meant for test editing of a file without overwriting the true exposure table. Default is False.
overwrite_file –
True. (bool. True if you want to overwrite the file on disk. Default is) –
joinsymb – Default is ‘,’.
'|'. (str. The symbol used to separate string elements that are being appended. Shouldn't be) – Default is ‘,’.
- desispec.scripts.editexptable.parse_int_list(input_string, allints=None, only_unique=True)[source]¶
Given a str this determines what integer values it represents. Whether that be “all” indicating all ints in the table column, a range of integers specified with ‘:’, ‘-’, or ‘..’, a single integer, or an indeterminant number of them in a comma separated list.
- Parameters:
input_string – multiple of these separated by a comma. ‘all’ requires allints.
range (str. String with either integer) – multiple of these separated by a comma. ‘all’ requires allints.
integer (single) – multiple of these separated by a comma. ‘all’ requires allints.
of (or 'all'. It can have a combination) – multiple of these separated by a comma. ‘all’ requires allints.
allints –
specified. (np.array. One dimensional array of all integers. Returns if 'all' is) –
only_unique – are kept.
string (bool. True if you want a unique set returned. Otherwise repeated entries in the input) – are kept.
- Returns:
out_array, np.array. Array of ints for the string specified.
- desispec.scripts.editexptable.parse_int_list_term(input_string, allints=None)[source]¶
Given a str this determines what integer values it represents. Whether that be “all” indicating all ints in the table column, a range of integers specified with ‘:’, ‘-’, or ‘..’, or a single integer. This should not be a list.
- Parameters:
input_string –
range (str. String with either integer) –
integer (single) –
allints (or 'all'. 'all' requires) –
allints –
specified. (np.array. One dimensional array of all integers. Returns if 'all' is) –
- Returns:
out_array, np.array. Array of ints for the string specified.
- desispec.scripts.editexptable.process_int_range_inclusive(input_string)[source]¶
Given a str indicating a range of integers, this auto-detects the symbol used and returns that range as an INCLUSIVE numpy array of ints. Symbol can be ‘:’, ‘-’, or ‘..’.
- Parameters:
input_string – 102 returns 100,101,102.
100 (str. String with integer range with the upper value being included in the output. E.g.) – 102 returns 100,101,102.
- Returns:
np.array. Array of ints for the range specified in the input_string.
- desispec.scripts.editexptable.validate_value(colname, value, joinsymb)[source]¶
Checks that the value provided matches the syntax of the colname given. If the syntax is incorrect an error is raised.
Warning: may change the value of “value” and returns it.
- Parameters:
colname –
edited. (str. The name of the column that is being) –
value –
to. (any scalar type. The value that the column's current value should be changed) –
- Returns:
- value, any scalar type. The value that the column’s current value should be changed to. This is verified to
have the proper syntax for the colname given.
desispec.scripts.emlinefit¶
desispec.scripts.exposure_qa¶
This script computes QA scores per exposure, after the cframe are done
- desispec.scripts.exposure_qa._func(arg)[source]¶
Wrapper function to compute_exposure_qa for multiprocessing
- desispec.scripts.exposure_qa.func(night, expid, specprod_dir, outfile=None)[source]¶
Wrapper function to compute_exposure_qa for multiprocessing
desispec.scripts.exposuretable¶
- desispec.scripts.exposuretable.create_exposure_tables(nights=None, night_range=None, path_to_data=None, exp_table_path=None, obstypes=None, exp_filetype='csv', cameras=None, bad_cameras=None, badamps=None, verbose=False, no_specprod=False, overwrite_files=False)[source]¶
Generates processing tables for the nights requested. Requires exposure tables to exist on disk.
- Parameters:
nights – str, int, or comma separated list. The night(s) to generate procesing tables for.
night_range – str, comma separated pair of nights in form YYYYMMDD,YYYYMMDD for first_night,last_night specifying the beginning and end of a range of nights to be generated. last_night should be inclusive.
path_to_data – str. The path to the raw data and request*.json and manifest* files.
exp_table_path – str. Full path to where to exposure tables should be saved, WITHOUT the monthly directory included.
obstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the exposure table.
exp_filetype – str. The file extension (without the ‘.’) of the exposure tables.
verbose – boolean. Whether to give verbose output information or not. True prints more information.
no_specprod – boolean. Create exposure table in repository location rather than the SPECPROD location
overwrite_files – boolean. Whether to overwrite processing tables if they exist. True overwrites.
cameras – str. Explicitly define the cameras for which you want to reduce the data. Should be a comma separated list. Only numbers assumes you want to reduce r, b, and z for that camera. Otherwise specify separately [brz][0-9].
bad_cameras – str. Explicitly define the cameras that you don’t want to reduce the data. Should be a comma separated list. Only numbers assumes you want to reduce r, b, and z for that camera. Otherwise specify separately [brz][0-9].
badamps – str. Define amplifiers that you know to be bad and should not be processed. Should be a list separated by comma or semicolon. Saved list will converted to semicolons. Each entry should be of the form {camera}{spectrograph}{amp}, i.e. [brz][0-9][A-D].
Returns: Nothing
desispec.scripts.extract¶
Extract spectra from DESI pre-processed raw data.
- desispec.scripts.extract._extract_and_save(img, psf, bspecmin, bnspec, specmin, wave, raw_wave, fibers, fibermap, outbundle, outmodel, bundlesize, args, log)[source]¶
Performs the main extraction and saving of extracted frames found in the body of the main loop. Refactored to be callable by both MPI and non-MPI versions of the code. This should be viewed as a shorthand for the following commands.
- desispec.scripts.extract.barycentric_correction_multiplicative_factor(header)[source]¶
Returns mult. barycentric correction factor using coords in header
header must contrain MJD or MJD-OBS; and TARGTRA,TARGTDEC or SKYRA,SKYDEC or TELRA,TELDEC or RA,DEC
- desispec.scripts.extract.gpu_specter_check_input_options(args)[source]¶
Perform pre-flight checks on input options
returns ok(True/False), message
desispec.scripts.fibercrosstalk¶
desispec.scripts.fiberflat¶
Utility functions to compute a fiber flat correction and apply it We try to keep all the (fits) io separated.
desispec.scripts.fluxcalibration¶
desispec.scripts.group_spectra¶
Regroup spectra by healpix.
desispec.scripts.healpix_redshifts¶
Script for running healpix-based coadds+redshifts, assuming that the spectral regrouping into healpix has already happened.
desispec.scripts.humidity_corrected_fiberflat¶
desispec.scripts.inspect_dark¶
desispec.scripts.interpolate_fiber_psf¶
desispec.scripts.makezmtl¶
desispec.scripts.master_nea¶
This script generates a master NEA (Noise Equivalent Area) file for a given camera.
- desispec.scripts.master_nea.process_one(w, psf, ifiber)[source]¶
Compute the 1D NEA for a given wavelength, fiber and desispec psf instance.
- input:
w: wavelength, Angstroms. psf: desispec psf instance. ifiber: fiber indexing integer [0,500].
- Returns:
list of 1D nea value [pixles] and angstroms per pix. for this fiber and wavelength.
desispec.scripts.mergebundles¶
exspec extracts individual bundles of spectra with one bundle per output file. This script merges them back together into a single file combining all bundles.
This workflow is hacky. Release early, release often, but also refactor often.
Stephen Bailey, LBL March 2014
desispec.scripts.night¶
Automated nightly processing.
desispec.scripts.nightly_bias¶
functions for bin/desi_compute_nightly_bias script
desispec.scripts.pipe¶
Interactive control of the pipeline
desispec.scripts.pipe_exec¶
Run one or more pipeline tasks.
desispec.scripts.preproc¶
Command line wrappers for pre-processing a DESI raw exposure
- desispec.scripts.preproc._preproc_file_kwargs_wrapper(opts)[source]¶
This function just unpacks opts dict for preproc_file so that it can be used with multiprocessing.Pool.map
- desispec.scripts.preproc.preproc_file(infile, camera, outfile=None, outdir=None, fibermap=None, zero_masked=False, **preproc_opts)[source]¶
Preprocess a single camera from a single input file
- Parameters:
infile – input raw data file
camera – camera, e.g. ‘b0’, ‘r1’, ‘z9’
- Options:
outfile: output preprocessed image file to write outdir: output directory; derive filename from infile NIGHT and EXPID fibermap: fibermap filename to include in output zero_masked (bool): set masked pixels to 0 preproc_opts: dictionary to pass to preproc
Returns error code (1=error, 0=success) but will not raise exception if there is an I/O or preprocessing failure (allows other parallel procs to proceed).
Note: either outfile or outdir must be provided
desispec.scripts.proc¶
One stop shopping for processing a DESI exposure
Examples at NERSC:
# ARC: 18 min on 2 nodes
time srun -N 2 -n 60 -C haswell -t 25:00 --qos realtime desi_proc --mpi -n 20191029 -e 22486
# FLAT: 13 min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22487
# TWILIGHT: 8min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22497
# SKY: 11 min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22536
# ZERO: 2 min
time srun -n 20 -N 1 -C haswell -t 15:00 --qos realtime desi_proc --mpi -n 20191029 -e 22561
desispec.scripts.proc_joint_fit¶
desispec.scripts.proc_tilenight¶
Script for science processing of a given DESI tile and night
desispec.scripts.procdashboard¶
- desispec.scripts.procdashboard.main(args=None)[source]¶
Code to generate a webpage for monitoring of desi_dailyproc production status Usage: -n can be ‘all’ or series of nights separated by comma or blank like 20200101,20200102 or 20200101 20200102 Normal Mode: desi_proc_dashboard -n 3 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/ desi_proc_dashboard -n 20200101,20200102 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/
- desispec.scripts.procdashboard.populate_night_info(night, check_on_disk=False, night_json_info=None, skipd_expids=None)[source]¶
For a given night, return the file counts and other other information for each exposure taken on that night input: night output: a dictionary containing the statistics with expid as key name FLAVOR: FLAVOR of this exposure OBSTYPE: OBSTYPE of this exposure EXPTIME: Exposure time SPECTROGRAPHS: a list of spectrographs used n_spectrographs: number of spectrographs n_psf: number of PSF files n_ff: number of fiberflat files n_frame: number of frame files n_sframe: number of sframe files n_cframe: number of cframe files n_sky: number of sky files
desispec.scripts.processingtable¶
- desispec.scripts.processingtable.create_processing_tables(nights=None, night_range=None, exp_table_path=None, proc_table_path=None, obstypes=None, overwrite_files=False, verbose=False, no_specprod_exptab=False, exp_filetype='csv', prod_filetype='csv', joinsymb='|')[source]¶
Generates processing tables for the nights requested. Requires exposure tables to exist on disk.
- Parameters:
nights – str, int, or comma separated list. The night(s) to generate procesing tables for.
night_range – str, comma separated pair of nights in form YYYYMMDD,YYYYMMDD for first_night,last_night specifying the beginning and end of a range of nights to be generated. last_night should be inclusive.
exp_table_path – str. Full path to where to exposure tables are stored, WITHOUT the monthly directory included.
proc_table_path – str. Full path to where to processing tables to be written.
obstypes – str or comma separated list of strings. The exposure OBSTYPE’s that you want to include in the processing table.
overwrite_files – boolean. Whether to overwrite processing tables if they exist. True overwrites.
verbose – boolean. Whether to give verbose output information or not. True prints more information.
no_specprod_exptab – boolean. Read exposure table in repository location rather than the SPECPROD location.
exp_filetype – str. The file extension (without the ‘.’) of the exposure tables.
prod_filetype – str. The file extension (without the ‘.’) of the processing tables.
joinsymb – str. Symbol to use to indicate the separation of array values when converting to and from strings for saving to csv. Default is highly advised and is ‘|’. Using a comma will break many things.
Returns: Nothing
Notes
Requires exposure tables to exist on disk. Either in the default location or at the location specified using the function arguments.
desispec.scripts.procexp¶
This script processes an exposure by applying fiberflat, sky subtraction, spectro-photometric calibration depending on input. Optionally, includes tsnr in the scores hdu.
desispec.scripts.qa_exposure¶
Script for generating QA for a full exposure
desispec.scripts.qa_frame¶
Script for generating QA for a single Frame
desispec.scripts.qa_night¶
Script for analyzing QA from a Night
desispec.scripts.qa_prod¶
Script for generating QA from a Production run
desispec.scripts.qproc¶
Run DESI qproc on a given exposure
desispec.scripts.qsomgii¶
- desispec.scripts.qsomgii.save_dataframe_to_fits(dataframe, filename, DESI_TARGET, clobber=True)[source]¶
Save info from pandas dataframe in a fits file. Need to write the dtype array because of the list in the pandas dataframe (no other solution found)
- desispec.scripts.qsomgii.select_targets_with_mgii_fitter(redrock, fibermap, sel_to_mgii, spectra_name, redrock_name, param_mgii_fitter, DESI_TARGET, save_target)[source]¶
Run QuasarNet to the object with index_to_QN == True from spectra_name. Then, Re-Run RedRock for the targetids which are selected by QN as a QSO.
- Parameters:
redrock – fitsio hdu ‘REDSHIFTS’ from redrock file
fibermap – fitsio hdu ‘FIBERMAP’ from redrock file
sel_to_mgii (bool array) – size 500. Select on which objects mgii will be apply (index based on redrock table)
redrock_name (spectra_name /) – The name of the spectra / associated redrock file
param_mgii_fitter (dict) – contains info for the MgII fitter as lambda_width, max_sigma min_sigma, min_deltachi2, min_signifiance_A, min_A
DESI_TARGET (str) – name of DESI_TARGET for the wanted version of the target selection
save_target (str) – restricted (save only IS_QSO_MGII==true targets) / all (save all the sample)
- Returns:
contains all the information useful to build the QSO cat
- Return type:
QSO_sel (pandas dataframe)
desispec.scripts.qsoqn¶
- desispec.scripts.qsoqn.collect_redshift_with_new_RR_run(spectra_name, targetid, z_prior, param_RR, comm=None)[source]¶
Wrapper to run Redrock on targetid (numpy array) from the spectra_name_file with z_prior using the template contained in template_file
- Parameters:
spectra_name (str) – The name of the spectra file.
targetid (int array) – Array of the targetid (contained in the spectra_name_file) on which RR will be rerun with prior and qso template.
z_prior (float array) – Array of the same size than targetid with the redshift estimated by QN for the associated targetid
param_RR (dict) – Contains info to re-run RR as the template_filename, filename_priors, filename_output_rerun_RR, filename_redrock_rerun_RR
comm (object, optional) – MPI communicator to pass to redrock; must be size=1
- Returns:
A tuple containing:
redshift (numpy array): Array containing best redshift estimation by the new run of RR
err_redshift (numpy array): Array containing the associated error for the redshift
coeffs (numpy array): array containing the coefficient for the best fit given by RR even we work only with QSO template, it has a “shape” of redshift.size x 10; warning: they have to be converted into a list (with .tolist()) to be added in the pandas dataframe
- Return type:
- desispec.scripts.qsoqn.save_dataframe_to_fits(dataframe, filename, DESI_TARGET, clobber=True)[source]¶
Save info from pandas dataframe in a fits file. Need to write the dtype array because of the list in the pandas dataframe (no other solution found)
- desispec.scripts.qsoqn.selection_targets_with_QN(redrock, fibermap, sel_to_QN, DESI_TARGET, spectra_name, param_QN, param_RR, save_target, comm=None)[source]¶
Run QuasarNet to the object with index_to_QN == True from spectra_name. Then, Re-Run RedRock for the targetids which are selected by QN as a QSO.
- Parameters:
redrock – fitsio hdu ‘REDSHIFTS’ from redrock file
fibermap – fitsio hdu ‘FIBERMAP’ from redrock file
sel_to_QN (bool array) – Select on which objects QN will be apply (index based on redrock)
DESI_TARGET (str) – name of DESI_TARGET for the wanted version of the target selection
spectra_name (str) – The name of the spectra file
param_QN (dict) – contains info for QN as n_thresh and c_thresh
param_RR (dict) – contains info to re-run RR as the template_filename, filename_priors, filename_output_rerun_RR, filename_redrock_rerun_RR
save_target (str) – restricted (save only IS_QSO_QN_NEW_RR==true targets) / all (save all the sample)
comm – MPI communicator to pass to redrock; must be size=1
optional – MPI communicator to pass to redrock; must be size=1
- Returns:
contains all the information useful to build the QSO cat
- Return type:
QSO_sel (pandas dataframe)
desispec.scripts.quicklook¶
Command line wrapper for running a QL pipeline
QuickLook team @Southern Methodist University (SMU) First version Spring 2016 Latest revision July 2018
Running QuickLook:
desi_quicklook -i qlconfig_science.yaml -n 20191001 -c r0 -e 3577
This requires having necessary input files and setting the following environment variables:
QL_SPEC_DATA: directory containing raw/fibermap files (full path: $QL_SPEC_DATA/night/expid)
QL_SPEC_REDUX: directory for QL output (full path: $QL_SPEC_REDUX/exposures/night/expid)
DESI_CALIBRATION_DATA: directory containing calibration files
Necessary Quicklook command line arguments:
-i,--config_file : path to QL configuration file
-n,--night : night to be processed
-c,--camera : camera to be processed
-e,--expid : exposure ID to be processed
Optional QuickLook arguments:
--rawdata_dir : directory containing raw/fibermap files (overrides $QL_SPEC_DATA)
--specprod_dir : directory for QL output (overrides $QL_SPEC_REDUX)
Plotting options:
-p (including path to plotting configuration file) : generate configured plots
-p (only using -p with no configuration file) : generate QL hardcoded plots
- desispec.scripts.quicklook.parse()[source]¶
Should have either a pre existing config file, or need to generate one using config module
desispec.scripts.rejectcosmics¶
This script finds cosmics in a pre-processed image and write the result in the mask extension of an output image (output can be same as input).
desispec.scripts.repair_mask¶
Patch holes in spectrograph 2D cosmic-ray masks using morphological binary image analysis.
desispec.scripts.select_calib_stars¶
desispec.scripts.sky¶
desispec.scripts.skysubresid¶
Script for generating plots on SkySub residuals
desispec.scripts.specex¶
Run PSF estimation.
- desispec.scripts.specex.compatible(head1, head2)[source]¶
Return bool for whether two FITS headers are compatible for merging PSFs
- desispec.scripts.specex.mean_psf(inputs, output)[source]¶
Average multiple input PSF files into an output PSF file
- Parameters:
inputs – list of input PSF files
output – output filename
- desispec.scripts.specex.merge_psf(inputs, output)[source]¶
Merge individual per-bundle PSF files into full PSF
- Parameters:
inputs – list of input PSF filenames
output – output filename
- desispec.scripts.specex.run(comm, cmds, cameras)[source]¶
Run PSF fits with specex on a set of ccd images in parallel using the run method of the desispec.workflow.schedule.Schedule (Schedule) class.
- Parameters:
comm – MPI communicator containing all processes available for work and scheduling (usually MPI_COMM_WORLD); at least 21 processes should be available, one for scheduling and (group_size=) 20 to fit all bundles for a given ccd image. Otherwise there is no constraint on the number of ranks available, but (comm.Get_size()-1)%group_size will be unused, since every job is assigned exactly group_size=20 ranks. The variable group_size is set at the number of bundles on a ccd, and there is currently no support for any other number, due to the way merging of bundles is currently done.
cmds – dictionary keyed by a camera string (e.g. ‘b0’, ‘r1’, …) with values being the ‘desi_compute_psf …’ string that one would run on the command line.
cameras – list of camera strings identifying the entries in cmds to be run as jobs in parallel jobs, one entry per ccd image to be fit. Processes assigned to cameras not present as keys in cmds will write a message to the log instead of running a PSF fit.
The function first defines the procedure to call specex for a given ccd image with the “fitframe” inline function, passes the fitframe function to the Schedule initialization method, and then calls the run method of the Schedule class to call fitframe len(cameras) times, each with group_size = 20 processes.
desispec.scripts.specscore¶
Compute some information scores on spectra in frames
desispec.scripts.stdstars¶
Get the normalized best template to do flux calibration.
- desispec.scripts.stdstars.get_gaia_ab_correction()[source]¶
Get the dictionary with corrections from AB magnitudes to Vega magnitudes (as the official gaia catalog is in vega)
- desispec.scripts.stdstars.get_magnitude(stdwave, model, model_filters, cur_filt)[source]¶
Obtain magnitude for a filter taking into account the ab/vega correction if needed. Wwe assume the flux is in units of 1e-17 erg/s/cm^2/A
- desispec.scripts.stdstars.main(args=None, comm=None)[source]¶
finds the best models of all standard stars in the frame and normlize the model flux. Output is written to a file and will be called for calibration.
desispec.scripts.submit_night¶
- desispec.scripts.submit_night.get_completed_tiles(specstatus_path=None, complete_tiles_thrunight=None)[source]¶
Uses a tiles-specstatus.ecsv file and selection criteria to determine what tiles have beeen completed. Takes an optional argument to point to a custom specstatus file. Returns an array of TILEID’s.
- Parameters:
specstatus_path – Location of the surveyops specstatus table.
str – Location of the surveyops specstatus table.
$DESI_SURVEYOPS/ops/tiles-specstatus.ecsv. (optional. Default is) – Location of the surveyops specstatus table.
complete_tiles_thrunight – on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None.
int – on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None.
completed (optional. Default is None. Only tiles) – on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None.
- Returns:
array-like. The tiles from the specstatus file determined by the selection criteria to be completed.
- desispec.scripts.submit_night.submit_night(night, proc_obstypes=None, z_submit_types=None, queue='realtime', reservation=None, system_name=None, exp_table_path=None, proc_table_path=None, tab_filetype='csv', dry_run_level=0, dry_run=False, no_redshifts=False, error_if_not_available=True, append_to_proc_table=False, ignore_proc_table_failures=False, dont_check_job_outputs=False, dont_resubmit_partial_jobs=False, tiles=None, surveys=None, laststeps=None, use_tilenight=False, all_tiles=False, specstatus_path=None, use_specter=False, do_cte_flat=False, complete_tiles_thrunight=None, all_cumulatives=False)[source]¶
Creates a processing table and an unprocessed table from a fully populated exposure table and submits those jobs for processing (unless dry_run is set).
- Parameters:
night (int) – The night of data to be processed. Exposure table must exist.
proc_obstypes (list or np.array, optional) – A list of exposure OBSTYPE’s that should be processed (and therefore added to the processing table).
z_submit_types (list of str or comma-separated list of str, optional) – The “group” types of redshifts that should be submitted with each exposure. If not specified, default for daily processing is [‘cumulative’, ‘pernight-v0’]. If false, ‘false’, or [], then no redshifts are submitted.
queue (str, optional) – The name of the queue to submit the jobs to. Default is “realtime”.
reservation (str, optional) – The reservation to submit jobs to. If None, it is not submitted to a reservation.
system_name (str) – batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu
exp_table_path (str) – Full path to where to exposure tables are stored, WITHOUT the monthly directory included.
proc_table_path (str) – Full path to where to processing tables to be written.
tab_filetype (str, optional) – The file extension (without the ‘.’) of the exposure and processing tables.
dry_run_level (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written but not submitted. If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
dry_run (bool, optional) – When to run without submitting scripts or not. If dry_run_level is defined, then it over-rides this flag. dry_run_level not set and dry_run=True, dry_run_level is set to 2 (no scripts generated or run). Default for dry_run is False.
no_redshifts (bool, optional) – Whether to submit redshifts or not. If True, redshifts are not submitted.
error_if_not_available (bool, optional) – Default is True. Raise as error if the required exposure table doesn’t exist, otherwise prints an error and returns.
append_to_proc_table (bool, optional) – True if you want to submit jobs even if a processing table already exists. Otherwise jobs will be appended to it. Default is False
ignore_proc_table_failures (bool, optional) – True if you want to submit other jobs even the loaded processing table has incomplete jobs in it. Use with caution. Default is False.
dont_check_job_outputs (bool, optional) – Default is False. If False, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is False, then the script will not be submitted. If some files exist and this is False, only the subset of the cameras without the final data products will be generated and submitted.
dont_resubmit_partial_jobs (bool, optional) – Default is False. Must be used with dont_check_job_outputs=False. If this flag is False, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
tiles (array-like, optional) – Only submit jobs for these TILEIDs.
surveys (array-like, optional) – Only submit science jobs for these surveys (lowercase)
laststeps (array-like, optional) – Only submit jobs for exposures with LASTSTEP in these laststeps (lowercase)
use_tilenight (bool, optional) – Default is False. If True, use desi_proc_tilenight for prestdstar, stdstar, and poststdstar steps for science exposures.
all_tiles (bool, optional) – Default is False. Set to NOT restrict to completed tiles as defined by the table pointed to by specstatus_path.
specstatus_path (str, optional) – Default is $DESI_SURVEYOPS/ops/tiles-specstatus.ecsv. Location of the surveyops specstatus table.
use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.
do_cte_flat (bool, optional) – Default is False. If True, one second flat exposures are processed for cte identification.
complete_tiles_thrunight (int, optional) – Default is None. Only tiles completed on or before the supplied YYYYMMDD are considered completed and will be processed. All complete tiles are submitted if None or all_tiles is True.
all_cumulatives (bool, optional) – Default is False. Set to run cumulative redshifts for all tiles even if the tile has observations on a later night.
desispec.scripts.submit_prod¶
- desispec.scripts.submit_prod.assign_survey(night, conf)[source]¶
Takes a desi production configuration (yaml) dictionary and determines the survey corresponding to a given night based on the contents of the conf dictionary, if psosible. Otherwise returns None.
- desispec.scripts.submit_prod.get_all_nights()[source]¶
Returns a full list of all nights availabel in the DESI Raw data directory.
- Returns:
nights, list. A list of nights on or after Jan 1 2020 in which data exists at NERSC.
- desispec.scripts.submit_prod.submit_production(production_yaml, dry_run=False, error_if_not_available=False)[source]¶
Interprets a production_yaml file and submits the respective nights for processing within the defined production.
- Parameters:
production_yaml (str) – Pathname of the yaml file that defines the production.
dry_run (bool, optional) – Default is False. Should the jobs written to the processing table actually be submitted for processing.
error_if_not_available (bool, optional) – Default is True. Raise as error if the required exposure table doesn’t exist, otherwise prints an error and returns.
- Returns:
None.
desispec.scripts.tile_redshifts¶
- desispec.scripts.tile_redshifts.batch_tile_redshifts(tileid, exptable, group, camword=None, submit=False, queue='regular', reservation=None, max_gpuprocs=None, no_gpu=False, dependency=None, system_name=None, run_zmtl=False, no_afterburners=False)[source]¶
Generate batch script for spectra+coadd+redshifts for a tile
- Parameters:
- Options:
camword (str): camword of cameras to include submit (bool): also submit batch script to queue queue (str): batch queue name reservation (str): batch reservation name max_gpuprocs (int): Number of gpu processes no_gpu (bool): Default false. If true it doesn’t use GPU’s even if available. dependency (str): passed to sbatch –dependency upon submit system_name (str): batch system name, e.g. cori-haswell, perlmutter-gpu run_zmtl (bool): if True, also run make_zmtl_files no_afterburners (bool): if True, do not run QSO afterburners
- Returns tuple (scriptpath, error):
scriptpath (str): full path to generated script err (int): return code from submitting job (0 if submit=False)
By default this generates the script but don’t submit it
- desispec.scripts.tile_redshifts.generate_tile_redshift_scripts(group, nights=None, tileid=None, expids=None, explist=None, spectrographs=None, camword=None, max_gpuprocs=None, no_gpu=False, run_zmtl=False, no_afterburners=False, batch_queue='regular', batch_reservation=None, batch_dependency=None, system_name=None, nosubmit=False)[source]¶
Creates a slurm script to run redshifts per tile. By default it also submits the job to Slurm. If nosubmit is True, the script is created but not submitted to Slurm.
- Parameters:
group (str) – Type of coadd redshifts to run. Options are cumulative, pernight, perexp, or a custom name.
nights (int, or list or np.array of int's) – YEARMMDD nights.
tileid (int) – Tile ID.
explist (str) – File with columns TILE NIGHT EXPID to use
spectrographs (str or list of int) – spectrographs to include
camword (str) – camword of cameras to include
max_gpuprocs (int) – Number of gpu processes
no_gpu (bool) – Default false. If true it doesn’t use GPU’s even if available.
run_zmtl (bool) – If True, also run make_zmtl_files
no_afterburners (bool) – If True, do not run QSO afterburners
batch_queue (str) – Batch queue name. Default is ‘regular’.
batch_reservation (str) – Batch reservation name.
batch_dependency (str) – Job dependencies passed to sbatch –dependency .
system_name (str) – Batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu.
nosubmit (bool) – Generate scripts but don’t submit batch jobs. Default is False.
- Returns:
- The path names of the scripts created during the function call
that returned a null batcherr.
- failed_jobs (list of str): The path names of the scripts created during the function call
that returned a batcherr.
- Return type:
Note: specify
spectrographs
orcamword
but not both
desispec.scripts.tile_redshifts_bash¶
- desispec.scripts.tile_redshifts_bash.batch_tile_redshifts(tileid, exptable, group, spectrographs=None, submit=False, queue='realtime', reservation=None, dependency=None, system_name=None, run_zmtl=False, noafterburners=False)[source]¶
Generate batch script for spectra+coadd+redshifts for a tile
- Parameters:
- Options:
spectrographs (list of int): spectrographs to include submit (bool): also submit batch script to queue queue (str): batch queue name reservation (str): batch reservation name dependency (str): passed to sbatch –dependency upon submit system_name (str): batch system name, e.g. cori-haswell, perlmutter-gpu run_zmtl (bool): if True, also run make_zmtl_files noafterburners (bool): if True, do not run QSO afterburners
- Returns tuple (scriptpath, error):
scriptpath (str): full path to generated script err (int): return code from submitting job (0 if submit=False)
By default this generates the script but don’t submit it
- desispec.scripts.tile_redshifts_bash.generate_tile_redshift_scripts(group, night=None, tileid=None, expid=None, explist=None, spectrographs=None, run_zmtl=False, noafterburners=False, batch_queue='realtime', batch_reservation=None, batch_dependency=None, system_name=None, nosubmit=False)[source]¶
Creates a slurm script to run redshifts per tile. By default it also submits the job to Slurm. If nosubmit is True, the script is created but not submitted to Slurm.
- Parameters:
group (str) – Type of coadd redshifts to run. Options are cumulative, pernight, perexp, or a custom name.
night (int, or list or np.array of int's) – YEARMMDD nights.
tileid (int) – Tile ID.
explist (str) – File with columns TILE NIGHT EXPID to use
spectrographs (str or list of int) – spectrographs to include
run_zmtl (bool) – If True, also run make_zmtl_files
noafterburners (bool) – If True, do not run QSO afterburners
batch_queue (str) – Batch queue name. Default is ‘realtime’.
batch_reservation (str) – Batch reservation name.
batch_dependency (str) – Job dependencies passed to sbatch –dependency .
system_name (str) – Batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu.
nosubmit (bool) – Generate scripts but don’t submit batch jobs. Default is False.
- Returns:
- The path names of the scripts created during the function call
that returned a null batcherr.
- failed_jobs (list of str): The path names of the scripts created during the function call
that returned a batcherr.
- Return type:
- desispec.scripts.tile_redshifts_bash.write_redshift_script(batchscript, outdir, jobname, num_nodes, group, spectro_string, suffix, frame_glob=None, expfile=None, healpix=None, extra_header=None, queue='regular', system_name=None, onetile=True, tileid=None, night=None, expid=None, nexps=0, run_zmtl=False, noafterburners=False, redrock_nodes=1, redrock_cores_per_rank=1)[source]¶
Write a batch script for running coadds, redshifts, and afterburners
- Parameters:
batchscript (str) – filepath to batch script to write
outdir (str) – output directory to write data
jobname (str) – slurm job name
num_nodes (int) – number of nodes to allocate
group (str) – used for tile redshifts, e.g. ‘cumulative’
spectro_string (str) – e.g. ‘0 1 2 3’ spectrographs to run
suffix (str) – filename suffix (e.g. TILEID-thruNIGHT)
- Options:
frame_glob (str): glob for finding input cframes expfile (str): filename with NIGHT EXPID SPECTRO healpix (int): healpix number (to use with group=healpix) extra_header (dict): extra key/value pairs to add to header queue (str): queue name system_name (str): e.g. cori-haswell, cori-knl, perlmutter-gpu onetile (bool): coadd assuming input is for a single tile? tileid (int): tileid to process; only needed for group=’cumulative’ night (int): process through or on night YEARMMDD; for group=’cumulative’ and ‘pernight’ expid (int): expid for group=’perexp’ nexps (int): number of exposures to be fit run_zmtl (bool): if True, also run zmtl noafterburners (bool): if True, skip QSO afterburners redrock_nodes (int): number of nodes for each redrock call redrock_cores_per_rank (int): number of cores/rank to use for redrock
Note: some of these options are hacked to also be used by healpix_redshifts, e.g. by providing spectro_string=’sv3’ instead of list of spectrographs.
Note: Use redrock_cores_per_rank > 1 to reserve extra memory per rank for large input coadd files (e.g. sv3 healpix).
Note: must specify frame_glob for tile-based groups, and expfile for group=healpix.
desispec.scripts.tileqa¶
This script computes QA scores per exposure, after the cframe are done
- desispec.scripts.tileqa._wrap_make_tile_qa_plot(qafitsfile, specprod_dir=None)[source]¶
Utility wrapper to make qa plot with try/except/log wrappers
- Parameters:
qafitsfile (str) – full path to
tile-qa-*.fits
data
- Options:
specprod_dir (str): full path to production directory
Returns: full path to qapngfile, or None upon failure
Notes
if plotting fails, this prints traceback and logs error but doesn’t raise and exception itself, returning None instead
- desispec.scripts.tileqa.func(night, tileid, specprod_dir, exposure_qa_dir, outfile=None, group='cumulative')[source]¶
Wrapper function to compute_tile_qa for multiprocessing
desispec.scripts.trace_shifts¶
desispec.scripts.update_spectra¶
Update healpix-grouped spectra from a set of input cframe files
desispec.scripts.updateexptables¶
- desispec.scripts.updateexptables.update_exposure_tables(nights=None, night_range=None, path_to_data=None, exp_table_path=None, obstypes=None, orig_filetype='csv', out_filetype='csv', verbose=False, no_specprod=False, dry_run=False)[source]¶
Generates updated exposure tables for the nights requested. Requires exposure tables to exist on disk.
- Parameters:
nights – str, int, or comma separated list. The night(s) to generate procesing tables for.
night_range – str. comma separated pair of nights in form YYYYMMDD,YYYYMMDD for first_night,last_night specifying the beginning and end of a range of nights to be generated. last_night should be inclusive.
path_to_data – str. The path to the raw data and request*.json and manifest* files.
exp_table_path – str. Full path to where to exposure tables should be saved, WITHOUT the monthly directory included.
obstypes – str. The exposure OBSTYPE’s that you want to include in the exposure table. Can be a comma separated list.
orig_filetype – str. The file extension (without the ‘.’) of the exposure tables.
out_filetype – str. The file extension for the outputted exposure tables (without the ‘.’).
verbose – boolean. Whether to give verbose output information or not. True prints more information.
no_specprod – boolean. Create exposure table in repository location rather than the SPECPROD location
- Returns:
Nothing
Combine individual redrock files into a single zcatalog
Stephen Bailey Lawrence Berkeley National Lab Fall 2015
substantially updated Fall 2023
- desispec.scripts.zcatalog._wrap_read_redrock(optdict)[source]¶
read_redrock wrapper to expand dictionary of named args for multiprocessing
- desispec.scripts.zcatalog.load_sv1_ivar_w12(hpix, targetids)[source]¶
Load FLUX_IVAR_W1/W2 from sv1 target files for requested targetids
- Parameters:
hpix (int) – nside=8 nested healpix
targetids (array) – TARGETIDs to include
Returns table of TARGETID, FLUX_IVAR_W1, FLUX_IVAR_W2
Note: this is only for the special case of sv1 dark/bright and the FLUX_IVAR_W1/W2 columns which were not included in fiberassign for tiles designed before 20201212.
Note: nside=8 nested healpix is hardcodes for simplicity because that is what was used for sv1 target selection and this is not trying to be a more generic targetid lookup function.
- desispec.scripts.zcatalog.read_redrock(rrfile, group=None, recoadd_fibermap=False, minimal=False, pertile=False, counter=None)[source]¶
Read a Redrock file, combining REDSHIFTS, FIBERMAP, and TSNR2 HDUs
- Parameters:
rrfile (str) – full path to redrock filename
- Options:
group (str): add group-specific columns for cumulative, pernight, healpix readcoadd_fibermap (bool): recoadd fibermap from spectra file in same dir minimal (bool): only propagate minimal subet of columns pertile (bool): input Redrock file is single tile (not healpix) counter (tuple): (i,n) log loading ith file out of n
Returns (zcat, expfibermap) where zcat is a join of the redrock REDSHIFTS catalog and the coadded FIBERMAP
desispec.scripts.zproc¶
One stop shopping for redshifting DESI spectra
- desispec.scripts.zproc.distribute_ranks_to_blocks(nblocks, rank=None, size=None, comm=None, log=None, split_comm=False)[source]¶
Function to split up a set of ranks of size ‘size’ into nblock number of blocks or roughly equal size.
- Parameters:
- Returns:
A tuple containing:
nblocks, int: the achievable number of block based on size
block_size, int: the number of ranks in the assigned block of current rank
block_rank. int: the rank in the assigned block of the current rank
block_num, int: the block number (of nblocks blocks) in which the rank was assigned
block_comm (optional): if split_comm is true, returns a communicator of only the ranks in the current block. Splits from the world communicator
- Return type:
- desispec.scripts.zproc.parse(options=None)[source]¶
Parses an argparser object for use with desi_zproc and returns arguments
desispec.scripts.zprocdashboard¶
- desispec.scripts.zprocdashboard.main(args=None)[source]¶
Code to generate a webpage for monitoring of desi_dailyproc production status Usage: -n can be ‘all’ or series of nights separated by comma or blank like 20200101,20200102 or 20200101 20200102 Normal Mode: desi_proc_dashboard -n 3 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/ desi_proc_dashboard -n 20200101,20200102 –output-dir /global/cfs/cdirs/desi/www/collab/dailyproc/
- desispec.scripts.zprocdashboard.populate_night_zinfo(night, doem=True, doqso=True, dotileqa=True, check_on_disk=False, night_json_zinfo=None, skipd_tileids=None)[source]¶
For a given night, return the file counts and other other information for each exposure taken on that night input: night output: a dictionary containing the statistics with expid as key name FLAVOR: FLAVOR of this exposure OBSTYPE: OBSTYPE of this exposure EXPTIME: Exposure time SPECTROGRAPHS: a list of spectrographs used n_spectrographs: number of spectrographs n_psf: number of PSF files n_ff: number of fiberflat files n_frame: number of frame files n_sframe: number of sframe files n_cframe: number of cframe files n_sky: number of sky files
desispec.sky¶
Utility functions to compute a sky model and subtract it.
- desispec.sky._model_variance(frame, cskyflux, cskyivar, skyfibers)[source]¶
look at chi2 per wavelength and increase sky variance to reach chi2/ndf=1
- desispec.sky.calculate_throughput_corrections(frame, skymodel)[source]¶
Calculate the throughput corrections for each fiber based on the skymodel.
- Parameters:
frame (Frame object) – frame containing the data that may need to be corrected
skymodel (SkyModel object) – skymodel object that contains the information about the sky for the given exposure/frame
- Output:
- corrections (1D array): 1D array where the index corresponds to the fiber % 500 and the values are the multiplicative corrections that would
be applied to the fluxes in frame.flux to correct them based on the input skymodel
- desispec.sky.compute_sky(frame, nsig_clipping=4.0, max_iterations=100, model_ivar=False, add_variance=True, adjust_wavelength=False, adjust_lsf=False, only_use_skyfibers_for_adjustments=True, pcacorr=None, fit_offsets=False, fiberflat=None, skygradpca=None, min_iterations=5, tpcorrparam=None, exclude_sky_targetids=None, override_sky_targetids=None)[source]¶
Compute a sky model.
Sky[fiber,i] = R[fiber,i,j] Flux[j]
Input flux are expected to be flatfielded! We don’t check this in this routine.
- Parameters:
frame – Frame object, which includes attributes - wave : 1D wavelength grid in Angstroms - flux : 2D flux[nspec, nwave] density - ivar : 2D inverse variance of flux - mask : 2D inverse mask flux (0=good) - resolution_data : 3D[nspec, ndiag, nwave] (only sky fibers)
nsig_clipping – [optional] sigma clipping value for outlier rejection
max_iterations – int, maximum number of iterations
model_ivar – replace ivar by a model to avoid bias due to correlated flux and ivar. this has a negligible effect on sims.
add_variance – evaluate calibration error and add this to the sky model variance
adjust_wavelength – adjust the wavelength of the sky model on sky lines to improve the sky subtraction
adjust_lsf – adjust the LSF width of the sky model on sky lines to improve the sky subtraction
only_use_skyfibers_for_adjustments – interpolate adjustments using sky fibers only
pcacorr – SkyCorrPCA object to interpolate the wavelength or LSF adjustment from sky fibers to all fibers
fit_offsets – fit offsets for regions defined in calib
fiberflat – desispec.FiberFlat object used for the fit of offsets
skygradpca – SkyGradPCA object to use to fit sky gradients, or None
min_iterations – int, minimum number of iterations
tpcorrparam – TPCorrParam object to use to fit fiber throughput variations, or None
returns SkyModel object with attributes wave, flux, ivar, mask
- desispec.sky.get_sky_fibers(fibermap, override_sky_targetids=None, exclude_sky_targetids=None)[source]¶
Retrieve the fiber indices of sky fibers
- Parameters:
fibermap – Table from frame FIBERMAP HDU (frame.fibermap)
- Options:
override_sky_targetids (array of int): TARGETIDs to use, overriding fibermap exclude_sky_targetids (array of int): TARGETIDs to exclude
- Returns:
array of indices of sky fibers to use
By default we rely on fibermap[‘OBJTYPE’]==’SKY’, but we can also exclude some targetids by providing a list of them through exclude_sky_targetids or by just providing all the sky targetids directly (in that case the OBJTYPE information is ignored)
- desispec.sky.qa_skysub(param, frame, skymodel, quick_look=False)[source]¶
Calculate QA on SkySubtraction
Note: Pixels rejected in generating the SkyModel (as above), are not rejected in the stats calculated here. Would need to carry along current_ivar to do so.
- Parameters:
param – dict of QA parameters : see qa_frame.init_skysub for example
frame – desispec.Frame object; Should have been flat fielded
skymodel – desispec.SkyModel object
quick_look – bool, optional If True, do QuickLook specific QA (or avoid some)
- Returns:
- dict of QA outputs
Need to record simple Python objects for yaml (str, float, int)
- Return type:
qadict
- desispec.sky.subtract_sky(frame, skymodel, apply_throughput_correction_to_lines=True, apply_throughput_correction=False, zero_ivar=True)[source]¶
Subtract skymodel from frame, altering frame.flux, .ivar, and .mask
- Parameters:
frame – desispec.Frame object
skymodel – desispec.SkyModel object
- Option:
- apply_throughput_correctionif True, fit for an achromatic throughput
correction. This is to absorb variations of Focal Ratio Degradation with fiber flexure. This applies the residual throughput corrections on top of the model throughput corrections already included in the sky model.
zero_ivar : if True , set ivar=0 for masked pixels
desispec.skycorr¶
Class with sky model corrections.
desispec.skygradpca¶
Class with sky model gradient templates.
- desispec.skygradpca.compute_pcs(skies, topn=6, niter=5)[source]¶
Computes principal components of sky spectra.
This computes the PCs used for the skygradpca product.
- Parameters:
skies – output of gather skies; the sky spectra to do PCA on
topn – (optional int) number of PCA to compute
niter – (optional int) number of iterations to do, rejecting discrepant pixels.
- Returns:
Output of np.linalg.svd on the clipped, mean-subtracted sky spectra.
- desispec.skygradpca.configure_for_xyr(skygradpca, x, y, R, skyfibers=None)[source]¶
Add additional information to a SkyGradPCA object for fitting.
A SkyGradPCA object contains the templates used to fit out sky gradients. When these templates are actually used in fitting, it’s helpful to additionally know things like the locations of the fibers on the particular exposure and the mean resolution matrix for the exposure, etc. This function mutates the skygradpca object, adding new attributes for fitting.
- Parameters:
skygradpca – SkyGradPCA object to extend
x – 1D[nfiber] array of x fiber coordinates on exposure (mm)
y – 1D[nfiber] array of y fiber coordinates on exposure (mm)
R – list of resolution matrices for each fiber (unitless)
skyfibers – (optional) indices of sky fibers in list of all fibers
- desispec.skygradpca.doall(specprod='fuji')[source]¶
Perfoms and writes out analysis of sframe sky residuals.
Gathers all of the sframe residual sky spectra for each camera/petal, runs a PCA on them, makes the appropriate SkyGradPCA objects, and writes them out to DESI_SPECTRO_CALIB.
- Parameters:
specprod (str, optional) – specprod to run on
- desispec.skygradpca.evaluate_model(skygradpca, R, coeff, mean=None)[source]¶
Evaluate a skygradpca object for a frame.
The skygradpca object must first be configured for the frame with configure_for_xyr.
- Parameters:
skygradpca – SkyGradPCA instance, configured with configure_for_xyr
R – resolution matrices for frame
coeff (array, 1D) – coefficients for eigenvectors
mean (optional) – mean sky spectrum
- Returns:
sky spectrum (array[nfiber, nwave]) given mean spectrum, per-fiber resolution matrices, and gradient amplitudes
- desispec.skygradpca.gather_skies(fn=None, camera='r', petal=0, n=numpy.inf, heliocor=True, specprod='daily')[source]¶
Gather sframes for performing bulk analyses; e.g., PCA.
This function gathers residual sky spectra together for many exposures, bundling it together for later analysis.
- Parameters:
fn – (list[str], optional) sframe file names to gather If not set, all of the sframes in the specprod product in the camera camera and petal petal will be gathered.
camera – (str, optional) camera to gather
petal – (int, optional) petal to gather
n – (int, optional) only gather at most this many exposures worth of sky spectra
heliocor – (bool, optional) if True, apply correction to sframe files to bring them to earth frame
- Returns:
ndarray with EXPID, FIBER, CAMERA, X, Y, WAVE, FLUX, IVAR for each sky fiber from sframe files.
- desispec.skygradpca.make_all_pcs(specprod, minnight=20200101, cameras='brz', petals=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], **kw)[source]¶
Makes SkyGradPCA objects derived from residual skies from a specprod.
This grabs all of the sframe files passing certain criteria using gather_skies and runs a PCA on them, producing SkyGradPCA objects for each camera/petal. It uses 30 processes, one for each camera/petal combination.
- Parameters:
- Returns:
dictionary[camera, petal] containing the corresponding SkyGradPCA object.
- desispec.skygradpca.make_all_pcs_by_filter(**kw)[source]¶
Wrapper for make_all_pcs, doing one filter at a time.
- desispec.skygradpca.make_all_pcs_wrapper(args)[source]¶
Wrapper function; gathers skies for cam/petal combinations.
- desispec.skygradpca.write_pcs(pcs)[source]¶
Writes SkyGradPCA objects to DESI_SPECTRO_CALIB.
Takes the output of make_all_pcs and writes it to DESI_SPECTRO_CALIB.
- Parameters:
pcs (dict[camera, petal] of SkyGradPCA) – output of make_all_pcs
desispec.skymag¶
Utility function to compute the sky magnitude per arcmin2 based from the measured sky model of an exposure and a static model of the instrument throughput.
- desispec.skymag._get_average_calibration(filename)[source]¶
Use a dictionnary referenced by a global variable to keep a copy of the calibration instead of reading it at each function call.
- desispec.skymag.compute_skymag(night, expid, specprod_dir=None)[source]¶
Computes the sky magnitude for a given exposure. Uses the sky model and apply a fixed calibration for which the fiber aperture loss is well understood.
- Parameters:
night – int, YYYYMMDD
expid – int, exposure id
specprod_dir – str, optional, specify the production directory. default is $DESI_SPECTRO_REDUX/$SPECPROD
- Returns:
(gmag,rmag,zmag) AB magnitudes per arcsec2, tuple with 3 float values
desispec.specscore¶
Spectral scores routines.
- desispec.specscore.compute_coadd_scores(coadd, specscores=None, update_coadd=True)[source]¶
Compute scores for a coadded Spectra object
- Parameters:
coadd – a Spectra object from a coadd
- Options:
update_coadd: if True, update coadd.scores specscores: scores Table from the uncoadded spectra including a TARGETID column
Returns tuple of dictionaries (scores, comments); see compute_frame_scores
specscores
is used to update TSNR2 scores by summing inputs
- desispec.specscore.compute_coadd_tsnr_scores(specscores)[source]¶
Compute coadded TSNR2 scores (TSNR2=Template Signal-to-Noise squared)
- Parameters:
specscores – uncoadded scores with TSNR2* columns (dict or Table-like)
Returns (tsnrscores, comments) tuple of dictionaries
- desispec.specscore.compute_frame_scores(frame, band=None, suffix=None, flux_per_angstrom=None)[source]¶
Computes scores in spectra of a frame.
The scores are sum,mean,medians in a predefined and fixed wavelength range for each DESI camera arm, or band, b, r or z. The band argument is optional because it can be automatically chosen from the wavelength range in the frame. The suffix is added to the key name in the output dictionnary, for instance ‘RAW’, ‘SKYSUB’, ‘CALIB’ … The boolean argument flux_per_angstrom is needed if there is no ‘BUNIT’ keyword in frame.meta (frame fits header)
- Parameters:
frame (
Frame
orQFrame
) – A Frame or a QFrame object.band (
str
, optional) – Spectrograph band,b
,r
,z
, autodetected by default.suffix (
str
, optional) – Character string added to the keywords in the output dictionary, for instance suffix=’RAW’flux_per_angstrom (
bool
, optional) – IfTrue
the spectra are assumed flux_per_angstrom, i.e. flux densities. IfFalse
, the spectra are assumed to be counts or photo-electrons per bin.None
by default in which case theframe.units
string is read to find out whether the flux quantity is per unit wavelenght or per bin.
- Returns:
A tuple containg a
dict
of 1D arrays of size = number of spectra in frame and adict
of string with comments on the type of scores.- Return type:
tuple()
desispec.specstatus¶
Update surveyops/ops/tile-specstatus.ecsv with spectro pipeline tiles.csv.
- desispec.specstatus.update_specstatus(specstatus, tiles, update_only=False, clear_qa=False)[source]¶
return new specstatus table, updated with tiles table
- Parameters:
specstatus – astropy Table from surveyops/ops/tiles-specstatus.ecsv
tiles – astropy Table from spectro/redux/daily/tiles.csv
clear_qa – bool indicating whether QA data should be cleared
Returns: updated specstatus table, sorted by TILEID
New TILEID found in tiles are added to specstatus, and any TILEID already in specstatus have their non-QA columns updated from the entries in tiles.
If update_only==True, add any new tiles and update non-QA columns for tiles with new data (tiles[‘LASTNIGHT’] > specstatus[‘LASTNIGHT’]) but don’t change entries for reprocessed tiles that have no new data.
The QA-related columns that are not changed here are USER, QA, OVERRIDE, ZDONE, QANIGHT, and ARCHIVEDATE.
This does not modify either of the input tables.
desispec.spectra¶
Class for dealing with a group of spectra from multiple bands and the associated fibermap information.
- class desispec.spectra.Spectra(bands=[], wave={}, flux={}, ivar={}, mask=None, resolution_data=None, fibermap=None, exp_fibermap=None, meta=None, extra=None, single=False, scores=None, scores_comments=None, extra_catalog=None)[source]¶
Represents a grouping of spectra.
This class contains an “extended” fibermap that has information about the night and exposure of each spectrum. For each band, this class has the wavelength grid, flux, ivar, mask, and resolution arrays.
- Parameters:
bands (
list
) – List of strings used to identify the bands.wave (
dict
) – Dictionary of arrays specifying the wavelength grid.flux (
dict
) – Dictionary of arrays specifying the flux for each spectrum.ivar (
dict
) – Dictionary of arrays specifying the inverse variance.mask (
dict
, optional) – Dictionary of arrays specifying the bitmask.resolution_data (
dict
, optional) – Dictionary of arrays specifying the block diagonal resolution matrix. The object for each band must be in one of the formats supported by the Resolution class constructor.fibermap – Extended fibermap to use. If not specified, a fake one is created.
Table-like – Extended fibermap to use. If not specified, a fake one is created.
optional – Extended fibermap to use. If not specified, a fake one is created.
exp_fibermap – Exposure-specific fibermap columns, which may not apply to a coadd.
Table-like – Exposure-specific fibermap columns, which may not apply to a coadd.
optional – Exposure-specific fibermap columns, which may not apply to a coadd.
meta (
dict
, optional) – Dictionary of arbitrary properties.extra (
dict
, optional) – Optional dictionary of dictionaries containing extra floating point arrays. The top-level is a dictionary over bands and each value is a dictionary containing string keys and values which are arrays of the same size as the flux array.single (
bool
, optional) – IfTrue
, store flux,ivar,resolution data in memory as single precision (np.float32).scores – QA scores table.
scores_comments – dict[column] = comment to include in output file
extra_catalog (numpy or astropy Table, optional) – optional table of metadata, rowmatched to fibermap, e.g. a redshift catalog for these spectra
- _get_slice(index, bands=None)[source]¶
Slice spectra by index. :param bands: optional list of bands to select. :type bands: list
- Note: This function is intended to be private,
to be used by __getitem__() and select().
- classmethod from_specutils(spectra)[source]¶
Convert
specutils
objects to aSpectra
object.- Parameters:
spectra (specutils.Spectrum1D or specutils.SpectrumList) – A
specutils
object.- Returns:
The corresponding DESI-internal object.
- Return type:
- Raises:
NameError – If
specutils
is not available in the environment.ValueError – If an unknown type is found in spectra.
- property ftype¶
the data type used for floating point numbers.
- Type:
- num_spectra()[source]¶
Get the number of spectra contained in this group.
- Returns (int):
Number of spectra contained in this group.
- num_targets()[source]¶
Get the number of distinct targets.
- Returns (int):
Number of unique targets with spectra in this object.
- select(nights=None, exposures=None, bands=None, targets=None, fibers=None, invert=False, return_index=False)[source]¶
Select a subset of the data.
This filters the data based on a logical AND of the different criteria, optionally inverting that selection.
- Parameters:
nights (list) – optional list of nights to select.
exposures (list) – optional list of exposures to select.
bands (list) – optional list of bands to select.
targets (list) – optional list of target IDs to select.
fibers (list) – list/array of fiber indices to select.
invert (bool) – after combining all criteria, invert selection.
return_index (bool) – if True, also return the indices of selected spectra.
- Returns:
a new Spectra object containing the selected data. indices (list, optional): indices of selected spectra. Only provided if return_index is True.
- Return type:
spectra
- target_ids()[source]¶
Return list of unique target IDs.
The target IDs are sorted by the order that they first appear.
- Returns (array):
an array of integer target IDs.
- to_specutils()[source]¶
Convert to
specutils
objects.- Returns:
A list with each band represented as a specutils.Spectrum1D object.
- Return type:
specutils.SpectrumList
- Raises:
NameError – If
specutils
is not available in the environment.
- update(other)[source]¶
Overwrite or append new data.
Given another Spectra object, compare the fibermap information with the existing one. For spectra that already exist, overwrite existing data with the new values. For spectra that do not exist, append that data to the end of the spectral data.
- Parameters:
other (Spectra) – the new data to add.
- Returns:
nothing (object updated in place).
Note: if fibermap, scores and extra_catalog exist in the new data, they are appended to the existing tables. If those new tables have different columns, only columns with identical names will be appended. Spectra.meta is unchanged.
- desispec.spectra.stack(speclist)[source]¶
Stack a list of spectra, return a new spectra object
- Parameters:
speclist – list of Spectra objects
returns stacked Spectra object
Note: all input spectra must have the same bands, wavelength grid, and include or not the same optional elements (mask, fibermap, extra, …). The returned Spectra have the meta from the first input Spectra.
Also see Spectra.update, which is less efficient but more flexible for handling heterogeneous inputs
desispec.tile_qa¶
Utility functions to compute an exposure QA scores (or is it tile?).
- desispec.tile_qa.compute_tile_qa(night, tileid, specprod_dir, exposure_qa_dir=None, group='cumulative')[source]¶
Computes the exposure_qa
- Parameters:
- Returns:
A tuple of astropy.table.Table:
fiberqa (with one row per target and at least a TARGETID column)
petalqa (with one row per petal and at least a PETAL_LOC column)
- Return type:
desispec.tile_qa_plot¶
Utility functions to generate the tile QA png.
- desispec.tile_qa_plot.assert_tracer(tracer)[source]¶
Asserts tracer in the list of allowed tracers.
- Parameters:
tracer – tracer name (string, upper case)
Note
Will exit with an error if not asserted.
- desispec.tile_qa_plot.deg2pix(dras, ddecs, width_deg, width_pix)[source]¶
Converts (dras,ddecs) to (xs,ys) in cutout img pixels.
- Parameters:
dras – projected distance (degrees) along R.A. to the center of the cutout (np.array of floats)
ddecs – projected distance (degrees) along Dec. to the center of the cutout (np.array of floats)
width_deg – width of the cutout in degrees (np.array of floats)
width_pix – width of the cutout in pixels (np.array of floats)
- Returns:
A tuple containing:
dxs: distance (pixels) along x to the center of the cutout (np.array of floats)
dys: distance (pixels) along y to the center of the cutout (np.array of floats)
- Return type:
Notes
not sure at the <1 pixel level…
Duplicated from fiberassign.fba_launch_io.deg2pix()
- desispec.tile_qa_plot.get_expid_vccdsec(expid, night, prod)[source]¶
Get the VCCDSEC header keyword for an exposure for each frame file.
- Parameters:
expid – exposure id (int)
night – night (int)
prod – full path to input reduction, e.g. /global/cfs/cdirs/desi/spectro/redux/daily (string)
- Returns:
dictionary with VCCDSEC value for each frame file (dict. of floats) pb_campets: list of {camera}{petal} which have VCCDSEC < threshold
- Return type:
vccdsec
Notes
The VCCDSEC threshold is set in desispec/data/qa/qa-params.yaml
- If VCCDSEC is not present for a given frame file, returns -99s
(so that it gets attention)
If no frame files, returns an empty dictionary
- desispec.tile_qa_plot.get_expids_efftimes(tileqafits, prod)[source]¶
Get the EFFTIME and EFFTIMEQA for the EXPIDs from the coadd.
- Parameters:
tileqafits – path to the tile-qa-TILEID-NIGHT.fits file
prod – full path to input reduction, e.g. /global/cfs/cdirs/desi/spectro/redux/daily (string)
- Returns:
An array with the following keys:
EXPID, NIGHT, EFFTIME_SPEC, QA_EFFTIME_SPEC
- Return type:
structured array
Notes
We work from the
spectra-*.fits
files; if not present in the same folder as tileqafits, we look into the expected path using prod.As this is run before desi_tsnr_afterburner, we compute here the EFFTIME_SPEC values.
If no GOALTYPE in tileqafits header, we default to dark.
TBD: we purposely do not use TSNR2 keys from qa-params.yaml, as those do not handle the TSNR2_ELG->TSNR2_LRG change from 2021 shutdown. We use:
dark before 20210901: TSNR2_ELG
dark after 20210901: TSNR2_LRG
bright: TSNR2_BGS
backup: TSNR2_BGS
Method assessed against all Main exposures until 20211013 in daily tsnr-exposures.fits.
- desispec.tile_qa_plot.get_petalqa_props(key)[source]¶
For a given key, returns specific properties, to display diagnoses on the QA plot.
- Parameters:
key – column name present in the PETALQA extension of qa-tile-TILEID-NIGHT.fits (string)
- Returns:
A tuple containing:
short: shortname of the key (string)
precision: number of digits to report (int)
okmin: minimum value to be considered as ok (float)
okmax: maximum value to be considered as ok (float)
combine: None, “sum”, “mean”, i.e. how to combine the stats for all petals (string)
- Return type:
- desispec.tile_qa_plot.get_qa_badmsks()[source]¶
Returns the bitmask values for bad_qafstatus_mask and bad_petal_mask..
- Parameters:
None –
- Returns:
The bitmask values for bad_qafstatus_mask and bad_petal_mask.
- desispec.tile_qa_plot.get_qa_config()[source]¶
Reads the configuration file (data/qa/qa-params.yaml)
- Parameters:
None. –
- Returns:
Content of the qa-params.yaml file
- desispec.tile_qa_plot.get_quantz_cmap(name, n, cmin=0, cmax=1)[source]¶
Creates a quantized colormap.
- Parameters:
name – matplotlib colormap name (e.g. “tab20”) (string)
n – number of colors
cmin (optional, defaults to 0) – first color of the original colormap to use (between 0 and 1) (float)
cmax (optional, defaults to 1) – last color of the original colormap to use (between 0 and 1) (float)
- Returns:
A matplotlib cmap object.
Notes
- desispec.tile_qa_plot.get_radec_mw(ras, decs, org)[source]¶
Converts R.A. and Dec. coordinates to be used in a Mollweide projection.
- Parameters:
ras – R.A. coordinates in degrees (float array)
decs – Dec. coordinates in degrees (float array)
- Returns:
Two float arrays with transformed R.A. and Dec.
- desispec.tile_qa_plot.get_tilecov(tileid, surveys='main', programs=None, lastnight=None, indesi=True, nside=1024, outpng=None, plot_tiles=False, verbose=False)[source]¶
Computes the average number of observed tiles covering a given tile.
- Parameters:
tileid (int) – tileid
surveys (optional, defaults to "main") – comma-separated list of surveys to consider (reads the tiles-SURVEY.ecsv file) (str)
programs (optional, defaults to None) – comma-separated list of programs (case-sensitive) to consider in the tiles-SURVEY.ecsv file (str)
lastnight (optional, defaults to today) – only consider tiles observed up to lastnight (int)
surveys – comma-separated list of surveys to consider (reads the tiles-SURVEY.ecsv file) (str)
indesi (optional, defaults to True) – restrict to IN_DESI=True tiles? (bool)
nside (optional, defaults to 1024) – healpix pixel nside (int)
outpng (optional, defaults to None) – if provided, output file with a plot (str)
plot_tiles (optional, defaults to False) – plot overlapping tiles? (bool)
verbose (optional, defaults to False) – print log.info() (bool)
- Returns:
A tuple containing:
pixs: list of the healpix pixels covering the tile (np.array(float))
pix_ntiles: list of the nb of tiles covering each pixel from pixs (np.array(float))
nside: healpix pixel nside (int)
nest: healpix NESTED? (boolean)
outdict: a dictionary, with an entry for each observed, overlapping tile, containing the list of observed overlapping tiles (dict)
- Return type:
Notes
The “regular” use is to provide a single PROGRAM in programs (e.g., programs=”DARK”).
This function relies on the following files:
$DESI_SURVEYOPS/ops/tiles-{SURVEY}.ecsv for SURVEY in surveys (to get the tiles to consider)
$DESI_ROOT/spectro/redux/daily/exposures-daily.fits (to get the existing observations up to lastnight)
If one wants to consider the latest observations, one should wait the 10am pacific update of exposures-daily.fits.
- desispec.tile_qa_plot.get_tracer(tracer, d, fstatus_key='QAFIBERSTATUS')[source]¶
- For a given tracer, returns the selection used for the tile QA n(z):
(fstatus_key & bad_qafstatus_mask) == 0
desi_mask or bgs_mask
if ELG_LOP: additional cut on (d[“DESI_TARGET”] & desi_mask[“QSO”]) == 0
- Parameters:
tracer – “BGS_BRIGHT”, “BGS_FAINT”, “LRG”, “ELG_LOP”, or “QSO” (string)
d – structured array with at least FIBERSTATUS, DESI_TARGET, BGS_TARGET
fstatus_key (optional, defaults to QAFIBERSTATUS) – key to use as FIBERSTATUS (string)
- Returns:
selected tracer sample (boolean array)
- Return type:
sel
- desispec.tile_qa_plot.get_tracer_zminmax(tracer)[source]¶
Returns some fiducial redshift range per tracer to compute basic stats.
- Parameters:
tracer – tracer name (see get_tracer_names()) (string)
- Returns:
A tuple containing:
zmin: minimum redshift (float)
zmax: minimum redshift (float)
- Return type:
- desispec.tile_qa_plot.get_tracer_zok(tracer, dchi2_min, d, fstatus_key='QAFIBERSTATUS')[source]¶
- For a given tracer, returns the spectro. valid sample for the tile QA n(z):
(fstatus_key & bad_qafstatus_mask) == 0
DELTACHI2 > dchi2_min
if QSO: additional cut on SPECTYPE=”QSO”
- Parameters:
tracer – “BGS_BRIGHT”, “BGS_FAINT”, “LRG”, “ELG_LOP”, or “QSO” (string)
dchi2_min – minimum DELTACHI2 value (read from YAML file)
d – structured array with at least DELTACHI2, SPECTYPE
fstatus_key (optional, defaults to QAFIBERSTATUS) – key to use as FIBERSTATUS (string)
- Returns:
spectro. valid sample (boolean array)
- Return type:
sel
- desispec.tile_qa_plot.get_viewer_cutout(tileid, tilera, tiledec, tmpoutdir='/tmp/tmpqw_r0v8g', width_deg=4, pixscale=10, dr='dr9', timeout=15)[source]¶
Downloads a cutout of the tile region from legacysurvey.org/viewer.
- Parameters:
tileid – TILEID (int)
tilera – tile center R.A. (float)
tiledec – tile center Dec. (float)
tmpoutdir (optional, defaults to a temporary directory) – temporary directory where
width_deg (optional, defaults to 4) – width of the cutout in degrees (float)
pixscale (optional, defaults to 10) – pixel scale of the cutout
dr (optional, default do "dr9") – imaging data release
timeout (optional, defaults to 15) – time (in seconds) after which we quit the wget call (int)
- Returns:
output of mpimg.imread() reading of the cutout (np.array of floats)
- Return type:
img
Notes
Duplicating fiberassign.fba_launch_io.get_viewer_cutout()
20220109 : adding a check on img dimension..
- desispec.tile_qa_plot.get_zbins()[source]¶
Redshift grid for the tile QA n(z)
- Parameters:
None –
- Returns:
The redshift grid (float array)
- desispec.tile_qa_plot.get_zhists(tileids, tracer, dchi2_min, d, fstatus_key='QAFIBERSTATUS', tileid_key=None)[source]¶
Returns the fractional, per tileid, n(z) for a given tracer.
- Parameters:
tileids – int or list or numpy array of int
tracer – one of the tracers defined in get_tracer_names() (string)
dchi2_min – DELTACHI2 threshold for a valid redshift
d – structured array with at least FIBERSTATUS, DESI_TARGET, BGS_TARGET, DELTACHI2, SPECTYPE, Z
fstatus_key (optional, defaults to QAFIBERSTATUS) – key to use as FIBERSTATUS (string)
tileid_key (optional, defaults to None) – column name for TILEID (string)
- Returns:
A tuple containing:
bins: the redshift bin grid (float array)
zhists: fractional, per tileid, n(z) (numpy array of shape (nbin) if tileids is int or has length=1 (nbin, len(tileids)) else.
- Return type:
Notes
If tileid_key is not provided, assumes all spectra are from the same tile.
If tileid_key is provided, will identify the tileids with np.unique(d[tileid_key]).
- desispec.tile_qa_plot.make_tile_qa_plot(tileqafits, prod, pngoutfile=None, dchi2_min=None, tsnr2_key=None, refdir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/desispec/checkouts/stable/py/desispec/data/qa'))[source]¶
Generate the tile QA png file. Will replace .fits by .png in tileqafits for the png filename.
- Parameters:
tileqafits – path to the tile-qa-TILEID-NIGHT.fits file
prod – full path to input reduction, e.g. /global/cfs/cdirs/desi/spectro/redux/daily (string)
pngoutfile (optional) – output filename; default to tileqafits .fits -> .png
dchi2_min (optional, defaults to value in qa-params.yaml) – minimum DELTACHI2 for a valid zspec (float)
tsnr2_key (optional, defaults to value in qa-params.yaml) – TSNR2 key used for plot (string)
refdir (optional, defaults to "desispec","data/qa") – path to folder with reference measurements for the n(z) and the TSNR2 (string)
Note
If hdr[“SURVEY”] is not “main”, will not plot the n(z).
If hdr[“FAPRGRM”].lower() is not “bright” or “dark”, will not plot the TSNR2 plot nor the skymap.
20220109 : add safety around plot_cutout() call.
- desispec.tile_qa_plot.plot_cutout(ax, tileid, tilera, tiledec, width_deg, petal_c='w', ebv_c='orange')[source]¶
Plots a ls-dr9 cutout, with overlaying the petals and the EBV contours.
- Parameters:
ax – pyplot object
tileid – TILEID (int)
tilera – tile center R.A. (float)
tiledec – tile center Dec. (float)
width_deg – width of the cutout in degrees (np.array of floats)
petal_c (optional, defaults to "w") – color used to display petals (string)
ebv_c (optional, default to "y") – color used to display the EBV contours (string)
Notes
Different than fiberassign.fba_launch_io.plot_cutout().
- desispec.tile_qa_plot.plot_mw_skymap(fig, ax, tileid, tilera, tiledec, survey, program, org=120)[source]¶
Plots the target density sky map with a Mollweide projection, with highlighting the tile position.
- Parameters:
fig – pyplot figure object
ax – pyplot object with projection=”mollweide”
tileid – TILEID (int)
tilera – tile R.A. in degrees (float)
tiledec – tile Dec. in degrees (float)
survey – usually “main”, “sv1”, “sv3” (string)
program – “dark” or “bright” (string)
org – R.A. at the center of the plot (default=120) (float)
Note
If survey is not “main” and program is not “bright” or “dark”, will color-code EBV, not the target density.
- desispec.tile_qa_plot.print_petal_infos(ax, petalqa, fiberqa)[source]¶
Print some diagnoses for each petal, and for the whole tile.
- Parameters:
ax – pyplot object
petalqa – the PETALQA extension data of the tile-qa-TILEID-NIGHT.fits file
- desispec.tile_qa_plot.set_mwd(ax, org=120)[source]¶
Prepare a plot for a Mollweide map
- Parameters:
ax – a plt.subplots() instance
org – R.A. at the center of the plot (default=120) (float)
desispec.tilecompleteness¶
Routines to determine the survey progress and tiles completion.
- desispec.tilecompleteness.compute_tile_completeness_table(exposure_table, specprod_dir, auxiliary_table_filenames, min_number_of_petals=8)[source]¶
Computes a summary table of the observed tiles
- Parameters:
exposure_table – astropy.table.Table with exposure summary table from a prod (output of desi_tsnr_afterburner)
specprod_dir – str, production directory name
auxiliary_table_filenames – list(str), list of auxiliary table names, optional
min_number_of_petals – int, minimum number of petals to declare a tile done
Returns: astropy.table.Table with one row per TILEID, with completeness and exposure time.
- desispec.tilecompleteness.merge_tile_completeness_table(previous_table, new_table)[source]¶
Merges tile summary tables.
- Parameters:
previous_table – astropy.table.Table
new_table – astropy.table.Table
Returns: astropy.table.Table with merged entries.
desispec.tpcorrparam¶
This module implements a model for how the throughput of a fiber varies.
It has the following components:
We find that the fibers don’t follow the nighly flat field exactly, and apply a fixed mean correction from the nightly flat field. This is thought to reflect the different range of angles entering the fiber from the flat field relative to the sky.
We find that fibers’ throughput depends modestly on the fibers’ location within their patrol radii. We apply a cubic polynomial in (x, y) within the patrol disk to correct for this.
We find additional coherent variations in throughput from exposure to exposure. These are modeled with a PCA and fit exposure-by-exposure.
The TPCorrParam object has mean, spatial, and pca attributes describing each of these three components.
- desispec.tpcorrparam.cubic(p, x, y)[source]¶
Returns a cubic polynomial with coefficients p in x & y.
- desispec.tpcorrparam.fit_tpcorr_per_fiber(tpcorr)[source]¶
Fit the spatial throughput variations for each fiber.
- Parameters:
tpcorr (ndarray) – output from gather_tpcorr with tpcorr measurements and related metadata for each fiber
- Returns:
X (float): central X position of each fiber used in fit (mm) Y (float): central Y position of each fiber used in fit (mm) FIBER (int): fiber number PARAM (ndarray[float], length 10): 2D cubic polynomial coefficients
- Return type:
dict[camera] of ndarray including the following fields
- desispec.tpcorrparam.gather_dark_tpcorr(specprod=None)[source]¶
Gathers TPCORR for dark exposures expected to be well behaved.
This wraps gather_tpcorr, making a sensible selection of dark exposures from the given specprod.
- Parameters:
specprod (str) – specprod to use
- Returns:
ndarray from gather_tpcorr with tpcorr information from selected exposures.
- desispec.tpcorrparam.gather_tpcorr(recs)[source]¶
Gather TPCORR measurements for exposures.
- Parameters:
recs (ndarray) – ndarray array containing information about exposures from which to gather TPCORR information. Includes at least the following fields: NIGHT (int): night on which exposure was observed EXPID (int): exposure id
- Returns: ndarray with the following fields for each exposure
X: x coordinate of fiber (mm) Y: y coordinate of fiber (mm) TPCORR: tpcorr of fiber CAMERA: camera for exposure NIGHT: night for exposure EXPID: expid of exposure
- desispec.tpcorrparam.make_tpcorrparam_files(tpcorr=None, spatial=None, dirname=None)[source]¶
Gather, fit, and write tpcorrparam files to output directory.
This function is intended for producing tpcorrparam files to DESI_SPECTRO_CALIB. It can gather and fit tpcorr measurements and write the results.
- Parameters:
tpcorr (optional) – result of gather_tpcorr. If None, this is populated with the result of gather_dark_tpcorr.
spatial (optional) – spatial fit results. If None, this is populated with fit_tpcorr_per_fiber(tpcorr).
dirname (optional) – directory to which to write files. Defaults to DESI_SPECTRO_CALIB.
- desispec.tpcorrparam.pca_tpcorr(tpcorr)[source]¶
Run a PCA on TPCORR measurements.
This uses the tpcorr gathered by gather_tpcorr and runs a PCA on them.
- Parameters:
tpcorr (ndarray) – result of gather_tpcorr
- Returns:
A dictionary containing:
dict[filter] of (tpcorrmed, pc_info, exp_info), describing the results of the fit.
tpcorrmed: the median TPCORR of the whole sample; i.e., how different the nightly flat usually is from the sky.
pc_info: ndarray with:
pca: the principal components
amplitude: their corresponding singular values
exp_info: ndarray with:
expid: exposure id
coeff: pc component coefficients for this exposure
- Return type:
- desispec.tpcorrparam.tpcorrmodel(tpcorrparam, xfib, yfib, pcacoeff=None)[source]¶
Evaluates full TPCORR throughput model.
The full model consistents of a PCA and a spatial within-patrol-radius component. These two components are evaluated and summed.
- Parameters:
tpcorrparam – TPCorrParam object describing the model
xfib – x coordinates of fibers (mm)
yfib – y coordinates of fibers (mm)
pcacoeff (optional) – PCA coefficients of throughput model If not set, no TPCORR PCA terms are fit.
- Returns:
Throughput model evaluated at xfib, yfib with pca coefficients pcacoeff.
- desispec.tpcorrparam.tpcorrspatialmodel(param, xfib, yfib)[source]¶
Evaluate spatial tpcorr throughput model at xfib, yfib.
Evaluates the throughput model at xfib, yfib. param carries the polynomial coefficients of the model as well as the central fiber location.
- Parameters:
param – ndarray describing fit. Contains at least the following fields: PARAM: cubic polynomial coefficients X: central fiber X location (mm) Y: central fiber Y location (mm)
xfib – x location at which model is to be evaluated (mm)
yfib – y location at which model is to be evaluated (mm)
- Returns:
throughput model evaluated at xfib, yfib
desispec.trace_shifts¶
- desispec.trace_shifts.boxcar_extraction_from_filenames(image_filename, psf_filename, fibers=None, width=7)[source]¶
Fast boxcar extraction of spectra from a preprocessed image and a trace set
- Parameters:
image_filename – input preprocessed fits filename
psf_filename – input PSF fits filename
- Optional:
fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : extraction boxcar width, default is 7
- Returns:
2D np.array of shape (nfibers,n0=image.shape[0]), sum of pixel values per row of length=width per fiber ivar : 2D np.array of shape (nfibers,n0), ivar[f,j] = 1/( sum_[j,b:e] (1/image.ivar) ), ivar=0 if at least 1 pixel in the row has image.ivar=0 or image.mask!=0 wave : 2D np.array of shape (nfibers,n0), determined from the traces
- Return type:
flux
- desispec.trace_shifts.compute_dx_dy_using_psf(psf, image, fibers, lines)[source]¶
Computes trace shifts along x and y from a preprocessed image, a PSF (with trace coords), and a set of emission lines, by doing a forward model of the image. Calls compute_fiber_bundle_trace_shifts_using_psf.
- Parameters:
psf – specter psf object
image – DESI preprocessed image object
fibers – 1D array with list of fibers
lines – 1D array of wavelength of emission lines (in Angstrom)
- Returns:
1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD dy : 1D array of shifts along y coordinates on CCD sx : 1D array of uncertainties on dx sy : 1D array of uncertainties on dy fiber : 1D array of fiber ID wave : 1D array of wavelength
- Return type:
x
- desispec.trace_shifts.compute_dx_from_cross_dispersion_profiles(xcoef, ycoef, wavemin, wavemax, image, fibers, width=7, deg=2, image_rebin=4)[source]¶
Measure x offsets from a preprocessed image and a trace set
- Parameters:
xcoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to XCCD
ycoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to YCCD
wavemin – float
wavemax – float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1 used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])
image – DESI preprocessed image object
fibers – 1D np.array of int (default is all fibers, the first fiber is always = 0)
- Optional:
width : extraction boxcar width, default is 5 deg : degree of polynomial fit as a function of y, only used to find and mask outliers image_rebin : rebinning of CCD rows to run faster (with rebin=4 loss of precision <0.01 pixel)
- Returns:
1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD ex : 1D array of uncertainties on dx fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength
- Return type:
x
- desispec.trace_shifts.compute_dy_from_spectral_cross_correlation(flux, wave, refflux, ivar=None, hw=3.0, calibrate=False, prior_width_dy=None)[source]¶
Measure y offsets from two spectra expected to be on the same wavelength grid. refflux is the assumed well calibrated spectrum. A relative flux calibration of the two spectra is done internally.
- Parameters:
flux – 1D array of spectral flux as a function of wavelenght
wave – 1D array of wavelength (in Angstrom)
refflux – 1D array of reference spectral flux
- Optional:
ivar : 1D array of inverse variance of flux hw : half width in Angstrom of the cross-correlation chi2 scan, default=3A corresponding approximatly to 5 pixels for DESI prior_width_dy: applied sigma of the Gaussian prior on dy
- Returns:
1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD ex : 1D array of uncertainties on dx fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength
- Return type:
x
- desispec.trace_shifts.compute_dy_from_spectral_cross_correlations_of_frame(flux, ivar, wave, xcoef, ycoef, wavemin, wavemax, reference_flux, n_wavelength_bins=4)[source]¶
Measures y offsets from a set of resampled spectra and a reference spectrum that are on the same wavelength grid. reference_flux is the assumed well calibrated spectrum. Calls compute_dy_from_spectral_cross_correlation per fiber
- Parameters:
flux – 2D np.array of shape (nfibers,nwave)
ivar – 2D np.array of shape (nfibers,nwave) , inverse variance of flux
wave – 1D array of wavelength (in Angstrom) of size nwave
refflux – 1D array of reference spectral flux of size nwave
- Optional:
n_wavelength_bins : number of bins along wavelength
- Returns:
1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dy : 1D array of shifts along y coordinates on CCD ey : 1D array of uncertainties on dy fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength
- Return type:
x
- desispec.trace_shifts.compute_dy_using_boxcar_extraction(xytraceset, image, fibers, width=7, degyy=2)[source]¶
Measures y offsets (internal wavelength calibration) from a preprocessed image and a trace set using a cross-correlation of boxcar extracted spectra. Uses boxcar_extraction , resample_boxcar_frame , compute_dy_from_spectral_cross_correlations_of_frame
- Parameters:
xytraceset – XYTraceset object
image – DESI preprocessed image object
- Optional:
fibers : 1D np.array of int (default is all fibers, the first fiber is always = 0) width : int, extraction boxcar width, default is 7 degyy : int, degree of polynomial fit of shifts as a function of y, used to reject outliers.
- Returns:
1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dy : 1D array of shifts along y coordinates on CCD ey : 1D array of uncertainties on dy fiber : 1D array of fiber ID (first fiber = 0) wave : 1D array of wavelength
- Return type:
x
- desispec.trace_shifts.compute_fiber_bundle_trace_shifts_using_psf(fibers, line, psf, image, maxshift=2.0)[source]¶
Computes trace shifts along x and y from a preprocessed image, a PSF (with trace coords), and a given emission line, by doing a forward model of the image.
- Parameters:
fibers – 1D array with list of fibers
line – float, wavelength of an emission line (in Angstrom)
psf – specter psf object
image – DESI preprocessed image object
- Optional:
maxshift : float maximum shift in pixels for 2D chi2 scan
- Returns:
1D array of x coordinates on CCD (axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction) y : 1D array of y coordinates on CCD (axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis) dx : 1D array of shifts along x coordinates on CCD dy : 1D array of shifts along y coordinates on CCD sx : 1D array of uncertainties on dx sy : 1D array of uncertainties on dy
- Return type:
x
- desispec.trace_shifts.legx(wave, wavemin, wavemax)[source]¶
Reduced coordinate (range [-1,1]) for calls to legval and legfit
- Parameters:
wave – ND np.array
wavemin – float, min. val
wavemax – float, max. val
- Returns:
array of same shape as wave
- desispec.trace_shifts.monomials(x, y, degx, degy)[source]¶
Computes monomials as a function of x and y of a 2D polynomial of degrees degx and degy
- Parameters:
x – ND array
y – ND array of same shape as x
degx – int (>=0), polynomial degree along x
degy – int (>=0), polynomial degree along y
- Returns :
monomials : ND array of shape ( (degx+1)*(degy+1) , x shape )
- desispec.trace_shifts.polynomial_fit(z, ez, xx, yy, degx, degy)[source]¶
Computes and 2D polynomial fit of z as a function of (x,y) of degrees degx and degy
- Parameters:
z – ND array
ez – ND array of same shape as z, uncertainties on z
x – ND array of same shape as z
y – ND array of same shape as z
degx – int (>=0), polynomial degree along x
degy – int (>=0), polynomial degree along y
- Returns:
1D array of size (degx+1)*(degy+1) with polynomial coefficients (as defined by routine monomials) covariance : 2D array of covariance of coeff error_floor : float , extra uncertainty needed to get chi2/ndf=1 polval : ND array of same shape as z with values of pol(x,y) mask : ND array of same shape as z indicating the masked data points in the fit
- Return type:
coeff
- desispec.trace_shifts.recompute_legendre_coefficients(xcoef, ycoef, wavemin, wavemax, degxx, degxy, degyx, degyy, dx_coeff, dy_coeff)[source]¶
Modifies legendre coefficients of an input trace set using polynomial coefficents (as defined by the routine monomials)
- Parameters:
xcoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to XCCD
ycoef – 2D np.array of shape (nfibers,ncoef) containing Legendre coefficents for each fiber to convert wavelenght to YCCD
wavemin – float
wavemax – float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1 used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])
degxx – int, degree of polynomial for x shifts as a function of x (x is axis=1 in numpy image array, AXIS=0 in FITS, cross-dispersion axis = fiber number direction)
degxy – int, degree of polynomial for x shifts as a function of y (y is axis=0 in numpy image array, AXIS=1 in FITS, wavelength dispersion axis)
degyx – int, degree of polynomial for y shifts as a function of x
degyy – int, degree of polynomial for y shifts as a function of y
dx_coeff – 1D np.array of polynomial coefficients of size (degxx*degxy) as defined by the routine monomials.
dy_coeff – 1D np.array of polynomial coefficients of size (degyx*degyy) as defined by the routine monomials.
- Returns:
2D np.array of shape (nfibers,ncoef) with modified Legendre coefficents ycoef : 2D np.array of shape (nfibers,ncoef) with modified Legendre coefficents
- Return type:
xcoef
- desispec.trace_shifts.resample_boxcar_frame(frame_flux, frame_ivar, frame_wave, oversampling=2)[source]¶
Resamples the spectra in a frame obtained with boxcar extraction to the same wavelength grid, with oversampling. Uses resample_flux routine.
- Parameters:
frame_flux – 2D np.array of shape (nfibers,nwave), sum of pixel values per row of length=width per fiber
frame_ivar – 2D np.array of shape (nfibers,nwave), ivar[f,j] = 1/( sum_[j,b:e] (1/image.ivar) ), ivar=0 if at least 1 pixel in the row has image.ivar=0 or image.mask!=0
frame_wave – 2D np.array of shape (nfibers,nwave), determined from the traces
- Optional:
oversampling : int , oversampling factor , default is 2
- Returns:
2D np.array of shape (nfibers,nwave*oversampling) ivar : 2D np.array of shape (nfibers,nwave*oversampling) frame_wave : 1D np.array of size (nwave*oversampling)
- Return type:
flux
- desispec.trace_shifts.shift_ycoef_using_external_spectrum(psf, xytraceset, image, fibers, spectrum_filename, degyy=2, width=7, prior_width_dy=0.1)[source]¶
Measure y offsets (external wavelength calibration) from a preprocessed image , a PSF + trace set using a cross-correlation of boxcar extracted spectra and an external well-calibrated spectrum. The PSF shape is used to convolve the input spectrum. It could also be used to correct for the PSF asymetry (disabled for now). A relative flux calibration of the spectra is performed internally.
- Parameters:
psf – specter PSF
xytraceset – XYTraceset object
image – DESI preprocessed image object
fibers – 1D np.array of fiber indices
spectrum_filename – path to input spectral file ( read with np.loadtxt , first column is wavelength (in vacuum and Angstrom) , second column in flux (arb. units)
- Optional:
width : int, extraction boxcar width, default is 7 degyy : int, degree of polynomial fit of shifts as a function of y, used to reject outliers. prior_width_dy: float with of the Gaussian prior on dy
- Returns:
2D np.array of same shape as input, with modified Legendre coefficents for each fiber to convert wavelenght to YCCD
- Return type:
ycoef
- desispec.trace_shifts.write_traces_in_psf(input_psf_filename, output_psf_filename, xytraceset)[source]¶
Writes traces in a PSF.
- Parameters:
input_psf_filename – Path to input fits file which has to contain XTRACE and YTRACE HDUs
output_psf_filename – Path to output fits file which has to contain XTRACE and YTRACE HDUs
xytraceset – xytraceset
desispec.tsnr¶
- desispec.tsnr.calc_alpha(frame, fibermap, rdnoise_sigma, npix_1d, angperpix, angperspecbin, fiberflat, skymodel)[source]¶
Model Var = alpha * rdnoise component + sky.
Calculate the best-fit alpha using the sky fibers available to the frame.
- Parameters:
frame – desispec frame instance (should be uncalibrated, i.e. e/A).
fibermap – desispec fibermap instance.
rdnoise_sigma – e.g. RDNOISE value per Quadrant (float).
npix_1d – equivalent to 1D nea [pixels], calculated using read_nea().
angperpix – angstroms per pixel (float),
fiberflat – desispec fiberflat instance.
skymodel – desispec Sky instance.
alpha – nuisanve parameter to reweight rdnoise vs sky contribution to variance (float).
components – if True, return individual contributions to variance, else return total variance.
- Returns:
nuisance parameter to reweight rdnoise vs sky contribution to variance (float), obtained as the best fit to the uncalibrated sky fibers VAR.
- Return type:
alpha
- desispec.tsnr.calc_tsnr2(frame, fiberflat, skymodel, fluxcalib, alpha_only=False, include_poisson=True, include_fiberfracs=True)[source]¶
Compute template SNR^2 values for a given frame
- Parameters:
frame – uncalibrated Frame object for one camera
fiberflat – FiberFlat object
sky – SkyModel object
fluxcalib – FluxCalib object
- Returns
(tsnr2, alpha): tsnr2 dictionary, with keys labeling tracer (bgs,elg,etc.), of values holding nfiber length array of the tsnr^2 values for this camera, and alpha, the relative weighting btwn rdnoise & sky terms to model var.
Note
Assumes DESIMODEL is set and up to date.
- desispec.tsnr.calc_tsnr2_cframe(cframe)[source]¶
Given cframe, calc_tsnr2 guessing frame,fiberflat,skymodel,fluxcalib to use
- Parameters:
cframe – input cframe Frame object
- Returns:
(results, alpha) from calc_tsnr2
- Return type:
- desispec.tsnr.calc_tsnr_fiberfracs(fibermap, etc_fiberfracs, no_offsets=False)[source]¶
Nominal fiberfracs for effective depths. See: https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed
- Parameters:
fibermap – desispec instance.
etc_fiberfracs – propragated etc fiberfracs per tracer, json or fits header derived (dictionary).
no_offsets – ignore throughput loss due to fiber offsets.
- Returns:
dict of the nominal fiberloss of given type and seeing.
- desispec.tsnr.fb_rdnoise(fibers, frame, tset)[source]¶
Approximate the readnoise for a given fiber (on a given camera) for the wavelengths present in frame. wave.
- Parameters:
fibers – e.g. np.arange(500) to index fiber.
frame – frame instance for the given camera.
tset – xytraceset object with fiber traces coordinates.
- Returns:
(nfiber x nwave) array with the estimated readnosie. Same units as OBSRDNA, e.g. ang per pix.
- Return type:
rdnoise
- desispec.tsnr.gen_mask(frame, skymodel, hw=5.0)[source]¶
Generate a mask for the alpha computation, masking out bright sky lines.
- Parameters:
frame – uncalibrated Frame object for one camera
skymodel – SkyModel object
hw – (optional) float, half width of mask around sky lines in A
- Returns:
An array of same shape as frame, here mask=1 is good, 0 is bad
- desispec.tsnr.get_ensemble(dirpath=None, bands=['b', 'r', 'z'], smooth=0)[source]¶
Function that takes a frame object and a bitmask and returns ivar (and optionally mask) array(s) that have fibers with offending bits in fibermap[‘FIBERSTATUS’] set to 0 in ivar and optionally flips a bit in mask.
- Parameters:
- Returns:
Dictionary with keys labelling each tracer (bgs, lrg, etc.) for which each value is a Spectra class instance with wave, flux for BRZ arms. Note flux is the high frequency residual for the ensemble. See doc. 4723.
- desispec.tsnr.read_nea(path)[source]¶
Read a master noise equivalent area [sq. pixel] file.
- Parameters:
path – path to a master nea file for a given camera, e.g. b0.
- Returns:
A tuple containing:
nea: 2D split object to be evaluated at (fiber, wavelength)
angperpix: 2D split object to be evaluated at (fiber, wavelength), yielding angstrom per pixel.
- Return type:
- class desispec.tsnr.template_ensemble(tracer, config_filename=None)[source]¶
Generate an ensemble of templates to sample tSNR for a range of points in (z, m, OII, etc.) space.
If conditioned, uses deepfield redshifts and (currently r) magnitudes to condition simulated templates.
- compute(nmodel=5, smooth=100.0, nz_table_filename=None, single_mag=True, convolve_to_nz=True)[source]¶
Compute a template ensemble for template S/N measurements (tSNR)
- Parameters:
nmodel – number of template models to generate
smooth – smoothing scale for dF=<F - smooth(F)>
nz_table_filename – path to ASCII file with columns zmin,zmax,n
single_mag – generate all templates at same average magnitude to limit MC noise
convolve_to_nz – if True, each template dF^2 is convolved to match the n(z) (redshift distribution)
- effmag(m1, m2)[source]¶
returns an effective mag which is the magnitude that gives the same average flux^2 for the mag range specified [m1,m2] assuming a flat magnitude distribution
- generate_templates(nmodel, redshifts=None, mags=None, single_mag=True)[source]¶
Dedicated wrapper for desisim.templates.GALAXY.make_templates call, stipulating templates in a redshift range suggested by the FDR. Further, assume fluxes close to the expected (within ~0.5 mags.) in the appropriate band.
Class init will write ensemble stack to disk at outdir, for a given tracer [bgs, lrg, elg, qso], having generated nmodel templates. Optionally, provide redshifts and mags. to condition appropriately at cost of runtime.
- desispec.tsnr.tsnr2_to_efftime(tsnr2, target_type)[source]¶
Converts TSNR2 values to effective exposure time.
- Parameters:
tsnr2 – TSNR**2 values, float or numpy array
target_type – str, “ELG”,”BGS”,”LYA”, or other depending on content of data/tsnr/tsnr-efftime.yaml
- Returns:
Exptime in seconds, same type and shape if applicable as input tsnr2
- desispec.tsnr.var_model(rdnoise_sigma, npix_1d, angperpix, angperspecbin, fiberflat, skymodel, alpha=1.0, components=False)[source]¶
Evaluate a model for the 1D spectral flux variance, e.g. quadrature sum of readnoise and sky components.
- Parameters:
rdnoise_sigma –
npix_1d – equivalent to (1D) nea.
angperpix – Angstroms per pixel.
angperspecbin – Angstroms per bin.
fiberflat – fiberflat instance
skymodel – Sky instance.
alpha – empirical weighting of the rdnoise term to e.g. better fit sky fibers per exp. cam.
components – if True, return tuple of individual contributions to the variance. Else return variance.
- Returns:
nfiber x nwave array of the expected variance.
- desispec.tsnr.var_tracer(tracer, frame, angperspecbin, fiberflat, fluxcalib, exposure_seeing_fwhm=1.1)[source]¶
- Source Poisson term to the model ivar, following conventions defined at:
- Parameters:
tracer – [bgs, backup] string, defines program.
frame – desispec.frame instance
angperspecbin – float, angstroms per bin in spectral reductions
fiberflat – desispec instance
fluxcalib – desispec instance
fiber_diameter_arcsec –
- Returns:
nominal flux [e/specbin] corresponding to frame.wave
desispec.util¶
Utility functions for desispec.
- desispec.util.argmatch(a, b)[source]¶
Returns indices ii such that a[ii] == b
- Parameters:
a – array-like
b – array-like
Returns indices ii such that a[ii] == b
Both a and b are allowed to have repeats, and a values can be a superset of b, but b cannot contain values that are not in a because then no indices ii could result in a[ii] == b.
Related: desitarget.geomask.match_to which is similar, but doesn’t allow duplicates in b.
- desispec.util.combine_ivar(ivar1, ivar2)[source]¶
Returns the combined inverse variance of two inputs, making sure not to divide by 0 in the process.
ivar1 and ivar2 may be scalar or ndarray but must have the same dimensions
- desispec.util.dateobs2night(dateobs)[source]¶
Convert DATE-OBS ISO8601 UTC string to YEARMMDD int night of KPNO sunset
- desispec.util.header2night(header)[source]¶
Return YEARMMDD night from FITS header, handling common problems
- desispec.util.healpix_degrade_fixed(nside, pixel)[source]¶
Degrade a NEST ordered healpix pixel with a fixed ratio.
This degrades the pixel to a lower nside value that is fixed to half the healpix “factor”.
- Returns (tuple):
a tuple of ints, where the first value is the new NSIDE and the second value is the degraded pixel index.
- desispec.util.itemindices(a)[source]¶
Return dict[key] -> list of indices i where a[i] == key
- Parameters:
a – array-like of hashable values
Return dict[key] -> list of indices i where a[i] == key
The dict keys are inserted in the order that they first appear in a, and the value lists of indices are sorted
e.g. itemindices([10,30,20,30]) -> {10: [0], 30: [1, 3], 20: [2]}
- desispec.util.mask32(mask)[source]¶
Return an input mask as unsigned 32-bit
Raises ValueError if 64-bit input can’t be cast to 32-bit without losing info (i.e. if it contains values > 2**32-1)
- desispec.util.mpi_count_failures(num_cmd, num_err, comm=None)[source]¶
Sum num_cmd and num_err across MPI ranks
- Options:
comm: mpi4py communicator
- Returns:
sum(num_cmd), sum(num_err) summed across all MPI ranks
If
comm
is None, returns input num_cmd, num_err
- desispec.util.night2ymd(night)[source]¶
parse night YEARMMDD string into tuple of integers (year, month, day)
- desispec.util.option_list(opts)[source]¶
Convert key, value pairs into command-line options.
- Parameters:
opts (dict-like) – Convert a dictionary into command-line options.
- Returns:
A list of command-line options.
- Return type:
- desispec.util.ordered_unique(ar, return_index=False)[source]¶
Find the unique elements of an array in the order they first appear
Like numpy.unique, but preserves original order instead of sorting
- Parameters:
ar – array-like data to find unique elements
- Options:
return_index: if True also return indices in ar where items first appear
- desispec.util.parse_fibers(fiber_string, include_end=False)[source]¶
Short func that parses a string containing a comma separated list of integers, which can include “:” or “..” or “-” labeled ranges
- Parameters:
fiber_string (str) – list of integers or integer ranges
- Options:
include_end (bool): if True, include end-value in ranges
- Returns (array 1-D):
1D numpy array listing all of the integers given in the list, including enumerations of ranges given.
Note: this follows python-style ranges, i,e, 1:5 or 1..5 returns 1, 2, 3, 4 unless include_end is True, which then returns 1,2,3,4,5
- desispec.util.parse_int_args(arg_string, include_end=False)[source]¶
Short func that parses a string containing a comma separated list of integers, which can include “:” or “..” or “-” labeled ranges
- Parameters:
arg_string (str) – list of integers or integer ranges
- Options:
include_end (bool): if True, include end-value in ranges
- Returns (array 1-D):
1D numpy array listing all of the integers given in the list, including enumerations of ranges given.
Note: this follows python-style ranges, i,e, 1:5 or 1..5 returns 1,2,3,4 unless include_end is True, which then returns 1,2,3,4,5
- desispec.util.parse_keyval(keyval)[source]¶
Parse “key=val” string -> (key,val) tuple with int/float/str/bool val
- Parameters:
keyval (str) – “key=value” string
Returns (key, value) tuple where value has been promoted from string into int/float/bool if possible.
value=”True” or “False” becomes boolean True/False, but all other forms like “T”/”F” or “true”/”false” remain strings. 0 and 1 become ints, not bool.
- desispec.util.pid_exists(pid)[source]¶
Check whether pid exists in the current process table.
UNIX only. Should work the same as psutil.pid_exists().
- desispec.util.runcmd(cmd, args=None, expandargs=False, inputs=[], outputs=[], comm=None, clobber=False)[source]¶
Runs a command (function or script), checking for inputs and outputs
- Parameters:
cmd – function object or command string to run with subprocess.call()
- Options:
args : list of args to pass to the function or script expandargs: call function with
cmd(*args)
instead ofcmd(args)
inputs : list of filename inputs that must exist before running outputs : list of output filenames that should be created clobber : if True, run even if outputs already exist comm : MPI communicator to pass to cmd(…, comm=comm)
- Returns:
(result, success)
Notes
If any inputs are missing, don’t run cmd and return (None, False).
If outputs exist and have timestamps after all inputs, don’t run cmd and return (None, True).
If spawned as a script, return (returncode, (returncode==0)).
If function raises an exception, return (exception, False).
If function returns result but outputs are missing, return (result, False).
If function returns result and all outputs are present, return (result, True).
- desispec.util.set_backend(backend='agg')[source]¶
Set matplotlib to use a batch-friendly backend
This function is safe to call multiple times without tripping on a previously set backend (which remains set)
- desispec.util.sprun(com, capture=False, input=None)[source]¶
Run a command with subprocess and handle errors.
This runs a command and returns the lines of STDOUT as a list. Any contents of STDERR are logged. If an OSError is raised by the child process, that is also logged. If another exception is raised by the child process, the traceback from the child process is printed.
- Parameters:
- Returns:
- the return code and optionally the lines of STDOUT
from the child process.
- Return type:
- desispec.util.ymd2night(year, month, day)[source]¶
convert year, month, day integers into cannonical YEARMMDD night string
desispec.validredshifts¶
- desispec.validredshifts.actually_validate(cat, fiberstatus_cut=True, ignore_emline=False)[source]¶
Apply redshift quality criteria
- Parameters:
cat – astropy table with the necessary columns for redshift quality determination
- Options:
fiberstatus_cut: bool (default True), if True, impose requirements on COADD_FIBERSTATUS and ZWARN return_target_columns: bool (default False), if True, include columns that indicate if the object belongs to each class of DESI targets extra_columns: list of str (default None), additional columns to include in the output emline_path: str (default None), specify the location of the emline file; by default the emline file is in the same directory as the redrock file ignore_emline: bool (default False), if True, ignore the emline file and do not validate the ELG redshift
- Returns:
astropy table with boolean columns (e.g., GOOD_BGS) added
- Return type:
cat
- desispec.validredshifts.get_good_fiberstatus(cat, isqso=False)[source]¶
Validate the fiber status for a redrock catalog.
- Parameters:
cat – redrock catalog (e.g., in astropy table format)
- Options:
isqso: bool (default False), if True, use the specific QSO criteria
- Returns:
boolean array
- Return type:
good_fiberstatus
- desispec.validredshifts.validate(redrock_path, fiberstatus_cut=True, return_target_columns=False, extra_columns=None, emline_path=None, ignore_emline=False)[source]¶
Validate the redshift quality with tracer-dependent criteria for redrock+afterburner results.
- Parameters:
redrock_path – str, path of redrock FITS file
- Options:
fiberstatus_cut: bool (default True), if True, impose requirements on COADD_FIBERSTATUS and ZWARN return_target_columns: bool (default False), if True, include columns that indicate if the object belongs to each class of DESI targets extra_columns: list of str (default None), additional columns to include in the output emline_path: str (default None), specify the location of the emline file; by default the emline file is in the same directory as the redrock file ignore_emline: bool (default False), if True, ignore the emline file and do not validate the ELG redshift
- Returns:
- astropy table with basic columns such as TARGETID and boolean columns (e.g., GOOD_BGS)
that indicate if each object meets the redshift quality criteria of specific tracers
- Return type:
cat
desispec.workflow¶
Tools for workflow management and running the pipeline.
desispec.workflow.batch¶
Utilities for working with slurm batch queues.
- desispec.workflow.batch.default_system(jobdesc=None, no_gpu=False)[source]¶
Guess default system to use based on environment
- desispec.workflow.batch.get_config(name)[source]¶
Return configuration dictionary for system name
- Parameters:
name (str) – e.g. cori-haswell, cori-knl, dirac, perlmutter-gpu, …
- Returns dictionary with keys:
site: location of system, e.g. ‘NERSC’
cores_per_node: number of physical cores per node
threads_per_core: hyperthreading / SMT per core
memory: memory per node in GB
timefactor: scale time estimates by this amount on this system
gpus_per_node: number of GPUs per node
batch_opts: list of additional batch options for script header
desispec.workflow.desi_proc_funcs¶
- desispec.workflow.desi_proc_funcs.add_desi_proc_joint_fit_terms(parser)[source]¶
Add parameters to the argument parser that are only used by desi_proc_joint_fit
- desispec.workflow.desi_proc_funcs.add_desi_proc_singular_terms(parser)[source]¶
Add parameters to the argument parser that are only used by desi_proc
- desispec.workflow.desi_proc_funcs.add_desi_proc_tilenight_terms(parser)[source]¶
Add parameters to the argument parser that are only used by desi_proc_tilenight
- desispec.workflow.desi_proc_funcs.assign_mpi(do_mpi, do_batch, log)[source]¶
Based on whether the mpi flag is set and whether the batch flag is set, assign the appropriate MPI values for the communicator, rank, and number of ranks. Also verify that environment variables are set and provide useful information via the log
- Parameters:
do_mpi – bool, whether to use mpi or not
do_batch – bool, whether the script is meant to write a batch script and exit or not.
log – desi log object for reporting
- Returns:
A tuple containing:
comm: MPI communicator object
rank: int, the numeric number assigned to the currenct MPI rank
size: int, the total number of MPI ranks
- Return type:
- desispec.workflow.desi_proc_funcs.cameras_from_raw_data(rawdata)[source]¶
Takes a filepath or fitsio FITS object corresponding to a DESI raw data file and returns a list of cameras for which data exists in the file.
- desispec.workflow.desi_proc_funcs.create_desi_proc_batch_script(night, exp, cameras, jobdesc, queue, runtime=None, batch_opts=None, timingfile=None, batchdir=None, jobname=None, cmdline=None, system_name=None, use_specter=False, no_gpu=False)[source]¶
Generate a SLURM batch script to be submitted to the slurm scheduler to run desi_proc.
- Parameters:
exp (str, int, or list of int) – The exposure id(s) for the data.
cameras (str or list of str) – List of cameras to include in the processing.
jobdesc (str) – Description of the job to be performed. Used to determine requested resources and whether to operate in a more mpi parallelism (all except poststdstar) or less (only poststdstar). Directly relate to the obstype, with science exposures being split into two (pre, post)-stdstar, and adding joint fit categories stdstarfit, psfnight, and nightlyflat. Options include: ‘prestdstar’, ‘poststdstar’, ‘stdstarfit’, ‘arc’, ‘flat’, ‘psfnight’, ‘nightlyflat’
queue (str) – Queue to be used.
runtime (str, optional) – Timeout wall clock time.
batch_opts (str, optional) – Other options to give to the slurm batch scheduler (written into the script).
timingfile (str, optional) – Specify the name of the timing file.
batchdir (str, optional) – Specify where the batch file will be written.
jobname (str, optional) – Specify the name of the slurm script written.
cmdline (str, optional) – Complete command as would be given in terminal to run the desi_proc. Can be used instead of reading from argv.
system_name (str, optional) – name of batch system, e.g. cori-haswell, cori-knl
use_specter (bool, optional) – Use classic specter instead of gpu_specter for extractions
no_gpu (bool, optional) – Do not use GPU even if available
- Returns:
the full path name for the script written.
- Return type:
scriptfile
Note
batchdir and jobname can be used to define an alternative pathname, but may not work with assumptions in desi_proc. These optional arguments should be used with caution and primarily for debugging.
- desispec.workflow.desi_proc_funcs.create_desi_proc_tilenight_batch_script(night, exp, tileid, ncameras, queue, runtime=None, batch_opts=None, system_name=None, mpistdstars=True, use_specter=False, no_gpu=False, laststeps=None, cameras=None)[source]¶
Generate a SLURM batch script to be submitted to the slurm scheduler to run desi_proc.
- Parameters:
night – str or int. The night the data was acquired.
exp – int, or list of ints. The exposure id(s) for the data.
tileid – str or int. The tile id for the data.
ncameras – int. The number of cameras used for joint fitting.
queue – str. Queue to be used.
- Options:
runtime: str. Timeout wall clock time. batch_opts: str. Other options to give to the slurm batch scheduler (written into the script). system_name: name of batch system, e.g. cori-haswell, cori-knl. mpistdstars: bool. Whether to use MPI for stdstar fitting. use_specter: bool. Use classic specter instead of gpu_specter for extractions no_gpu: bool. Do not use GPU even if available laststeps: list of str. A list of laststeps to pass as the laststeps argument to tilenight cameras: str, must be camword.
- Returns:
the full path name for the script written.
- Return type:
scriptfile
- desispec.workflow.desi_proc_funcs.determine_resources(ncameras, jobdesc, queue, nexps=1, forced_runtime=None, system_name=None)[source]¶
Determine the resources that should be assigned to the batch script given what desi_proc needs for the given input information.
- Parameters:
ncameras (int) – number of cameras to be processed
jobdesc (str) – type of data being processed
queue (str) – the queue at NERSC to be submitted to. ‘realtime’ will force restrictions on number of nodes.
nexps (int, optional) – the number of exposures processed in this step
force_runtime (int, optional) – the amount of runtime in minutes to allow for the script. Should be left to default heuristics unless needed for some reason.
system_name (str, optional) – batch compute system, e.g. cori-haswell or perlmutter-gpu
- Returns:
A tuple containing:
ncores: int, number of cores (actually 2xphysical cores) that should be submitted via “-n {ncores}”
nodes: int, number of nodes to be requested in the script. Typically (ncores-1) // cores_per_node + 1
runtime: int, the max time requested for the script in minutes for the processing.
- Return type:
- desispec.workflow.desi_proc_funcs.find_most_recent(night, file_type='psfnight', cam='r', n_nights=30)[source]¶
Searches back in time for either psfnight or fiberflatnight (or anything supported by desispec.calibfinder.findcalibfile. This only works on nightly-based files, so exposure id information is not used.
- Parameters:
night – str. YYYYMMDD night to look back from
file_type – str. psfnight or fiberflatnight
cam – str. camera (b, r, or z).
n_nights – int. number of nights to step back before giving up
- Returns:
Full pathname to calibration file of interest. If none found, None is returned.
- Return type:
- desispec.workflow.desi_proc_funcs.get_desi_proc_batch_file_name(night, exp, jobdesc, cameras)[source]¶
Returns the default directory location to store a batch script file given a night
- Parameters:
night – str or int, defines the night (should be 8 digits)
exp – str, int, or array of ints, defines the exposure id(s) relevant to the job
jobdesc – str, type of data being processed
cameras – str or list of str. If str, must be camword, If list, must be list of cameras to include in the processing.
- Returns:
str, the default script name for a desi_proc batch script file
- Return type:
pathname
- desispec.workflow.desi_proc_funcs.get_desi_proc_batch_file_path(night, reduxdir=None)[source]¶
Returns the default directory location to store a batch script file given a night
- desispec.workflow.desi_proc_funcs.get_desi_proc_batch_file_pathname(night, exp, jobdesc, cameras, reduxdir=None)[source]¶
Returns the default directory location to store a batch script file given a night
- Parameters:
night – str or int, defines the night (should be 8 digits)
exp – str, int, or array of ints, defines the exposure id(s) relevant to the job
jobdesc – str, type of data being processed
cameras – str or list of str. If str, must be camword, If list, must be list of cameras to include in the processing.
reduxdir – str (optional), define the base directory where the /run/scripts directory should or does live
- Returns:
str, the default location and script name for a desi_proc batch script file
- Return type:
pathname
- desispec.workflow.desi_proc_funcs.get_desi_proc_joint_fit_parser()[source]¶
Create an argparser object for use with desi_proc_joint_fit based on arguments from sys.argv
- desispec.workflow.desi_proc_funcs.get_desi_proc_parser()[source]¶
Create an argparser object for use with desi_proc based on arguments from sys.argv
- desispec.workflow.desi_proc_funcs.get_desi_proc_tilenight_batch_file_name(night, tileid)[source]¶
Returns the filename for a tilenight batch script file given a night and tileid
- Parameters:
night – str or int, defines the night (should be 8 digits)
tileid – str or int, defines the tile id relevant to the job
- Returns:
str, the default script name for a desi_proc_tilenight batch script file
- Return type:
pathname
- desispec.workflow.desi_proc_funcs.get_desi_proc_tilenight_batch_file_pathname(night, tileid, reduxdir=None)[source]¶
Returns the default directory location to store a tilenight batch script file given a night and tileid
- Parameters:
night – str or int, defines the night (should be 8 digits)
tileid – str or int, defines the tile id relevant to the job
reduxdir – str (optional), define the base directory where the /run/scripts directory should or does live
- Returns:
str, the default location and script name for a desi_proc_tilenight batch script file
- Return type:
pathname
- desispec.workflow.desi_proc_funcs.get_desi_proc_tilenight_parser()[source]¶
Create an argparser object for use with desi_proc_tilenight based on arguments from sys.argv
Create an argparser object for use with desi_proc AND desi_proc_joint_fit based on arguments from sys.argv
- desispec.workflow.desi_proc_funcs.load_raw_data_header(pathname, return_filehandle=False)[source]¶
Open raw desi data file given at pathname and return the spectrograph header, which varied in name over time
- Parameters:
pathname – str, the full path to the raw data file
return_filehandle – bool, whether to return the open file handle or to close it and only return the header. Default is False.
- Returns:
A tuple containing:
hdr: fitsio header object from the file at pathname
fx: optional, fitsio file object returned only if return_filehandle is True
- Return type:
- desispec.workflow.desi_proc_funcs.log_timer(timer, timingfile=None, comm=None)[source]¶
Log timing info, optionally writing to json timingfile
- Parameters:
timer – desiutil.timer.Timer object
timingfile (str) – write json output to this file
comm – MPI communicator
Notes
If comm is not None, collect timers across ranks.
If timingfile already exists, read and append timing then re-write.
- desispec.workflow.desi_proc_funcs.update_args_with_headers(args)[source]¶
Update input argparse object with values from header if the argparse values are uninformative defaults (generally python None). This also returns the primary header and each camera’s header.
- Parameters:
args – argparse arguments object. Parsed from the command line arguments based on the parser defined by the function get_desi_proc_parser().
- Returns:
A tuple containing:
args: modified version of the input args where values have been updated if None using information from appropriate headers using either night+expid or an input file.
hdr: fitsio header object obtained using
read_header()
on input file or file determined from args information.camhdr: dict, dictionary of fitsio header objects for each camera in the input files.
- Return type:
Note
The input args is modified and returned here.
desispec.workflow.exptable¶
- desispec.workflow.exptable.aircorr_to_airmass(aircorr)[source]¶
Transforms an “air correction” term of survey speed to airmass: AIRCORR = X^1.75 https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed
- desispec.workflow.exptable.airfac_to_aircorr(airfac)[source]¶
Transforms an “AIRFAC” term of survey speed to an “air correction” term of survey speed https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed
- desispec.workflow.exptable.airfac_to_airmass(airfac, k=0.114)[source]¶
Transforms an “AIRFAC” term of survey speed to airmass: AIRFAC = 10^[k*(X-1)/2.5] https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed
- desispec.workflow.exptable.airmass_to_aircorr(airmass)[source]¶
Transforms an airmass to “air correction” term of survey speed: AIRCORR = X^1.75 https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed
- desispec.workflow.exptable.airmass_to_airfac(airmass, k=0.114)[source]¶
Transforms an airmass to “AIRFAC”: AIRFAC = 10^[k*(X-1)/2.5] https://desi.lbl.gov/trac/wiki/SurveyOps/SurveySpeed
- desispec.workflow.exptable.deconstruct_keyval_reporting(entry)[source]¶
Takes a reporting of the form
{colname}:{oldval}->{newval}
and returns colname, oldval, newval.- Parameters:
entry (str) – A string of the form
{colname}:{oldval}->{newval}
. colname should be an all upper case column name. oldval and newval can include any string characters except the specific combination “->”.- Returns:
A tuple containing:
key, str. The string that precedes the initial colon. The name of the column being reported.
val1, str. The string after the initial colon and preceding the ‘->’. The original value of the column.
val2, str. The string after the ‘->’. The value that the original was changed to.
- Return type:
- desispec.workflow.exptable.default_obstypes_for_exptable()[source]¶
Defines the exposure types to be recognized by the workflow and saved in the exposure table by default.
- Returns:
A list of default obstypes to be included in an exposure table.
- Return type:
- desispec.workflow.exptable.exposure_table_column_defs()[source]¶
Contains the column names, data types, and default row values for a DESI Exposure table. It returns the names, datatypes, and defaults as a list of 3-tuples.
- Returns:
List of tuples (name, type, default).
- Return type:
- desispec.workflow.exptable.get_exposure_flags()[source]¶
Defines the exposure flags that can be saved in the exposure table.
- Returns:
A list of exposure flags that can be included in an exposure table.
- Return type:
- desispec.workflow.exptable.get_exposure_table_column_defaults(asdict=True)[source]¶
Returns the default values for each column entry of a DESI pipeline Exposure table. It returns the names as keys and defaults as values in a dictionary if asdict=True. If False, the defaults are returned as a list (in the same order as the names returned by get_exposure_table_column_names().
- Parameters:
asdict (bool, optional) – Default is True. If you want the defaults as values in a dictionary with keys as the names of the columns. If False, an ordered list is returned.
- Returns:
If asdict, a dict is returned with column defaults as the values and columns names as the keys. If False, a list of defaults in the same order as the names returned from get_exposure_table_column_names().
- Return type:
- desispec.workflow.exptable.get_exposure_table_column_defs(return_default_values=False)[source]¶
Contains the column names, data types, and default row values for a DESI Exposure table. It returns the names and datatypes with the defaults being given with an optional flag. Returned as 2 (or 3) lists.
- Parameters:
return_default_values (bool, optional) –
True
if you want the default values returned.- Returns:
A tuple containing:
colnames, list. List of column names for an exposure table.
coltypes, list. List of column datatypes for the names in colnames.
coldeflts, list. Optionally returned if return_default_values is True. List of default values for the corresponding colnames.
- Return type:
- desispec.workflow.exptable.get_exposure_table_column_names()[source]¶
Contains the column names, data types, and default row values for a DESI Exposure table. It returns the names as a list.
- Returns:
List of column names for an exposure table.
- Return type:
- desispec.workflow.exptable.get_exposure_table_column_types(asdict=True)[source]¶
Returns the datatypes values for each column entry of a DESI pipeline Exposure table. It returns the names as keys and datatypes as values in a dictionary if asdict=True. If False, the datatypes are returned as a list (in the same order as the names returned by get_exposure_table_column_names().
- Parameters:
asdict (bool, optional) –
True
if you want the types as values in a dictionary with keys as the names of the columns. If False, an ordered list is returned. Default is True.- Returns:
If asdict, a dict is returned with column datatypes as the values and columns names as the keys. If False, a list of datatypes in the same order as the names returned from get_exposure_table_column_names().
- Return type:
- desispec.workflow.exptable.get_exposure_table_name(night, extension='csv')[source]¶
Defines the default exposure name given the night of the observations and the optional extension.
- Parameters:
- Returns:
The exposure table name given the input night and extension.
- Return type:
- desispec.workflow.exptable.get_exposure_table_path(night=None, usespecprod=True)[source]¶
Defines the default path to save an exposure table. If night is given, it saves it under a monthly directory to reduce the number of files in a large production directory.
- Parameters:
- Returns:
The full path to the directory where the exposure table should be written (or is already written). This does not including the filename.
- Return type:
- desispec.workflow.exptable.get_exposure_table_pathname(night, usespecprod=True, extension='csv')[source]¶
Defines the default pathname to save an exposure table.
- Parameters:
- Returns:
The full pathname where the exposure table should be written (or is already written). This includes the filename.
- Return type:
- desispec.workflow.exptable.get_last_step_options()[source]¶
Defines the LASTSTEP options that can be saved in the exposure table that will be understood by the pipeline.
- Returns:
A list of LASTSTEP’s that can be included in an exposure table.
- Return type:
- desispec.workflow.exptable.instantiate_exposure_table(colnames=None, coldtypes=None, rows=None)[source]¶
Create an empty exposure table with proper column names and datatypes. If rows is given, it inserts the rows into the table, otherwise it returns a table with no rows.
- Parameters:
colnames (list) – List of column names for an exposure table.
coldtypes (list) – List of column datatypes for the names in colnames.
rows (list or np.array of Table.Row or dict) – An iterable set of Table.Row or dict with keys/colnames and value pairs that match the default column names and data types of the default exposure table.
- Returns:
An astropy Table with the column names and data types for a DESI workflow exposure table. If the input rows was not None, it contains those rows, otherwise it has no rows.
- Return type:
Table
- desispec.workflow.exptable.keyval_change_reporting(keyword, original_val, replacement_val)[source]¶
Creates a reporting string to be saved in the HEADERERR or COMMENTS column of the exposure table. Give the keyword, the original value, and the value that it was replaced with.
- Parameters:
keyword (str. The value typically saved for that) –
to. (str. Keyword in the exposure table that the values correspond) –
original_val –
keyword –
deviates. (except when it) –
replacement_val –
column. (str. The value that was saved instead in that keyword) –
- Returns:
Of the format ‘ keyword:original->replacement ‘
- Return type:
- desispec.workflow.exptable.night_to_month(night)[source]¶
Trivial function that returns the month portion of a night. Can be given a string or int.
- desispec.workflow.exptable.summarize_exposure(raw_data_dir, night, exp, obstypes=None, colnames=None, coldefaults=None, verbosely=False)[source]¶
Given a raw data directory and exposure information, this searches for the raw DESI data files for that exposure and loads in relevant information for that flavor+obstype. It returns a dictionary if the obstype is one of interest for the exposure table, a string if the exposure signifies the end of a calibration sequence, and None if the exposure is not in the given obstypes.
- Parameters:
raw_data_dir – nightly subdirectories reside.
the (str. The path to where the raw data is stored. It should be the upper level directory where) – nightly subdirectories reside.
night –
in. (str or int. Used to know what nightly subdirectory to look for the given exposure) –
exp –
obstypes – information about that exposure is taken and returned for the exposure table. Otherwise None is returned (or str if it is an end-of-cal manifest). If None, the default list in default_obstypes_for_exptable() is used.
found (list or np.array of str's. The list of 'OBSTYPE' keywords to match to. If a match is) – information about that exposure is taken and returned for the exposure table. Otherwise None is returned (or str if it is an end-of-cal manifest). If None, the default list in default_obstypes_for_exptable() is used.
the – information about that exposure is taken and returned for the exposure table. Otherwise None is returned (or str if it is an end-of-cal manifest). If None, the default list in default_obstypes_for_exptable() is used.
colnames – get_exposure_table_column_defs().
None (list or np.array. List of default values for the corresponding colnames. If) – get_exposure_table_column_defs().
from (the defaults are taken) – get_exposure_table_column_defs().
coldefaults – are taken from get_exposure_table_column_defs().
None – are taken from get_exposure_table_column_defs().
defaults (the) – are taken from get_exposure_table_column_defs().
verbosely (True) or more succinct output (False) –
output (bool. Whether to print more detailed) –
- Returns:
A variety of types which could be:
dict: Dictionary with keys corresponding to the column names of an exposure table. Values are taken from the data when found, otherwise the values are the corresponding default given in coldefaults OR;
- str: If the exposures signifies the end of a calibration sequence, it returns a string describing the type of
sequence that ended. Either “(short|long|arc) calib complete” OR;
NoneType: If the exposure obstype was not in the requested types (obstypes).
- Return type:
desispec.workflow.proc_dashboard_funcs¶
- desispec.workflow.proc_dashboard_funcs._initialize_page(color_profile, titlefill='Processing')[source]¶
Initialize the html file for showing the statistics, giving all the headers and CSS setups.
- desispec.workflow.proc_dashboard_funcs._write_js_script(output_path)[source]¶
Return the javascript script to be added to the html file
- desispec.workflow.proc_dashboard_funcs.check_running(proc_name='desi_dailyproc', suppress_outputs=False)[source]¶
Check if the desi_dailyproc process is running
- desispec.workflow.proc_dashboard_funcs.find_new_exps(fileglob, known_exposures)[source]¶
Check the path given for new exposures
- desispec.workflow.proc_dashboard_funcs.generate_monthly_table_html(tables, statuses, month)[source]¶
Add a collapsible and extendable table to the html file for a specific month Input tables: list of tables generated by ‘nightly_table’ month: string of YYYYMM, e.g. 202001 output: The string to be added to the html file
- desispec.workflow.proc_dashboard_funcs.js_str()[source]¶
Return the javascript script to be added to the html file
desispec.workflow.procfuncs¶
- desispec.workflow.procfuncs.arc_joint_fit(ptable, arcs, internal_id, queue='realtime', reservation=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]¶
Wrapper function for desiproc.workflow.procfuns.joint_fit specific to the psfnight joint fit.
All variables are the same except:
Arg 'arcs' is mapped to the prows argument of joint_fit. The joint_fit argument descriptor is pre-defined as 'psfnight'.
- desispec.workflow.procfuncs.assign_dependency(prow, dependency)[source]¶
Given input processing row and possible arcjob (processing row for psfnight) and flatjob (processing row for nightlyflat), this defines the JOBDESC keyword and assigns the dependency appropriate for the job type of prow.
- Parameters:
prow – ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.
for (Table.Row or dict. Must include keyword accessible definitions for 'OBSTYPE'. A row must have column names) – ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.
dependency – for the job in prow. This must contain keyword accessible values for ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
Table.Row (NoneType or scalar/list/array of) – for the job in prow. This must contain keyword accessible values for ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
input (dict. Processing row corresponding to the required) – for the job in prow. This must contain keyword accessible values for ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
- Returns:
The same prow type and keywords as input except with modified values updated values for ‘JOBDESC’, ‘INT_DEP_IDS’. and ‘LATEST_DEP_ID’.
- Return type:
Table.Row or dict
Note
This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).
- desispec.workflow.procfuncs.batch_script_name(prow)[source]¶
Wrapper script that takes a processing table row (or dictionary with NIGHT, EXPID, JOBDESC, PROCCAMWORD defined) and determines the script file pathname as defined by desi_proc’s helper functions.
- desispec.workflow.procfuncs.check_for_outputs_on_disk(prow, resubmit_partial_complete=True)[source]¶
- Parameters:
prow (Table.Row or dict) – Must include keyword accessible definitions for processing_table columns found in desispect.workflow.proctable.get_processing_table_column_defs()
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
- Returns:
The same prow type and keywords as input except with modified values updated to reflect the change in job status after creating and submitting the job for processing.
- Return type:
Table.Row or dict
- desispec.workflow.procfuncs.checkfor_and_submit_joint_job(ptable, arcs, flats, sciences, calibjobs, lasttype, internal_id, z_submit_types=None, dry_run=0, queue='realtime', reservation=None, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]¶
Takes all the state-ful data from daily processing and determines whether a joint fit needs to be submitted. Places the decision criteria into a single function for easier maintainability over time. These are separate from the new standard manifest*.json method of indicating a calibration sequence is complete. That is checked independently elsewhere and doesn’t interact with this.
- Parameters:
ptable (Table) – Processing table of all exposures that have been processed.
arcs (list of dict) – list of the individual arc jobs to be used for the psfnight (NOT all the arcs, if multiple sets existed). May be empty if none identified yet.
flats (list of dict) – list of the individual flat jobs to be used for the nightlyflat (NOT all the flats, if multiple sets existed). May be empty if none identified yet.
sciences (list of dict) – list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet.
calibjobs (dict) – Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.
lasttype (str or None) – the obstype of the last individual exposure row to be processed.
internal_id (int) – an internal identifier unique to each job. Increments with each new job. This is the smallest unassigned value.
z_submit_types (list of str) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.
dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
queue (str, optional) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.
reservation (str, optional) – The reservation to submit jobs to. If None, it is not submitted to a reservation.
strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
system_name (str) – batch system name, e.g. cori-haswell, cori-knl, permutter-gpu
- Returns:
A tuple containing:
ptable, Table, Processing table of all exposures that have been processed.
calibjobs, dict. Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.
sciences, list of dicts, list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet or we just submitted them for processing.
internal_id, int, if no job is submitted, this is the same as the input, otherwise it is incremented upward from from the input such that it represents the smallest unused ID.
- Return type:
- desispec.workflow.procfuncs.create_and_submit(prow, queue='realtime', reservation=None, dry_run=0, joint=False, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None, use_specter=False, laststeps=None)[source]¶
Wrapper script that takes a processing table row and three modifier keywords, creates a submission script for the compute nodes, and then submits that script to the Slurm scheduler with appropriate dependencies.
- Parameters:
prow (Table.Row or dict) – Must include keyword accessible definitions for processing_table columns found in desispect.workflow.proctable.get_processing_table_column_defs()
queue (str, optional) – The name of the NERSC Slurm queue to submit to. Default is the realtime queue.
reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.
dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
joint (bool, optional) – Whether this is a joint fitting job (the job involves multiple exposures) and therefore needs to be run with desi_proc_joint_fit. Default is False.
strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu
use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.
laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.
- Returns:
The same prow type and keywords as input except with modified values updated to reflect the change in job status after creating and submitting the job for processing.
- Return type:
Table.Row or dict
Note
This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).
- desispec.workflow.procfuncs.create_batch_script(prow, queue='realtime', dry_run=0, joint=False, system_name=None, use_specter=False, laststeps=None)[source]¶
Wrapper script that takes a processing table row and three modifier keywords and creates a submission script for the compute nodes.
- Parameters:
prow – desispect.workflow.proctable.get_processing_table_column_defs()
in (Table.Row or dict. Must include keyword accessible definitions for processing_table columns found) – desispect.workflow.proctable.get_processing_table_column_defs()
queue –
queue. (str. The name of the NERSC Slurm queue to submit to. Default is the realtime) –
dry_run – If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
nonzero (int. If) – If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
submitted. (this is a simulated run. If dry_run=1 the scripts will be written but not) – If dry_run=2, the scripts will not be written nor submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
joint (the job involves multiple exposures) – run with desi_proc_joint_fit when not using tilenight. Default is False.
job (bool. Whether this is a joint fitting) – run with desi_proc_joint_fit when not using tilenight. Default is False.
system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu
use_specter –
bool –
True (optional. Default is False. If) –
specter (use) –
default. (otherwise use gpu_specter by) –
laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.
- Returns:
The same prow type and keywords as input except with modified values updated values for scriptname.
- Return type:
Table.Row or dict
Note
This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).
- desispec.workflow.procfuncs.define_and_assign_dependency(prow, calibjobs, use_tilenight=False)[source]¶
Given input processing row and possible calibjobs, this defines the JOBDESC keyword and assigns the dependency appropriate for the job type of prow.
- Parameters:
prow – ‘OBSTYPE’. A row must have column names for ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.
for (Table.Row or dict. Must include keyword accessible definitions) – ‘OBSTYPE’. A row must have column names for ‘JOBDESC’, ‘INT_DEP_IDS’, and ‘LATEST_DEP_ID’.
calibjobs – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
'nightlybias' (dict. Dictionary containing) – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
'ccdcalib' – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
'psfnight' – and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job. Each value that isn’t None must contain ‘INTID’, and ‘LATEST_QID’. If None, it assumes the dependency doesn’t exist and no dependency is assigned.
use_tilenight – for prestdstar, stdstar,and poststdstar steps for science exposures.
True (bool. Default is False. If) – for prestdstar, stdstar,and poststdstar steps for science exposures.
desi_proc_tilenight (use) – for prestdstar, stdstar,and poststdstar steps for science exposures.
- Returns:
The same prow type and keywords as input except with modified values updated values for ‘JOBDESC’, ‘INT_DEP_IDS’. and ‘LATEST_DEP_ID’.
- Return type:
Table.Row or dict
Note
This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).
- desispec.workflow.procfuncs.desi_proc_command(prow, system_name, use_specter=False, queue=None)[source]¶
Wrapper script that takes a processing table row (or dictionary with NIGHT, EXPID, OBSTYPE, JOBDESC, PROCCAMWORD defined) and determines the proper command line call to process the data defined by the input row/dict.
- Parameters:
prow (Table.Row or dict) – Must include keyword accessible definitions for ‘NIGHT’, ‘EXPID’, ‘JOBDESC’, and ‘PROCCAMWORD’.
system_name (str) – batch system name, e.g. cori-haswell, cori-knl, perlmutter-gpu
queue (str, optional) – The name of the NERSC Slurm queue to submit to. Default is None (which leaves it to the desi_proc default).
use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.
- Returns:
The proper command to be submitted to desi_proc to process the job defined by the prow values.
- Return type:
- desispec.workflow.procfuncs.desi_proc_joint_fit_command(prow, queue=None)[source]¶
Wrapper script that takes a processing table row (or dictionary with NIGHT, EXPID, OBSTYPE, PROCCAMWORD defined) and determines the proper command line call to process the data defined by the input row/dict.
- Parameters:
- Returns:
The proper command to be submitted to desi_proc_joint_fit to process the job defined by the prow values.
- Return type:
- desispec.workflow.procfuncs.flat_joint_fit(ptable, flats, internal_id, queue='realtime', reservation=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]¶
Wrapper function for desiproc.workflow.procfuns.joint_fit specific to the nightlyflat joint fit.
All variables are the same except:
Arg 'flats' is mapped to the prows argument of joint_fit. The joint_fit argument descriptor is pre-defined as 'nightlyflat'.
- desispec.workflow.procfuncs.get_type_and_tile(erow)[source]¶
Trivial function to return the OBSTYPE and the TILEID from an exposure table row
- Parameters:
erow –
keywords. (Table.Row or dict. Must contain 'OBSTYPE' and 'TILEID' as) –
- Returns:
tuple (str, str), corresponding to the OBSTYPE and TILEID values of the input erow.
- desispec.workflow.procfuncs.joint_fit(ptable, prows, internal_id, queue, reservation, descriptor, z_submit_types=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]¶
Given a set of prows, this generates a processing table row, creates a batch script, and submits the appropriate joint fitting job given by descriptor. If the joint fitting job is standard star fitting, the post standard star fits for all the individual exposures also created and submitted. The returned ptable has all of these rows added to the table given as input.
- Parameters:
ptable (Table) – The processing table where each row is a processed job.
prows (list or array of dict) – The rows corresponding to the individual exposure jobs that are inputs to the joint fit.
internal_id (int) – the next internal id to be used for assignment (already incremented up from the last used id number used).
queue (str) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.
reservation (str) – The reservation to submit jobs to. If None, it is not submitted to a reservation.
descriptor (str) – Description of the joint fitting job. Can either be ‘science’ or ‘stdstarfit’, ‘arc’ or ‘psfnight’, or ‘flat’ or ‘nightlyflat’.
z_submit_types (list of str, optional) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.
dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu
- Returns:
A tuple containing:
ptable, Table. The same processing table as input except with added rows for the joint fit job and, in the case of a stdstarfit, the poststdstar science exposure jobs.
joint_prow, dict. Row of a processing table corresponding to the joint fit job.
internal_id, int, the next internal id to be used for assignment (already incremented up from the last used id number used).
- Return type:
- desispec.workflow.procfuncs.make_joint_prow(prows, descriptor, internal_id)[source]¶
Given an input list or array of processing table rows and a descriptor, this creates a joint fit processing job row. It starts by copying the first input row, overwrites relevant columns, and defines the new dependencies (based on the input prows).
- Parameters:
prows – inputs to the joint fit.
are (list or array of dicts. The rows corresponding to the individual exposure jobs that) – inputs to the joint fit.
descriptor –
'stdstarfit' (str. Description of the joint fitting job. Can either be) –
'psfnight' –
'nightlyflat'. (or) –
internal_id (already incremented up from the last used id number used) –
int (already incremented up from the last used id number used) –
assignment (the next internal id to be used for) –
- Returns:
Row of a processing table corresponding to the joint fit job.
- Return type:
- desispec.workflow.procfuncs.make_redshift_prow(prows, tnight, descriptor, internal_id)[source]¶
Given an input list or array of processing table rows and a descriptor, this creates a joint fit processing job row. It starts by copying the first input row, overwrites relevant columns, and defines the new dependencies (based on the input prows).
- Parameters:
prows – the first steps of tilenight.
are (list or array of dicts. Unsumbitted rows corresponding to the individual prestdstar jobs that) – the first steps of tilenight.
tnight –
depends. (Table.Row object. Row corresponding to the tilenight job on which the redshift job) –
internal_id (already incremented up from the last used id number used) –
int (already incremented up from the last used id number used) –
assignment (the next internal id to be used for) –
- Returns:
Row of a processing table corresponding to the tilenight job.
- Return type:
- desispec.workflow.procfuncs.make_tnight_prow(prows, calibjobs, internal_id)[source]¶
Given an input list or array of processing table rows and a descriptor, this creates a joint fit processing job row. It starts by copying the first input row, overwrites relevant columns, and defines the new dependencies (based on the input prows).
- Parameters:
prows – the first steps of tilenight.
are (list or array of dicts. Unsumbitted rows corresponding to the individual prestdstar jobs that) – the first steps of tilenight.
calibjobs – None, with each table.Row() value corresponding to a calibration job on which the tilenight job depends.
or (dict. Dictionary containing keys that each corresponds to a Table.Row) – None, with each table.Row() value corresponding to a calibration job on which the tilenight job depends.
internal_id (already incremented up from the last used id number used) –
int (already incremented up from the last used id number used) –
assignment (the next internal id to be used for) –
- Returns:
Row of a processing table corresponding to the tilenight job.
- Return type:
- desispec.workflow.procfuncs.night_to_starting_iid(night=None)[source]¶
Creates an internal ID for a given night. The resulting integer is an 8 digit number. The digits are YYMMDDxxx where YY is the years since 2000, MM and DD are the month and day. xxx are 000, and are incremented for up to 1000 unique job ID’s for a given night.
- desispec.workflow.procfuncs.parse_previous_tables(etable, ptable, night)[source]¶
This takes in the exposure and processing tables and regenerates all the working memory variables needed for the daily processing script.
Used by the daily processing to define most of its state-ful variables into working memory. If the processing table is empty, these are simply declared and returned for use. If the code had previously run and exited (or crashed), however, this will all the code to re-establish itself by redefining these values.
- Parameters:
etable –
Table –
far. (Exposure table of all exposures that have been dealt with thus) –
ptable –
Table –
processed. (Processing table of all exposures that have been) –
night –
int (str or) –
taken. (the night the data was) –
- Returns:
A tuple containing:
arcs, list of dicts, list of the individual arc jobs used for the psfnight (NOT all the arcs, if multiple sets existed)
flats, list of dicts, list of the individual flat jobs used for the nightlyflat (NOT all the flats, if multiple sets existed)
sciences, list of dicts, list of the most recent individual prestdstar science exposures (if currently processing that tile)
calibjobs, dict. Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘badcol’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.
curtype, None, the obstype of the current job being run. Always None as first new job will define this.
lasttype, str or None, the obstype of the last individual exposure row to be processed.
curtile, None, the tileid of the current job (if science). Otherwise None. Always None as first new job will define this.
lasttile, str or None, the tileid of the last job (if science). Otherwise None.
internal_id, int, an internal identifier unique to each job. Increments with each new job. This is the latest unassigned value.
- Return type:
- desispec.workflow.procfuncs.recursive_submit_failed(rown, proc_table, submits, id_to_row_map, ptab_name=None, resubmission_states=None, reservation=None, dry_run=0)[source]¶
Given a row of a processing table and the full processing table, this resubmits the given job. Before submitting a job, it checks the dependencies for failures in the processing table. If a dependency needs to be resubmitted, it recursively follows dependencies until it finds the first job without a failed dependency and resubmits that. Then resubmits the other jobs with the new Slurm jobID’s for proper dependency coordination within Slurm.
- Parameters:
rown –
Table.Row –
resubmit. (the row of the processing table that you want to) –
proc_table –
Table –
job. (the processing table with a row per) –
submits –
int –
scheduler. (the number of submissions made to the queue. Used for saving files and in not overloading the) –
id_to_row_map (INTID's) – in the processing table.
dict (INTID's) – in the processing table.
ids (lookup dictionary where the keys are internal) – in the processing table.
ptab_name –
str –
saved. (the full pathname where the processing table should be) –
resubmission_states – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted
strings (list or array of) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted
a (each element should be a capitalized string corresponding to) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted
reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.
dry_run – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
int – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
nonzero (If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
- Returns:
A tuple containing:
proc_table: Table, a table with the same rows as the input except that Slurm and jobid relevant columns have been updated for those jobs that needed to be resubmitted.
submits: int, the number of submissions made to the queue. This is incremented from the input submits, so it is the number of submissions made from this function call plus the input submits value.
- Return type:
Note
This modifies the inputs of both proc_table and submits and returns them.
- desispec.workflow.procfuncs.science_joint_fit(ptable, sciences, internal_id, queue='realtime', reservation=None, z_submit_types=None, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, system_name=None)[source]¶
Wrapper function for desiproc.workflow.procfuns.joint_fit specific to the stdstarfit joint fit and redshift fitting.
All variables are the same except:
Arg 'sciences' is mapped to the prows argument of joint_fit. The joint_fit argument descriptor is pre-defined as 'science'.
- desispec.workflow.procfuncs.set_calibrator_flag(prows, ptable)[source]¶
- Sets the “CALIBRATOR” column of a procesing table row to 1 (integer representation of True)
for all input rows. Used within joint fitting code to flag the exposures that were input to the psfnight or nightlyflat for later reference.
- Parameters:
prows – inputs to the joint fit.
are (list or array of Table.Rows or dicts. The rows corresponding to the individual exposure jobs that) – inputs to the joint fit.
ptable –
job. (Table. The processing table where each row is a processed) –
- Returns:
The same processing table as input except with added rows for the joint fit job and, in the case of a stdstarfit, the poststdstar science exposure jobs.
- Return type:
Table
- desispec.workflow.procfuncs.still_a_dependency(dependency)[source]¶
Defines the criteria for which a dependency is deemed complete (and therefore no longer a dependency).
- Args:
- dependency, Table.Row or dict. Processing row corresponding to the required input for the job in prow.
This must contain keyword accessible values for ‘STATUS’, and ‘LATEST_QID’.
- Returns:
False if the criteria indicate that the dependency is completed and no longer a blocking factor (ie no longer a genuine dependency). Returns True if the dependency is still a blocking factor such that the slurm scheduler needs to be aware of the pending job.
- Return type:
- desispec.workflow.procfuncs.submit_batch_script(prow, dry_run=0, reservation=None, strictly_successful=False)[source]¶
Wrapper script that takes a processing table row and three modifier keywords and submits the scripts to the Slurm scheduler.
- Parameters:
prow – desispect.workflow.proctable.get_processing_table_column_defs()
in (Table.Row or dict. Must include keyword accessible definitions for processing_table columns found) – desispect.workflow.proctable.get_processing_table_column_defs()
dry_run – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
nonzero (int. If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.
strictly_successful – less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
processing (bool. Whether all jobs require all inputs to have succeeded. For daily) – less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
is (this) – less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
- Returns:
The same prow type and keywords as input except with modified values updated values for scriptname.
- Return type:
Table.Row or dict
Note
This modifies the input. Though Table.Row objects are generally copied on modification, so the change to the input object in memory may or may not be changed. As of writing, a row from a table given to this function will not change during the execution of this function (but can be overwritten explicitly with the returned row if desired).
- desispec.workflow.procfuncs.submit_redshifts(ptable, prows, tnight, internal_id, queue, reservation, dry_run=0, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, z_submit_types=None, system_name=None)[source]¶
Given a set of prows, this generates a processing table row, creates a batch script, and submits the appropriate tilenight job given by descriptor. The returned ptable has all of these rows added to the table given as input.
- Parameters:
ptable (Table) – The processing table where each row is a processed job.
prows (list or array of dict) – Unsubmitted prestdstar jobs that are first steps in tilenight.
tnight (Table.Row) – The processing table row of the tilenight job on which the redshifts depend.
internal_id (int) – the next internal id to be used for assignment (already incremented up from the last used id number used).
queue (str) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.
reservation (str) – The reservation to submit jobs to. If None, it is not submitted to a reservation.
dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
z_submit_types (list of str) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.
system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu
- Returns:
A tuple containing:
ptable, Table. The same processing table as input except with added rows for the joint fit job.
internal_id, int, the next internal id to be used for assignment (already incremented up from the last used id number used).
- Return type:
- desispec.workflow.procfuncs.submit_tilenight(ptable, prows, calibjobs, internal_id, queue, reservation, dry_run=0, strictly_successful=False, resubmit_partial_complete=True, system_name=None, use_specter=False, laststeps=None)[source]¶
Given a set of prows, this generates a processing table row, creates a batch script, and submits the appropriate tilenight job given by descriptor. The returned ptable has all of these rows added to the table given as input.
- Parameters:
ptable (Table) – The processing table where each row is a processed job.
prows (list or array of dict) – Unsubmitted prestdstar jobs that are first steps in tilenight.
calibjobs (dict) – Dictionary containing ‘nightlybias’, ‘ccdcalib’, ‘psfnight’ and ‘nightlyflat’. Each key corresponds to a Table.Row or None. The table.Row() values are for the corresponding calibration job.
internal_id (int) – the next internal id to be used for assignment (already incremented up from the last used id number used).
queue (str) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.
reservation (str) – The reservation to submit jobs to. If None, it is not submitted to a reservation.
dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
system_name (str) – batch system name, e.g. cori-haswell or perlmutter-gpu
use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.
laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.
- Returns:
A tuple containing:
ptable, Table. The same processing table as input except with added rows for the joint fit job.
tnight_prow, dict. Row of a processing table corresponding to the tilenight job.
internal_id, int, the next internal id to be used for assignment (already incremented up from the last used id number used).
- Return type:
- desispec.workflow.procfuncs.submit_tilenight_and_redshifts(ptable, sciences, calibjobs, lasttype, internal_id, dry_run=0, queue='realtime', reservation=None, strictly_successful=False, check_for_outputs=True, resubmit_partial_complete=True, z_submit_types=None, system_name=None, use_specter=False, laststeps=None)[source]¶
Takes all the state-ful data from daily processing and determines whether a tilenight job needs to be submitted.
- Parameters:
ptable (Table) – Processing table of all exposures that have been processed.
sciences (list of dict) – list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet.
lasttype (str or None) – the obstype of the last individual exposure row to be processed.
internal_id (int) – an internal identifier unique to each job. Increments with each new job. This is the smallest unassigned value.
dry_run (int, optional) – If nonzero, this is a simulated run. If dry_run=1 the scripts will be written or submitted. If dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
queue (str, optional) – The name of the queue to submit the jobs to. If None is given the current desi_proc default is used.
reservation (str, optional) – The reservation to submit jobs to. If None, it is not submitted to a reservation.
strictly_successful (bool, optional) – Whether all jobs require all inputs to have succeeded. For daily processing, this is less desirable because e.g. the sciences can run with SVN default calibrations rather than failing completely from failed calibrations. Default is False.
check_for_outputs (bool, optional) – Default is True. If True, the code checks for the existence of the expected final data products for the script being submitted. If all files exist and this is True, then the script will not be submitted. If some files exist and this is True, only the subset of the cameras without the final data products will be generated and submitted.
resubmit_partial_complete (bool, optional) – Default is True. Must be used with check_for_outputs=True. If this flag is True, jobs with some prior data are pruned using PROCCAMWORD to only process the remaining cameras not found to exist.
z_submit_types (list of str, optional) – The “group” types of redshifts that should be submitted with each exposure. If not specified or None, then no redshifts are submitted.
system_name (str) – batch system name, e.g. cori-haswell, cori-knl, permutter-gpu
use_specter (bool, optional) – Default is False. If True, use specter, otherwise use gpu_specter by default.
laststeps (list of str, optional) – A list of laststeps to pass as the laststeps argument to tilenight.
- Returns:
A tuple containing:
ptable, Table, Processing table of all exposures that have been processed.
sciences, list of dicts, list of the most recent individual prestdstar science exposures (if currently processing that tile). May be empty if none identified yet or we just submitted them for processing.
internal_id, int, if no job is submitted, this is the same as the input, otherwise it is incremented upward from from the input such that it represents the smallest unused ID.
- Return type:
- desispec.workflow.procfuncs.update_and_recurvsively_submit(proc_table, submits=0, resubmission_states=None, ptab_name=None, dry_run=0, reservation=None)[source]¶
Given an processing table, this loops over job rows and resubmits failed jobs (as defined by resubmission_states). Before submitting a job, it checks the dependencies for failures. If a dependency needs to be resubmitted, it recursively follows dependencies until it finds the first job without a failed dependency and resubmits that. Then resubmits the other jobs with the new Slurm jobID’s for proper dependency coordination within Slurm.
- Parameters:
proc_table –
Table –
job. (the processing table with a row per) –
submits –
int –
scheduler. (the number of submissions made to the queue. Used for saving files and in not overloading the) –
resubmission_states – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted
strings (list or array of) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted
a (each element should be a capitalized string corresponding to) – possible Slurm scheduler state, where you wish for jobs with that outcome to be resubmitted
ptab_name –
str –
saved. (the full pathname where the processing table should be) –
dry_run – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
int – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
nonzero (If) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
If (this is a simulated run. If dry_run=1 the scripts will be written or submitted.) – dry_run=2, the scripts will not be writter or submitted. Logging will remain the same for testing as though scripts are being submitted. Default is 0 (false).
reservation – str. The reservation to submit jobs to. If None, it is not submitted to a reservation.
- Returns:
A tuple containing:
proc_table: Table, a table with the same rows as the input except that Slurm and jobid relevant columns have been updated for those jobs that needed to be resubmitted.
submits: int, the number of submissions made to the queue. This is incremented from the input submits, so it is the number of submissions made from this function call plus the input submits value.
- Return type:
Note
This modifies the inputs of both proc_table and submits and returns them.
desispec.workflow.proctable¶
- desispec.workflow.proctable.default_obstypes_for_proctable()[source]¶
Defines the exposure types to be recognized by the workflow and saved in the processing table by default.
- Returns:
list. A list of default obstypes to be included in a processing table.
- desispec.workflow.proctable.default_prow()[source]¶
Creates a processing table row. The columns are filled with default values.
- Parameters:
None –
- Returns:
prow, dict. The output processing table row.
- desispec.workflow.proctable.erow_to_prow(erow)[source]¶
Converts an exposure table row to a processing table row. The columns unique to a processing table are filled with default values. If comments are made in COMMENTS or HEADERERR, those are ignored.
- Parameters:
erow – If comments are made in COMMENTS or HEADERERR, those are ignored.
table. (Table.Row or dict. An exposure table row. The row will be converted to a row of an processing) – If comments are made in COMMENTS or HEADERERR, those are ignored.
- Returns:
prow, dict. The output processing table row.
- desispec.workflow.proctable.exptable_to_proctable(input_exptable, obstypes=None)[source]¶
Converts an exposure table to a processing table and an unprocessed table. The columns unique to a processing table are filled with default values. If comments are made in COMMENTS or HEADERERR, those will be adjusted in the values stored in the processing table.
- Parameters:
input_exptable – comments are made in COMMENTS or HEADERERR, those will be adjusted in the values stored in the processing table.
If (Table. An exposure table. Each row will be converted to a row of an processing table.) – comments are made in COMMENTS or HEADERERR, those will be adjusted in the values stored in the processing table.
obstypes – added to the processing table).
therefore (list or np.array. Optional. A list of exposure OBSTYPE's that should be processed (and) – added to the processing table).
- Returns:
processing_table, Table. The output processing table. Each row corresponds with an exposure that should be processed. unprocessed_table, Table. The output unprocessed table. Each row is an exposure that should not be processed.
- desispec.workflow.proctable.get_processing_table_column_defs(return_default_values=False, overlap_only=False, unique_only=False)[source]¶
Contains the column names, data types, and default row values for a DESI processing table. It returns the names and datatypes with the defaults being given with an optional flag. Returned as 2 (or 3) lists.
- Parameters:
return_default_values –
returned. (bool. True if you want the default values) –
overlap_only –
tables. (bool. Only return the columns that are common to both processing and exposure) –
unique_only –
table. (bool. Only return columns that are not found in an exposure) –
- Returns:
colnames, list. List of column names for an processing table. coldtypes, list. List of column datatypes for the names in colnames. coldeflts, list. Optionally returned if return_default_values is True. List of default values for the corresponding colnames.
- desispec.workflow.proctable.get_processing_table_name(specprod=None, prodmod=None, extension='csv')[source]¶
Defines the default processing name given the specprod of the production and the optional extension.
- Parameters:
specprod –
None (str or None. The name of the production. If) –
variable. (it will be taken from the environment) –
prodmod – Used in daily workflow to add the night to the name and make it unique from other nightly tables.
it. (str. Additional str that can be added to the production table name to further differentiate) – Used in daily workflow to add the night to the name and make it unique from other nightly tables.
extension (str. The) – Default is ‘csv’.
extension – Default is ‘csv’.
- Returns:
str. The processing table name given the input night and extension.
- desispec.workflow.proctable.get_processing_table_path(specprod=None)[source]¶
Defines the default path to save a processing table. If specprod is not given, the environment variable ‘SPECPROD’ must exist.
- Parameters:
specprod –
None (str or None. The name of the production. If) –
variable. (it will be taken from the environment) –
- Returns:
- str. The full path to the directory where the processing table should be written (or is already written). This
does not including the filename.
- desispec.workflow.proctable.get_processing_table_pathname(specprod=None, prodmod=None, extension='csv')[source]¶
Defines the default pathname to save a processing table.
- Parameters:
specprod –
None (str or None. The name of the production. If) –
variable. (it will be taken from the environment) –
prodmod – Used in daily workflow to add the night to the name and make it unique from other nightly tables.
it. (str. Additional str that can be added to the production table name to further differentiate) – Used in daily workflow to add the night to the name and make it unique from other nightly tables.
extension (str. The) – Default is ‘csv’.
extension – Default is ‘csv’.
- Returns:
- str. The full pathname where the processing table should be written (or is already written). This
includes the filename.
- desispec.workflow.proctable.instantiate_processing_table(colnames=None, coldtypes=None, rows=None)[source]¶
Create an empty processing table with proper column names and datatypes. If rows is given, it inserts the rows into the table, otherwise it returns a table with no rows.
- Parameters:
colnames –
table. (list. List of column names for a procesing) –
coldtypes –
colnames. (list. List of column datatypes for the names in) –
rows – pairs that match the default column names and data types of the default exposure table.
value (list or np.array of Table.Rows or dicts. An iterable set of Table.Row's or dicts with keys/colnames and) – pairs that match the default column names and data types of the default exposure table.
- Returns:
- processing_table, Table. An astropy Table with the column names and data types for a DESI workflow processing
table. If the input rows was not None, it contains those rows, otherwise it has no rows.
- desispec.workflow.proctable.table_row_to_dict(table_row)[source]¶
Helper function to convert a table row to a dictionary, which is much easier to work with for some applications
- Parameters:
table_row – each key is a column name and the values are the column entry.
where (Table.Row or dict. The row of an astropy table that you want to convert into a dictionary) – each key is a column name and the values are the column entry.
- Returns:
out, dict. Dictionary where each key is a column name and the values are the column entry.
desispec.workflow.queue¶
- desispec.workflow.queue.any_jobs_not_complete(statuses, termination_states=None)[source]¶
Returns True if any of the job statuses in the input column of the processing table, statuses, are not complete (as based on the list of acceptable final states, termination_states, given as an argument. These should be states that are viewed as final, as opposed to job states that require resubmission.
- Parameters:
statuses – be a string.
should (Table.Column or list or np.array. The statuses in the processing table "STATUS". Each element) – be a string.
termination_states – by the Slurm scheduler that should be deemed terminal state.
returned (list or np.array. Each element should be a string signifying a state that is) – by the Slurm scheduler that should be deemed terminal state.
- Returns:
- bool. True if any of the statuses of the jobs given in statuses are NOT a member of the termination states.
Otherwise returns False.
- desispec.workflow.queue.get_resubmission_states()[source]¶
Defines what Slurm job failure modes should be resubmitted in the hopes of the job succeeding the next time.
Possible values that Slurm returns are:
CA or ca or CANCELLED for cancelled jobs will only show currently running jobs in queue unless times are explicitly given BF BOOT_FAIL Job terminated due to launch failure CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated. CD COMPLETED Job has terminated all processes on all nodes with an exit code of zero. DL DEADLINE Job terminated on deadline. F FAILED Job terminated with non-zero exit code or other failure condition. NF NODE_FAIL Job terminated due to failure of one or more allocated nodes. OOM OUT_OF_MEMORY Job experienced out of memory error. PD PENDING Job is awaiting resource allocation. PR PREEMPTED Job terminated due to preemption. R RUNNING Job currently has an allocation. RQ REQUEUED Job was requeued. RS RESIZING Job is about to change size. RV REVOKED Sibling was removed from cluster due to other cluster starting the job. S SUSPENDED Job has an allocation, but execution has been suspended and CPUs have been released for other jobs. TO TIMEOUT Job terminated upon reaching its time limit.
- Returns:
list. A list of strings outlining the job states that should be resubmitted.
- desispec.workflow.queue.get_termination_states()[source]¶
Defines what Slurm job states that are final and aren’t in question about needing resubmission.
Possible values that Slurm returns are:
CA or ca or CANCELLED for cancelled jobs will only show currently running jobs in queue unless times are explicitly given BF BOOT_FAIL Job terminated due to launch failure CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated. CD COMPLETED Job has terminated all processes on all nodes with an exit code of zero. DL DEADLINE Job terminated on deadline. F FAILED Job terminated with non-zero exit code or other failure condition. NF NODE_FAIL Job terminated due to failure of one or more allocated nodes. OOM OUT_OF_MEMORY Job experienced out of memory error. PD PENDING Job is awaiting resource allocation. PR PREEMPTED Job terminated due to preemption. R RUNNING Job currently has an allocation. RQ REQUEUED Job was requeued. RS RESIZING Job is about to change size. RV REVOKED Sibling was removed from cluster due to other cluster starting the job. S SUSPENDED Job has an allocation, but execution has been suspended and CPUs have been released for other jobs. TO TIMEOUT Job terminated upon reaching its time limit.
- Returns:
list. A list of strings outlining the job states that are considered final (without human investigation/intervention)
- desispec.workflow.queue.queue_info_from_qids(qids, columns='jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode', dry_run=0)[source]¶
Queries the NERSC Slurm database using sacct with appropriate flags to get information within a specified time window of all jobs submitted or executed during that time.
- Parameters:
jobids (list or array of ints) – Slurm QID’s at NERSC that you want to return information about.
columns (str) – Comma seperated string of valid sacct column names, in lower case. To be useful for the workflow, it should have MUST have columns “JOBID” and “STATE”. Other columns available that aren’t included in the default list are: jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode. Other options include: suspended,derivedexitcode,reason,priority,jobname.
dry_run (int) – Whether this is a simulated run or real run. If nonzero, it is a simulation and it returns a default table that doesn’t query the Slurm scheduler.
- Returns:
Table with the columns defined by the input variable ‘columns’ and information relating to all jobs submitted by the specified user in the specified time frame.
- Return type:
Table
- desispec.workflow.queue.queue_info_from_time_window(start_time=None, end_time=None, user=None, columns='jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode', dry_run=0)[source]¶
Queries the NERSC Slurm database using sacct with appropriate flags to get information within a specified time window of all jobs submitted or executed during that time.
- Parameters:
start_time (str) – String of the form YYYY-mm-ddTHH:MM:SS. Based on the given night and the earliest hour you want to see queue information about.
end_time (str) – String of the form YYYY-mm-ddTHH:MM:SS. Based on the given night and the latest hour you want to see queue information about.
user (str) – The username at NERSC that you want job information about. The default is an the environment name if if exists, otherwise ‘desi’.
columns (str) – Comma seperated string of valid sacct column names, in lower case. To be useful for the workflow, it should have MUST have columns “JOBID” and “STATE”. Other columns available that aren’t included in the default list are: jobid,jobname,partition,submit,eligible,start,end,elapsed,state,exitcode. Other options include: suspended,derivedexitcode,reason,priority,jobname.
dry_run (int) – Whether this is a simulated run or real run. If nonzero, it is a simulation and it returns a default table that doesn’t query the Slurm scheduler.
- Returns:
Table with the columns defined by the input variable ‘columns’ and information relating to all jobs submitted by the specified user in the specified time frame.
- Return type:
Table
- desispec.workflow.queue.update_from_queue(ptable, qtable=None, dry_run=0, ignore_scriptnames=False)[source]¶
Given an input prcessing table (ptable) and query table from the Slurm queue (qtable) it cross matches the Slurm job ID’s and updates the ‘state’ in the table using the current state in the Slurm scheduler system.
- Parameters:
ptable – have at least columnns ‘LATEST_QID’ and ‘STATUS’.
Must (Table. Processing table that contains the jobs you want updated with the most recent queue table.) – have at least columnns ‘LATEST_QID’ and ‘STATUS’.
qtable – to all jobs submitted by the specified user in the specified time frame.
relating (Table. Table with the columns defined by the input variable 'columns' and information) – to all jobs submitted by the specified user in the specified time frame.
ignore_scriptnames – want to check whether the scriptname matches the jobname return by the slurm scheduler.
not (bool. Default is False. Set to true if you do) – want to check whether the scriptname matches the jobname return by the slurm scheduler.
provided (The following are only used if qtable is not) –
- dry_run, int. Whether this is a simulated run or real run. If nonzero, it is a simulation and it returns a default
table that doesn’t query the Slurm scheduler.
- Returns:
- ptable, Table. The same processing table as the input except that the “STATUS” column in ptable for all jobs is
updated based on the ‘STATE’ in the qtable (as matched by “LATEST_QID” in the ptable and “JOBID” in the qtable).
desispec.workflow.redshifts¶
- desispec.workflow.redshifts.create_desi_zproc_batch_script(group, tileid=None, cameras=None, thrunight=None, nights=None, expids=None, healpix=None, survey=None, program=None, queue='regular', batch_opts=None, runtime=None, timingfile=None, batchdir=None, jobname=None, cmdline=None, system_name=None, max_gpuprocs=None, no_gpu=False, run_zmtl=False, no_afterburners=False)[source]¶
Generate a SLURM batch script to be submitted to the slurm scheduler to run desi_proc.
- Parameters:
group (str) – Description of the job to be performed. zproc options include: ‘perexp’, ‘pernight’, ‘cumulative’.
tileid (int) – The tile id for the data.
cameras (str or list of str) – List of cameras to include in the processing or a camword.
thrunight (int) – For group=cumulative, include exposures through this night
healpix (list of int) – healpixels to process (group=’healpix’)
queue (str) – Queue to be used.
batch_opts (str) – Other options to give to the slurm batch scheduler (written into the script).
runtime (str) – Timeout wall clock time in minutes.
timingfile (str) – Specify the name of the timing file.
batchdir (str) – can define an alternative location to write the file. The default is to SPECPROD under run/scripts/tiles/GROUP/TILE/NIGHT
jobname (str) – name to save this batch script file as and the name of the eventual log file. Script is save within the batchdir directory.
cmdline (str or list of str) – Complete command as would be given in terminal to run the desi_zproc, or list of args. Can be used instead of reading from argv.
system_name (str) – name of batch system, e.g. cori-haswell, cori-knl
max_gpuprocs (int) – Number of gpu processes
no_gpu (bool) – Default false. If true it doesn’t use GPU’s even if available.
run_zmtl (bool) – Default false. If true it runs zmtl.
no_afterburners (bool) – Default false. If true it doesn’t run afterburners.
- Returns:
the full path name for the script written.
- Return type:
scriptfile
Note
- batchdir and jobname can be used to define an alternative pathname, but
may not work with assumptions in the spectro pipeline.
- desispec.workflow.redshifts.get_zpix_redshift_script_pathname(healpix, survey, program)[source]¶
Return healpix-based coadd+redshift+afterburner script pathname
- desispec.workflow.redshifts.get_ztile_relpath(tileid, group, night=None, expid=None)[source]¶
Determine the relative output directory of the tile redshift batch script for spectra+coadd+redshifts for a tile
- desispec.workflow.redshifts.get_ztile_script_pathname(tileid, group, night=None, expid=None)[source]¶
Generate the pathname of the tile redshift batch script for spectra+coadd+redshifts for a tile
- desispec.workflow.redshifts.get_ztile_script_suffix(tileid, group, night=None, expid=None)[source]¶
Generate the suffix of the tile redshift batch script for spectra+coadd+redshifts for a tile
- desispec.workflow.redshifts.read_minimal_exptables_columns(nights=None, tileids=None)[source]¶
Read exposure tables while handling evolving formats
- Parameters:
- Returns exptable with just columns TILEID, NIGHT, EXPID, ‘CAMWORD’,
‘BADCAMWORD’, filtered by science exposures with LASTSTEP=’all’ and TILEID>=0
- Note: the returned table is the full pipeline exposures table. It is trimmed
to science exposures that have LASTSTEP==’all’
desispec.workflow.schedule¶
Tools for scheduling MPI jobs using mpi4py.
desispec.workflow.tableio¶
- desispec.workflow.tableio.ensure_scalar(val, joinsymb='|', comma_replacement=';')[source]¶
Ensures that the object in val is a scalar that can be save to a Table cell (i.e. row of a column or column of a row). If the it is an array or list, it uses joinsymb to turn them into a single string.
- Parameters:
val (a scalar datatype, list, or array) – The value to be converted to a scalar quantity (returning the val if it is already a scalar).
joinsymb (str) – A string symbol other than comma that will be used to join the multiple values of a list or array.
comma_replacement (str) – A string symbol that should be used to replace any existing commas in the data, such that the value can be saved in a csv format.
- Returns:
The output string which is a scalar quantity capable of being written to a single table cell (in a csv or fits file, for example).
- Return type:
val or outstr, any scalar type or string
- desispec.workflow.tableio.guess_default_by_dtype(typ)[source]¶
Returns a default value given a data type. To be used in filling a table if no default is given.
- Parameters:
typ (DataType) – The datatype of the element you want a default value for.
- Returns:
default value for that type. Can be int, float, str, list, or array. If it can’t guess, it returns the integer -99 .
- Return type:
number
- desispec.workflow.tableio.load_table(tablename=None, tabletype=None, joinsymb='|', verbose=False, process_mixins=True, use_specprod=True, suppress_logging=False)[source]¶
Workflow function to read in exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are generated from strings using the joinsymb. It reads from the file given by tablename (or the default for table of type tabletype).
- Parameters:
tablename (str) – Full pathname of where the table should be saved, including the extension. Originally save to
*.temp.{ext}
and then moved to*.{ext}
. If None, it looks up the default for typetable. If tabletype is None it uses this to try and identify the tabletype and uses that to get the default column names and types.tabletype (str) – ‘exptable’, ‘proctable’, or ‘unproctable’. Used if tablename is None to get the default name for the type of table. Also used to get the column datatypes and defaults.
joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.
verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.
process_mixins (bool) – Whether to look for and try to split strings into lists/arrays. The default is True. Warning: The exposure and processing tables have default data types which are multi-value. If this is set to False, the default data types will be incorrect and issues are likely to arise.
use_specprod (bool) – If True and tablename not specified and tabletype is exposure table, this looks for the table in the SPECPROD rather than the exptab repository. Default is True.
suppress_logging (bool) – If True, the log.info() messages are skipped. This is useful in scripts looping over many tables to reduce the amount of things printed to the screen.
- Returns:
table – Either exposure table or processing table that was loaded from tablename (or from default name based on tabletype). Returns None if the file doesn’t exist.
- Return type:
Table
- desispec.workflow.tableio.load_tables(tablenames=None, tabletypes=None, verbose=False)[source]¶
Workflow function to read in multiple exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are generated from strings using the joinsymb. It reads from the files given by tablenames (or the default for tables of types in tabletypes).
- Parameters:
tablename (list/array of str) – List or array of the full pathnames of where the tables should be saved, including the extension.
tabletype (list/array of str) – List or array of the table types, which are used if tablenames is None to get the default name for the type of table. They are also used to get the column datatypes and defaults.
verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.
- Returns:
tabs – Either exposure table or processing table that was loaded from tablename (or from default name based on tabletype). Returns None if the file doesn’t exist.
- Return type:
list of Table
- desispec.workflow.tableio.process_column(data, typ, mask=None, default=None, joinsymb='|', process_mixins=True, comma_replacement=';', verbose=False)[source]¶
Used with load_table to process a Table.Column after being read in. It fills in masked values with defaults, and identifies and splits mixin columns (columns that should be a list/array) back into their list/array from their string representation.
- Parameters:
data (Table.Column or Table.MaskedColumn) – Column of data to be checked for masked rows (to be filled with default) and string-ed versions of lists/arrays that need to be expanded out.
typ (DataType) – The expected datatype of the data in data. May differ from the type of the input data, in which case the data will be transformed.
mask (np.array) – A mask array with True in row elements of the input data array that are masked and False in row elements that are not masked.
default (any type) – The default value to be used for masked rows.
joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.
process_mixins (bool) – Whether to look for and try to split strings into lists/arrays. The default is True. Warning: The exposure and processing tables have default data types which are multi-value. If this is set to False, the default data types will be incorrect and issues are likely to arise.
comma_replacement (str) – Replace instances of this symbol with commas when loading scalar columns in a table, as e.g. BADAMPS is used in the pipeline and symbols like ‘;’ are problematic on the command line.
verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.
- Returns:
col, list or np.array – A new data vector similar to input ‘data’ except with masked values filled in and mixin strings expanded back into np.array’s.
DataType – The data type of a row element in the return col.
- desispec.workflow.tableio.split_str(val, joinsymb='|', comma_replacement=';')[source]¶
Attempts to intelligently interpret an input scalar. If it is a string it looks to see if it was a list or array objects that was joined to be a single string using joinsymb. If it identifies that, it will split that into the original list/array. Otherwise it will return the val as-is.
- Parameters:
val (any datatype) – The input to be checked to see if it is in fact a list/array that was joined into a string for saving in a Table.
joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.
comma_replacement (str) – Replace instances of this symbol with commas when loading ONLY scalar columns in a table, as e.g. BADAMPS is used in the pipeline and symbols like ‘;’ are problematic on the command line. Comment arrays do not need to be converted back and forth.
- Return type:
val or split_list, any datatype or np.array.
- desispec.workflow.tableio.standardize_tabletype(tabletype)[source]¶
Given the user defined type of table it returns the proper ‘tabletype’ expected by the pipeline
- desispec.workflow.tableio.translate_type_to_pathname(tabletype, use_specprod=True)[source]¶
Given the type of table it returns the proper file pathname
- Parameters:
tabletype (str) – Allows for a flexible number of input options, but should refer to either the ‘exposure’, ‘processing’, or ‘unprocessed’ table types.
use_specprod (bool) – If True and tablename not specified and tabletype is exposure table, this looks for the table in the SPECPROD rather than the exptab repository. Default is True.
- Returns:
tablename – Full pathname including extension of the table type. Uses environment variables to determine the location.
- Return type:
- desispec.workflow.tableio.write_table(origtable, tablename=None, tabletype=None, joinsymb='|', overwrite=True, verbose=False, comma_replacement=';', write_empty=False, use_specprod=True)[source]¶
Workflow function to write exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are reduced to strings using the joinsymb. It writes to a temp file before moving the fully written file to the name given by tablename (or the default for table of type tabletype).
- Parameters:
origtable (Table) – Either exposure table or processing table.
tablename (str) – Full pathname of where the table should be saved, including the extension. Originally save to
*.temp.{ext}
and then moved to*.{ext}
. IfNone
, it looks up the default for typetable.tabletype (str) – Used if tablename is None to get the default name for the type of table.
joinsymb (str) – The symbol used to join values in a list/array when saving. Should not be a comma.
overwrite (bool) – Whether to overwrite the file on disk if it already exists. Default is currently True.
verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.
write_empty (bool) – Whether to write an empty table to disk. The default is False. Warning: code is less robust to column datatypes on read/write if the table is empty. May cause issues if this is set to True.
comma_replacement (str) – Replace instances of this symbol with commas when loading scalar columns in a table, as e.g. BADAMPS is used in the pipeline and symbols like ‘;’ are problematic on the command line.
use_specprod (bool) – If True and tablename not specified and tabletype is exposure table, this looks for the table in the SPECPROD rather than the exptab repository. Default is True.
- desispec.workflow.tableio.write_tables(tables, tablenames=None, tabletypes=None, write_empty=False, verbose=False, overwrite=True)[source]¶
Workflow function to write multiple exposure, processing, and unprocessed tables. It allows for multi-valued table cells, which are reduced to strings. It writes to a temp file before moving the fully written file to the name given by tablenames (or the default for table of types tabletypes).
- Parameters:
tables (list/array of Table) – List or array of exposure tables, unprocessed tables, and/or processing table.
tablenames (list/array of str) – List or array of the full pathnames to where the tables should be saved, including the extension. If None, it looks up the default for each of tabletypes.
tabletype (list/array of str) – List or array of table types to be used if tablenames is None to get the default name for each type of table.
write_empty (bool) – Whether to write an empty table to disk. The default is False. Warning: code is less robust to column datatypes on read/write if the table is empty. May cause issues if this is set to True.
overwrite (bool) – Whether to overwrite the file on disk if it already exists. Default is currently True.
verbose (bool) – Whether to give verbose amounts of information (True) or succinct/no outputs (False). Default is False.
desispec.workflow.timing¶
- desispec.workflow.timing.during_operating_hours(dry_run=False, starthour=None, endhour=None)[source]¶
Determines if the desi_daily_proc_manager should be running or not based on the time of day. Can be overwridden with dry_run for testing purposes.
- Parameters:
dry_run – bool. If true, this is a simulation so return True so that the simulation can proceed.
starthour – str or int. The number of hours (between 0 and 24) after midnight.
endhour – str or int. The number (between 0 and 24) of hours after midnight. Assumes an endhour smaller than starthour implies the following day.
- Returns:
bool. True if dry_run is true OR if the current time is between the starthour and endhour.
- desispec.workflow.timing.ensure_tucson_time()[source]¶
Define the start and end of a ‘night’ based on times at the mountain. So ensure that the times are with respect to Arizona.
- desispec.workflow.timing.get_nightly_end_time()[source]¶
Defines when a night ends for desi_daily_proc_manager in local Tucson time. Once this time is exceeded, the manager will enter into queue cleanup mode and then exit when the jobs have finished. End of night is altered slightly for summer vs. winter.
- Returns:
int. The number of hours after midnight that signifies the end of a night of observing.
- Return type:
end_night
- desispec.workflow.timing.get_nightly_start_time()[source]¶
Defines the time of day that the desi_daily_proc_manager should start (in Tucson local time). Before this time, the manager being woken by a cron job will exit immediately. Selected to give plenty of time between end of night and start of the next, but early enough to catch and begin running afternoon calibrations.
- Returns:
int. The number of hours after midnight that signifies the start of a new night of observing.
- Return type:
14
- desispec.workflow.timing.nersc_end_time(night=None, endhour=None)[source]¶
Transforms a night and time into a YYYY-MM-DD[THH:MM[:SS]] time string Slurm can interpret. Correctly accounts for the fact that the night is defined starting at Noon on a given day.
- Parameters:
night – str or int. In the form YYYMMDD, the night the jobs are being run.
endhour – str or int. The number (between 0 and 24) of hours after midnight where you stop submitting jobs to the queue.
- Returns:
MM:SS. Based on the given night and endhour
- Return type:
str. String of the form YYYY-mm-ddTHH
- desispec.workflow.timing.nersc_format_datetime(timetup=None)[source]¶
Given a time tuple from the time module, this will return a string in the proper format to be properly interpreted by the NERSC Slurm queue scheduler.
- Parameters:
timetup – tuple. A time.time() tuple object representing the time you want to trasnform into a Slurm readable string.
- Returns:
MM:SS.
- Return type:
str. String of the form YYYY-mm-ddTHH
- desispec.workflow.timing.nersc_start_time(night=None, starthour=None)[source]¶
Transforms a night and time into a YYYY-MM-DD[THH:MM[:SS]] time string Slurm can interpret
- Parameters:
night – str or int. In the form YYYMMDD, the night the jobs are being run.
starthour – str or int. The number of hours (between 0 and 24) after midnight where you began submitting jobs to the queue.
- Returns:
MM:SS. Based on the given night and starthour
- Return type:
str. String of the form YYYY-mm-ddTHH
desispec.workflow.utils¶
- desispec.workflow.utils.check_running(proc_name='desi_daily_proc_manager', suppress_outputs=False)[source]¶
Check if the given process name is running. Default is desi_daily_proc_manager.
- Parameters:
proc_name –
tables. (str. The name of the process as it would appear in the os's process) –
suppress_outputs –
printed. (bool. True if you don't want to anything to be) –
- Returns:
- running, bool. True if the process name was found in the list of processes and has a pid different that
the current process (signifying a second instance of that program).
- desispec.workflow.utils.define_variable_from_environment(env_name, var_descr)[source]¶
Returns the environment variable if it exists, otherwise raises an error telling the user that a variable must be specified either directly or by defining the environment variable. It exits with exit code 1 if no environment variable exists.
- Parameters:
env_name –
variable. (str. The name of the environment) –
var_descr –
env_name. (str. A description of the variable you are trying to define with environment variable) –
- Returns:
- str or Nothing. If the environment variable exists, it returns the value of that environment variable.
Otherwise it raises an exit code with status 1.
- desispec.workflow.utils.get_json_dict(reqpath)[source]¶
Return a dictionary representation of the json file at the specified location. If it doesn’t exist, return an empty dictionary.
- Parameters:
reqpath – as it is accurate based on the current working directory.
given (str. The full pathname including file name of the json file. A relative path can be) – as it is accurate based on the current working directory.
long (so) – as it is accurate based on the current working directory.
- Retrun:
req_dict, dict. A dictionary with keys and values defined by the json file.
- desispec.workflow.utils.get_printable_banner(input_str=None)[source]¶
Returns a string that when printed shows a banner with the given string in the center.
- Parameters:
input_str – str. Any string of moderate length roughly less than 30 that you want placed inside a pound/hashtag (#) banner.
- Returns:
str. A banner comprised of ascii pound symbols (#) and the given string in the center.
- Return type:
banner
- desispec.workflow.utils.globpath(*args)[source]¶
Helper function that takes the same arguments as os.path.join (i.e. a comma separated set of directory names, paths, and file names that can be joined to created a single unified file path), and returns the glob’ed list of files in that location that matched the specified parameters.
- Parameters:
args (i.e. pathjoin) – for which you want a list of files in that location. Any of the strings can be wildcards used by glob.glob, so long as they don’t confuse os.path.join().
os.path.join (undetermined number of input args that are each a string. The values to be given to) – for which you want a list of files in that location. Any of the strings can be wildcards used by glob.glob, so long as they don’t confuse os.path.join().
- Returns:
- list. Sorted list of files in the location defined by the input args that are consistent with all wildcards (if
any). Ignores Mac file .DS_STORE. If location doesn’t exist, it returns an empty list.
- desispec.workflow.utils.listpath(*args)[source]¶
Helper function that takes the same arguments as os.path.join (i.e. a comma separated set of directory names, paths, and file names that can be joined to created a single unified file path), and returns the list of files in that location.
- Parameters:
args (i.e. pathjoin) – for which you want a list of files in that location.
os.path.join (undetermined number of input args that are each a string. The values to be given to) – for which you want a list of files in that location.
- Returns:
- list. Sorted list of files in the location defined by the input args. Ignores Mac file .DS_STORE. If location
doesn’t exist, it returns an empty list.
- desispec.workflow.utils.sleep_and_report(sleep_duration, message_suffix='', logfunc=<built-in function print>, dry_run=False)[source]¶
Sleeps for the given number of seconds, printing a message before and afterward
- Parameters:
sleep_duration – int or float. Number of seconds for the process to sleep.
message_suffix – str. The message to be printed after “Sleeping XXs (suffix)” prior to sleeping.
logfunc – func. Default is print. The function used to serve the message to the user.
dry_run – bool. Default is False. Whether to perform the action (dry_run=False) or pretend (True).
- desispec.workflow.utils.verify_variable_with_environment(var, var_name, env_name)[source]¶
Helper function that assigns a variable based on the inputs and gives relevant outputs to understand what is being done. If the variable is defined, it will make sure that the environment variable (used by some lower-level code) is consistent before returning the user specified value. If it is not specified, then the environment variable is used. If the environment variable is also undefined, then it gives a useful output and then exits.
- Parameters:
var –
variable. (str. The name of the environment variable that would hold the value relevant to the var) –
var_name (used exclusively for outputting useful messages) –
variable (str. The name of the) –
env_name –
variable. –
- Returns:
var, any type. Either the input var if that is not NoneType. Otherwise the value from the environment variable. If neither is defined it exits with status 1 rather than returning anything.
desispec.xytraceset¶
Lightweight wrapper class for trace coordinates and wavelength solution, to be returned by read_xytraceset()
.
desispec.zcatalog¶
The zcatalog.py script contains the following utility functions for redshift catalogs:
find_primary_spectra:
Given an input table with possibly multiple spectra per TARGETID, this function returns arrays for a primary flag (whether a spectrum is the primary [“best”] spectrum based on the ZWARN value and on a sort column [default=’TSNR2_LRG’]) and for the number of spectra per TARGETID.
Usage:
nspec, spec_primary = find_primary_spectra(table, sort_column = 'TSNR2_LRG')
create_summary_catalog:
This function combines individual redshift catalogs from a given release into a single compilation catalog. The output catalog is saved in a FITS file in a user-specified location and filename.
This function can be used for ‘fuji’ (EDR) or ‘guadalupe’ (Main) by setting the keyword specprod. By default, this function aggregates all the columns for the redshift catalogs, and adds columns to quantify the number of coadded spectra listed in the catalogs for each TARGETID and to identify the primary (“best”) spectrum out of them using the find_primary_spectra function. Optionally, the script can return a list of pre-selected “summary” column, or a list of user-specified columns via the columns_list keyword.
Usage:
create_summary_catalog(specprod, specgroup = 'zpix', all_columns = True, columns_list = None, output_filename = './zcat-all.fits')
Ragadeepika Pucha, Stephanie Juneau, and DESI data team Version: 2022, March 31st
- desispec.zcatalog._get_survey_program_from_filename(filename)[source]¶
Return SURVEY,PROGRAM parsed from zpix/ztile filename; fragile!
- desispec.zcatalog.create_summary_catalog(specgroup, indir=None, specprod=None, all_columns=True, columns_list=None, output_filename=None)[source]¶
This function combines all the individual redshift catalogs for either ‘zpix’ or ‘ztile’ with the desired columns (all columns, or a pre-selected list, or a user-given list). It further adds ‘NSPEC’ and ‘PRIMARY’ columns, two for SV(or MAIN), and two for the entire combined redshift catalog.
- Parameters:
specgroup (str) – The option to run the code on ztile* files or zpix* files. It can either be ‘zpix’ or ‘ztile’
indir (str) – Input directory to look for zpix/ztile files.
specprod (str) – Internal Release Name for the DESI Spectral Release. Used to derive input directory if indir is not provided.
all_columns (bool) – Whether or not to include all the columns into the final table. Default is True.
columns_list (list) – If all_columns = False, list of columns to include in the final table. If None, a list of pre-decided summary columns will be used. Default is None. The ‘SV/MAIN’ primary flag columns as well as the primary flag columns for the entire catalog witll be included.
output_filename (str) – Path+Filename for the output summary redshift catalog. The output FITS file will be saved at this path. If not specified, the output filename will be derived from specgroup and $SPECPROD
- Returns:
The function saves a FITS file at the location and file name specified by output_filename with the summary redshift catalog in HDU1.
- Return type:
None
- desispec.zcatalog.find_primary_spectra(table, sort_column='TSNR2_LRG')[source]¶
Function to select the best “primary” spectrum for objects with multiple (coadded) spectra.
The best spectrum is selected based on the following steps:
Spectra with ZWARN=0 are given top preference.
Among multiple entries with ZWARN=0, spectra with the highest value of ‘sort_column’ are given the preference.
If there are no spectra with ZWARN=0, spectra with the highest value of ‘sort_column’ are given the preference.
- Parameters:
table (Numpy array or Astropy Table) – The input table should be a redshift catalog, with at least the following columns: ‘TARGETID’, ‘ZWARN’ and the sort_column (Default:’TSNR2_LRG’) Additional columns will be ignored.
sort_column (str) – Column name that will be used to select the best spectrum. Default = ‘TSNR2_LRG’ The higher values of the column will be considered as better (descending order).
- Returns:
nspec (Numpy int array) – Array of number of entries available per target
spec_primary (Numpy bool array) – Array of spec_primary (= TRUE for the best spectrum)
- desispec.zcatalog.update_table_columns(table, specgroup='zpix', all_columns=True, columns_list=None)[source]¶
This function fills the
*TARGET
masked columns and returns the final table with the required columns.- Parameters:
table (Astropy Table) – A table.
specgroup (str) – The option to run the code on ztile* files or zpix* files. It can either be ‘zpix’ or ‘ztile’. Default is ‘zpix’
all_columns (bool) – Whether or not to include all the columns into the final table. Default is True.
columns_list (list) – If all_columns = False, list of columns to include in the final table. If None, a list of pre-decided summary columns will be used. Default is None. The ‘SV/MAIN’ primary flag columns as well as the primary flag columns for the entire catalog witll be included.
- Returns:
t_final – Final table with non-masked columns with required columns.
- Return type:
Astropy Table
desispec.zmtl¶
Post-redrock ML processing for LyA Quasar object identification for creating zmtl files to be used by the merged target list (MTL) updates.
- desispec.zmtl.add_abs_data(zmtl, coaddname)[source]¶
Add the MgII absorption line finder data to the input zmtl array.
- Parameters:
zmtl (:class:'~numpy.array`) – The structured array that was created by make_new_zmtl()
coaddname (class:str) – The name of the coadd file corresponding to the redrock file used in make_new_zmtl()
- Returns:
The zmtl array with MgII Absorption data included in the columns:
Z_ABS - The highest redshift of MgII absorption
Z_ABS_CONF - The confidence value for this redshift.
- Return type:
array
Notes
The original function was written by Lucas Napolitano (LGN) and modified for this script by Eleanor Lyke (EBL).
- desispec.zmtl.add_qn_data(zmtl, coaddname, qnp_model, qnp_lines, qnp_lines_bal)[source]¶
Apply the QuasarNP model to the input zmtl and add data to columns.
- Parameters:
zmtl (
array
) – The structured array that was created by make_new_zmtl()coaddname (
str
) – The name of the coadd file corresponding to the redrock file used in make_new_zmtl()qnp_model (
h5.array
) – The array containing the pre-trained QuasarNP model.qnp_lines (
list
) – A list containing the names of the emission lines that quasarnp.process_preds() should use.qnp_lines_bal (
list
) – A list containing the names of the emission lines to check for BAL troughs.
- Returns:
The zmtl array with QuasarNP data included in the columns:
Z_QN - The best QuasarNP redshift for the object
Z_QN_CONF - The confidence of Z_QN
IS_QSO_QN - A binary flag indicated object is a quasar
- Return type:
array
- desispec.zmtl.add_sq_data(zmtl, coaddname, squeze_model)[source]¶
Apply the SQUEzE model to the input zmtl and add data to columns.
- Parameters:
zmtl (
array
) – The structured array that was created by make_new_zmtl()coaddname (class:str) – The name of the coadd file corresponding to the redrock file used in make_new_zmtl()
squeze_model (
numpy.array
) – The loaded SQUEzE model file
- Returns:
The zmtl array with SQUEzE data included in the columns:
Z_SQ - The best redshift from SQUEzE for each object.
Z_SQ_CONF - The confidence value of this redshift.
- Return type:
array
- desispec.zmtl.add_tileqa_data(zmtl, tileqafile)[source]¶
Modifies zmtl[‘ZWARN’] in-place to add tile QA flags
- Parameters:
zmtl (
array
) – The structured array that was created by make_new_zmtl()tileqafile (
str
) – The full filepath of the tile-qa filenot (Returns boolean array for whether targets were flagged as bad (True) or) –
- desispec.zmtl.create_zmtl(zmtldir, outputdir, tile=None, night=None, petal_num=None, qn_flag=False, qnp_model=None, qnp_model_file=None, qnp_lines=None, qnp_lines_bal=None, sq_flag=False, squeze_model=None, squeze_model_file=None, abs_flag=False, zcomb_flag=False)[source]¶
This will create a single zmtl file from a set of user inputs.
- Parameters:
zmtldir (
str
) –- If any of tile, night or petal_num are
None
: The name of a redrock redrock file.
- If none of tile, night and petal_num are
None
: The root directory from which to read redrock and coadd spectro files. The full directory is constructed as zmtldir + tile + night, with files redrock-/coadd-petal_num`*`night.fits.
- If any of tile, night or petal_num are
outputdir (
str
) –- If any of tile, night or petal_num are
None
: The name of an output file.
- If none of tile, night and petal_num are
None
: The output directory to which to write the output file. The full directory is constructed as outputdir + tile + night, with file zmtl-petal_num`*`night.fits.
- If any of tile, night or petal_num are
tile (
int
) – The TILEID of the tile to process.night (
int
) – The date associated with the observation of the ‘tile’ used. * Must be in YYYYMMDD formatpetal_num (
int
) – If ‘all_petals’ isn’t used, the single petal to create a zmtl for.qn_flag (
bool
, optional) – Flag to add QuasarNP data (or not) to the zmtl file.qnp_model (
h5 array
, optional) – The QuasarNP model file to be used for line predictions.qnp_model_file (
str
, optional) – File from which to load the QuasarNP model (qnp_model), qnp_lines and qnp_lines_bal if qnp_model isNone
. Also written to the output header of the zmtl file.qnp_lines (
list
, optional) – The list of lines to use in the QuasarNP model to test against.qnp_lines_bal (
list
, optional) – The list of BAL lines to use for QuasarNP to identify BALs.sq_flag (
bool
, optional) – Flag to add SQUEzE data (or not) to the zmtl file.squeze_model (
numpy.array
, optional) – The numpy array for the SQUEzE model file.squeze_model_file (
str
, optional) – File from which to load the SQUEzE model if squeze_model isNone
. Also written to the output header of the zmtl file.abs_flag (
bool
, optional) – Flag to add MgII Absorption data (or not) to the zmtl file.zcomb_flag (
bool
, optional) – Flag if a combined redshift (or not) was added to the zmtl file.
Notes
Writes a FITS catalog that incorporates redrock, and a range of afterburner redshifts and confidence values. This will write to the same directory of the redrock and coadd files unless a different output directory is passed.
- desispec.zmtl.get_qn_model_fname(qnmodel_fname=None)[source]¶
Convenience function to grab the $QN_MODEL_FILE environment variable.
- Parameters:
qnmodel_fname (
str
, optional, defaults to $QN_MODEL_FILE) – If qnmodel_fname is passed, it is returned from this function. If it’s not passed, the $QN_MODEL_FILE variable is returned.- Returns:
not passed, the directory stored in the $QN_MODEL_FILE environment variable is returned prepended to the default filename.
- Return type:
- desispec.zmtl.get_sq_model_fname(sqmodel_fname=None)[source]¶
Convenience function to grab the $SQ_MODEL_FILE environment variable.
- Parameters:
sqmodel_fname (
str
, optional, defaults to $SQ_MODEL_FILE) – If sqmodel_fname is passed, it is returned from this function. If it’s not passed, the $SQ_MODEL_FILE environment variable is returned.- Returns:
If sqmodel_fname is passed, it is returned from this function. If it’s not passed, the directory stored in the $SQ_MODEL_FILE environment variable is returned.
- Return type:
- desispec.zmtl.load_qn_model(model_filename)[source]¶
Convenience function to load the QuasarNP model and line lists.
- Parameters:
model_filename (
str
) – The filename and path of the QuasarNP model. Either input by user or defaults to get_qn_model_fname().- Returns:
array
– The QuasarNP model file loaded as an array.array
– An array of the emission line names to be used for quasarnp.process_preds().array
– An array of the BAL emission line names to be used by quasarnp.process_preds().
- desispec.zmtl.load_sq_model(model_filename)[source]¶
Convenience function for loading the SQUEzE model file.
- Parameters:
model_filename (
str
) – The filename and path of the SQUEzE model file. Either input by user or defaults to get_sq_model_fname().- Returns:
A numpy array of the SQUEzE model.
- Return type:
array
Notes
The input model file needs to be in the json file format.
- desispec.zmtl.make_new_zmtl(redrockname, qn_flag=False, sq_flag=False, abs_flag=False, zcomb_flag=False)[source]¶
Make the initial zmtl array with redrock data.
- Parameters:
redrockname (
str
) – Full filename and path for the redrock file to process.qn_flag (
bool'
optional) – Flag to add QuasarNP data (or not) to the zmtl file.sq_flag (
bool
, optional) – Flag to add SQUEzE data (or not) to the zmtl file.abs_flag (
bool
, optional) – Flag to add MgII Absorption data (or not) to the zmtl file.zcomb_flag (
bool
, optional) – Flag if a combined redshift (or not) was added to the zmtl file.
- Returns:
A zmtl in the official format (zmtldatamodel) compiled from the tile’, ‘night’, and ‘petal_num’, in `zmtldir. If the redrock file for that petal doesn’t exist, returns
False
.- Return type:
array
or bool
- desispec.zmtl.tmark(istring)[source]¶
A function to mark the time an operation starts or ends.
- Parameters:
istring (:class:'str') – The input string to print to the terminal.
Notes
A string with the date and time in ISO 8061 standard followed by the ‘istring’.
- desispec.zmtl.write_zmtl(zmtl, outputname, qn_flag=False, sq_flag=False, abs_flag=False, zcomb_flag=False, qnp_model_file=None, squeze_model_file=None)[source]¶
Writes the zmtl structured array out as a FITS file.
- Parameters:
zmtl (
array
) – The structured array that was created by make_new_zmtl()outputname (
str
) – The full filepathname of the zmtl output file.qn_flag (
bool
) – Flag if QuasarNP data (or not) was added to the zmtl file.sq_flag (
bool
) – Flag if SQUEzE data (or not) was added to the zmtl file.abs_flag (
bool
) – Flag if MgII Absorption data (or not) was added to the zmtl file.zcomb_flag (
bool
) – Flag if a combined redshift (or not) was added to the zmtl file.qnp_model_file (
str
, optional) – File from which the QuasarNP model was loaded. Written to the output header.squeze_model_file (
str
, optional) – File from which the SQUEzE model was loaded. Written to the output header.
- Returns:
The filename, with path, of the FITS file written out.
- Return type:
- desispec.zmtl.zcomb_selector(zmtl, proc_flag=False)[source]¶
Compare results from redrock, QuasarNP, SQUEzE, and MgII data.
- Parameters:
zmtl (
array
) – The structured array that was created by make_new_zmtl()proc_flag (
bool
) – Turn on extra comparison procedure.
- Returns:
The zmtl array with SQUEzE data included in the columns:
Z_COMB - The best models-combined redshift for each object.
Z_COMB_PROB - The combined probability value of that redshift.
- Return type:
array