desispec Change Log

0.62.0 (unreleased)

Major API/functional changes:

  • Dark missing from DESI_SPECTRO_DARK is now a fatal error unless preproc with --fallback-on-dark-not-found (PR #2162).

  • New CTE modeling on selected CCDs; requires matching DESI_SPECTRO_CALIB update post 0.5.1 (PR #2163, #2192, #2194).

  • Add fiberflat gradient correction option (PR #2180, #2190, #2208).

  • New desi_proc_night unified daily/production processing script with options to override default calibrations with config files (PR #2187, #2201, #2213, #2214).

Smaller items and new features:

  • Replace NaN TARGET_RA/DEC values with FIBER values (PR #2216).

  • Better handling of copyprod links (PR #2160).

  • Add desi_link_calibnight script (PR #2165).

  • Fix redrock API change templates_dir vs. template_path (PR #2168).

  • Add read_spectra_parallel (PR #2169).

  • Propagate units and comments in FIBERMAP HDUs (PR #2176).

  • Update dashboard for CTE flats (PR #2177).

  • Preproc interpolate overscan if active region entirely masked (PR #2178).

  • Simplify installation requirements (PR #2183).

  • Night QA support CCD 2amp mode (PR #2184).

  • Expand zcatalog “minimal” column set (PR #2185).

  • Increase dark model temperature difference acceptance window (PR #2188).

  • emlinefit do not fit EBV>2 spectra (PR #2195).

  • New wrap_rrdesi script to run Redrock on many input files (PR #2196).

  • Fix desi_group_spectra metadata with more than 1000 input files (PR #2199).

  • Fix qproc fallback on dark file not found (PR #2205).

  • Improve nightqa dark selection robustness (PR #2206, #2207).

0.61.0 (2024-01-15)

  • Add VCCDSEC info to tile_qa_plot (PR #2136).

  • Don’t skip exp_fibermap in io.spectra.read_spectra (PR #2137).

  • Don’t require a resolution matrix when coadding across cameras (PR #2139).

  • Add new methods to convert to and from spectra.Spectra and specutils.SpectrumList objects (PR #2141).

  • New row-by-row extraction and CTE correction modules (PR #2144).

  • Add --zscale and --mask options to plot_fiber_traces (PR #2145).

  • Fix traceshifts when half CCD amps are masked (PR #2150).

  • In create_petalnz_pdf, look for Redrock instead of zmtl files (PR #2151).

  • Fix failing io.photo unit tests at NERSC (PR #2154).

  • Remove deprecated test_suite functions from all unit tests that use it (PR #2155).

  • Various bug fixes and updates to io.read_tile_spectra (PR #2156).

  • Deprecate pkg_resources in favor of importlib (PR #2157).

  • Check for VCCDSEC and VCCDON when generating bias and dark models (PR #2158).

  • Check CCDTEMP when generating bias and dark models (PR #2159).

0.60.2 (2023-10-26)

  • Update validredshifts.validate() to work on SV observations (PR #2122).

  • Add desi_exposure_info script (PR #2124).

  • Fix exp_fibermap when specifying targetids to read_spectra (PR #2126).

  • Add desi_average_flux_calibration –unflat option (PR #2127).

  • io.photo use read-only mount (PR #2132).

0.60.1 (2023-09-25)

  • zcatalog FIRSTNIGHT/LASTNIGHT when missing petals (PR #2118).

  • night_qa: protect against missing OBSTYPE keyword (PR #2120).

  • zcatalog usage cleanup (PR #2121).

0.60.0 (2023-09-14)

Updates for patching Iron/DR1 redshift catalogs

  • coadd_fibermap handle RA wraparound (PR #2090).

  • Update COADD_FIBERSTATUS to bitwise OR when all inputs are bad (PR #2093).

  • Coadd bitwise OR of input targeting bits, e.g. DESI_TARGET (PR #2094).

  • Add IN_COADD/B/R/Z columns to EXP_FIBERMAP (PR #2100).

  • Fix dtype of *_NSPEC columns to int16 (PR #2103).

  • desi_zcatalog –recoadd-fibermap option (PR #2109, #2116).

  • fix HPXNEST header type (PR #2110).

  • desi_zcatalog –add-units option for DR1 patching (PR #2111).

  • Refactor desi_zcatalog options, add parallelism (PR #2117).

Algorithmic Updates

  • Fix incorrect weights in trace shift polyfit (PR #2105).

  • Improve trace shifts for bright/backup data (PR #2106, #2115).

Miscellaneous

  • Add read_spectra options to read subsets by targetids or rows (PR #2052).

  • Night QA flab calibs later than first science rather than absolute timestamp (PR #2089).

  • Night QA orient images and display CCD amp names (PR #2091).

  • Add desi_compute_fiberflat_vs_humidity –first-night option (PR #2101).

  • Add desi_compute_gains exposure time flexibility (PR #2107).

  • Update readthedocs configuration (PR #2112).

0.59.2 (2023-08-04)

  • Add recovery robustness for partially completed PSF jobs (PR #2059).

  • night_qa optional override of reprocessing darks (PR #2066).

  • desi_vi_tile defaults to only new tiles (PR #2073).

  • Optionally match to DR10 photometry (PR #2079).

  • Move desi_archive_tilenight contents into callable funcs (PR #2081).

  • Add (re-)archive options when missing petals (PR #2082).

  • Fix circular imports with trace_shifts (PR #2084).

0.59.1 (2023-06-23)

  • Add desi_daily_proc_manager –exp-cadence-time option (PR #2076)

0.59.0 (2023-06-12)

  • desispec.coaddition.coadd_fiberstatus bugfixes and add columns FIRSTNIGHT, LASTNIGHT, MIN_MJD, MEAN_MJD, MAX_MJD. Used for EDR zcat VAC. (PRs #2065, #2067, #2070)

0.58.4 (2023-06-02)

  • desispec.photo handle non-unique bricknames when patching 9010 tractor photometry (PR #2062, followup to #2057).

0.58.3 (2023-06-01)

  • Warn that purge_night was a dry run (PR #2040).

  • Restore desi_daily_proc_manager dry run and add sacct retries (PR #2044).

  • Add optional masking of specific CCD regions per exposure (PR #2050).

  • Avoid TSNR and EFFTIME NaN when traces are off CCD (PR #2053).

  • desispec.photo handle release 9010 vs. 9012 burst buffer bug, needed for lsdr9 photometry VAC (PR #2057).

0.58.2 (2023-05-03)

  • Add option to process cumulative redshifts for all tiles in desi_run_night (PR #2036).

0.58.1 (2023-05-02)

  • daily proc non-tilenight skybub exp fix [PR #2034, followup to PR #2028]

0.58.0 (2023-05-02)

Major:

  • Add tilenight support to desi_daily_proc_manager (PR #2028).

  • Change CTE fitting to be offset+ramp instead of just offset (PR #2033).

Smaller / bugfixes:

  • Flag calibration files performed on a different day (PR #2011).

  • Coadd spectra files with COADD_FIBERMAP instead of FIBERMAP (PR #2013).

  • Fix bug in desi_proc_tilenight which was causing it to ignore --cameras (PR #2014).

  • Propagate the --laststeps option to desi_proc_tilenight (PR #2015).

  • NightQA preproc dark bugfix when misisng cameras (PR #2016).

  • Fix failing io.photo unit tests (PR #2017).

  • Fix documentation test failures due to missing pytz (PR #2019).

  • Fix indexing error in targetid selection for plot_spectra (PR #2020).

  • Fix crash in desi_run_night when processing LASTSTEP=skysub exposures (PR #2022).

  • Fix failing bootcalib unit tests (PR #2029).

  • Don’t create exposures outdir for darks with no output (PR #2031).

0.57.0 (2023-02-17)

  • desi_compute_sky options to override sky TARGETIDs. Adds new SKYTARGETIDS HDU to output sky files (PR #2001).

  • Update functionality of io.photo to support Iron release (PR #2008).

0.56.5 (2023-02-01)

Last tag used for Iron run.

  • Fix healpix redshift (zpix) bookkeeping of bad data (PR #1991).

0.56.4 (2023-01-27)

Bugfix update for Iron re-processing.

  • Completely sync daily -> tiles-specstatus, not just new data (PR #1976).

  • Cleanup API documentation, changing many files (PR #1981).

  • Add calibnight jobs to exposure dashboard (PR #1982).

  • BUGFIX: define psfnight and nightlyflat cameras based upon all input arcs/flats instead of just the first one (PR #1984).

  • Add new desi_queue_status script (PR #1986).

  • Don’t hardcode $DESI_ROOT_READONLY in batch scripts (PR #1987).

0.56.3 (2023-01-24)

  • Minor fixes to io.photo (followup to PR #1971).

  • Remove imports from deprecated scipy.ndimage.filters namespace (PR #1977).

0.56.2 (2023-01-13)

  • Minor fixes to io.photo (PR #1971).

0.56.1 (2023-01-13)

  • Increase flat job time limit (PR #1970).

0.56.0 (2023-01-12)

QA and pipelining updates in support of daily ops and Iron/DR1:

  • Add update_survey_keywords to standardize early FA headers (PR #1858).

  • Standardize GPU options (opt-out with –no-gpu) and fix options for desi_run_night auto-deriving GPU vs. CPU per job (PR #1901).

  • cosmics parameters in yaml file for 500 micron mosaic z5 CCD (PR #1905).

  • cosmics_nsig default cleanup (PR #1906).

  • Reduce size of jobgraph diagrams (PR #1910).

  • Fluxcalib logging include cameras (PR #1912).

  • tilenight options and bug fixes (PR #1913).

  • poststd bugfix for expected cameras compared to stdstar fit (PR #1916).

  • QA updates (PRs #1917, #1926, #1938, #1944, #1945, #1947, #1954, #1962)

  • distribute_ranks_to_blocks bugfix (PR #1919).

  • desi_run_night robust to exposure tables without BADCAMWORD (PR #1921).

  • Improved calibfinder logging (PR #1922).

  • tsnr_afterburner MPI+multiprocessing updates; use fitsio (PR #1924).

  • NaN fix for negative CCD bkg (PR #1927, #1939).

  • Merge ongoing work on spectroscopic production database in preparation for EDR (PR #1928).

  • healpix redshifts added to zproc (PR #1931, #1966).

  • Check for type of dark option in preprocessing (PR #1936).

  • Add desi_run_night --do-cte-flat option (PR #1937).

  • Don’t set ivar=0 for unassigned fibers by default (PR #1942).

  • Fix tsnr_afterburner when no cframes on a night (PR #1950).

  • Submit cumulative redshifts only for last night of each tile (PR #1955).

  • Use desimodel/0.18.0 for testing for scipy/1.10.0 support (PR #1957).

  • Allow cutoff date for tile completion during reprocessing (PR #1959).

  • copyprod support for missing exposure tables (PR #1964).

  • Change zproc on cori realtime to 2 nodes and 30 minutes (PRs #1968, #1969).

0.55.0 (2022-11-11)

Major:

  • Restore sky-modeling sector offsets (PR #1825).

  • Add emlinefit afterburner to redshift scripts (PR #1852).

  • emlinefit (1+z) normalization bugfix (PR #1878).

  • Sky model use TPCORR to adjust sky line amplitudes, but not continuum (PRs #1886, #1895).

Minor:

  • Fix desi_proc stdstar args (PR #1822).

  • Set default system_name while parsing args (PR #1824).

  • Add desi_purge_night script (PR #1828).

  • Fix find_overscan_cosmics_trails in 2-amp CCD readout mode (PR #1831).

  • Fix iotime.parse_logfile when there aren’t any iotime messages (PR #1833).

  • Make $SPECPROD optional for running desi_preproc (PR #1835).

  • Catch exceptions in tilenight and update Perlmutter runtimes (PR #1837).

  • Add biasnight check to nightqa (PR #1841).

  • desi_run_night only launches obs completed tiles by default (PR #1843).

  • Fix 2-amp readout biasnight (PR #1844).

  • Add color coding to the nightly and monthly buttons in dashboards (PR #1845).

  • Sort on EXPID as well as TILEID in desi_run_night (PR #1848).

  • biasnight job shouldn’t “fail” if default bias is better anyway (PR #1850).

  • improve cframe file globbing to avoid tempfiles (PR #1853).

  • tilenight exit early if no good exposures (PR #1854).

  • fix psferr option in desi_extract_spectra (PR #1855).

  • Add support for NERSC CFS readonly mount (PRs #1856, #1888, #1891).

  • Use non-calibration zeros if not enough calib zeros are available (PR #1857).

  • Perlmutter use CPU for nightlybias, ccdcalib, arcs; GPUs otherwise (PR #1859).

  • Have desi_night_qa announce its completion (PR #1860).

  • Handle NaNs and zeros in skytpcorr, skygradpca x/y inputs (PR #1865).

  • Don’t wait for new data when override_night set in daily_proc_manager (PR #1866).

  • Ensure expids are ints in ccdcalib (PR #1868).

  • Give more time to stdstar jobs (PR #1869).

  • Fix stdstar indexing crash (PR #1872).

  • desi_run_night support case of no good zeros (PR #1875).

  • avoid stdstars joint fit makedirs race condition (PR #1879).

  • fix fallback to DESI_SPECTRO_CALIB if DESI_SPECTRO_DARK is set but doesn’t have suitable files for date range (PR #1882).

  • desi_proc options for sims (PR #1885)

  • Add copyprod --tiles option (PR #1893).

  • Fix fibermap indexing bug when mix of LEGACY and GAIA (PR #1894).

  • Add desi_job_graph to make job dependency graph webpages (PR #1896).

  • Apply job timefactor to nightlyflat jobs too (PR #1898).

  • Add GPU memory and rank allocation tools (PR #1899).

0.54.0 (2022-08-15)

  • Fix rank race condition bug in specex.py (PR #1809).

  • Stdstar use correct sky and flat fiber; scale ivar (PR #1817).

  • Stdstar memory optimization (PR #1820).

0.53.2 (2022-07-22)

  • Validate redshift catalog metadata and generate Tractor photometric catalogs for all objects with a redshift in a given data release (PR #1716).

  • Add batch script for desi_proc_tilenight jobs (PR #1792).

  • Add desi_psf_fit which wraps specex (PR #1794).

  • Parallelize desi_night_qa (PR #1795).

  • Enable redshift status board (PR #1797).

  • Correct flux units for emission lines (PR #1798).

  • Update QSO afterburners for new templates (PR #1805).

  • Fix function used by fastspecfit (PR #1808).

0.53.1 (2022-05-19)

  • QSO afterburners use temporary file when writing (PR #1768).

  • Fix desi_run_night crash with non-consecutive obs of same tile (PR #1771).

  • Set batch-friendly matplotlib backend (PR #1772).

  • Increase stdstar job requested time by 2 min (direct push, no PR).

0.53.0 (2022-05-15)

Major:

  • gzip intermediate files, including spectra files (PR #1756).

  • refactor pipeline to call functions instead of spawn scripts for MPI compatibility on NERSC Perlmutter. No user facing impact but major under-the-hood change (PR #1743).

Minor / backwards compatible:

  • Add ifmain wrapper to setup.py for testing (PR #1745).

  • Allow QA of already QAed tiles (PR #1747).

  • Update recipe for find_overscan_cosmic_trails (PR #1748, #1758).

  • desi_daily_proc_manager retry failed sbatch before giving up (PR #1749).

  • Enable trace shifts by default (PR #1750).

  • Improve error handling for specex (PR #1751).

  • desi_group_spectra --coadd option (PR #1753).

  • QA bugfix: handle nqso_rr if no valid fibers (PR #1754).

  • Improve fiberflat vs. humidity (PR #1757).

  • Night QA use findfile and fitsio (PR #1766.)

0.52.0 (2022-04-19)

Used to create combined zcatalog/zall-*.fits files in fuji and guadalupe.

  • qproc/nightwatch: fix deggy=0 for traceshifts (PR #1719).

  • Ignore overscan rows with large cosmic charge deposit in nearby columns (PR #1720)

  • Add tools to create combined/summary catalogs for specprods (PR #1721).

  • Use tile-median E(B-V) for LOWEFFTIME QA calculation (PR #1722).

  • tile QA: highlight secondary-only fibers in z vs. fiber (PR #1729).

  • Enable GPUs on perlmutter in rrdesi_mpi slurm script (PR #1730, #1734).

  • Adjust main survey minimum exposure efftime_etc (PR #1731).

  • Warning files produced by QSO afterburners will now be called .misscamera.txt (PR #1732).

  • Cleanup: make find_overscan_cosmic_trails separate function for testing (PR #1733).

  • Pipeline: add CTE flat class to dashboard so it reflects expected files (PR #1736).

  • Fix traceshift infinite iteration bug (PR #1742).

  • Fix doctest config for GitHub actions (PR #1744).

0.51.13 (2022-02-28)

  • desi_zcatalog --patch-missing-ivar-w12 option to patch missing FLUX_IVAR_W1/W2 values that weren’t propagated by early fiberassign (PR #1717).

0.51.12 (2022-02-23)

  • Remove unnecessary specter.psf import, which also allows desispec utilities to be imported without explicitly requiring specter (PR #1709).

  • Let plot_spectra show errors even with –rebin (PR #1714, #1708).

  • add SPGRPVAL to desi_zcatalog for custom coadds/redshift group tracking (PR #1712).

  • desi_assemble_tilepix replacement for desi_map_tilepix (PR #1713).

  • fix read_tile_spectra for group=’cumulative’ (PR #1696).

0.51.11 (2022-02-21)

  • qso_qn afterburner fix for case when all inputs are masked (PR #1704).

0.51.10 (2022-02-18)

  • tile-qa avoid divide-by-zero crash on unknown goaltime (PR #1698).

  • propagate HEALPIX into zpix redshift catalogs (PR #1699).

  • Fix GOALTIME in exposures FRAMES HDU; avoid EFFTIME_ETC NaN (PR #1701).

0.51.9 (2022-02-17)

Fuji cleanup bugfixes.

  • tile-qa goaltime special case for tiles 80715,80718 (PR #1689).

  • qso afterburner output breadcrumb file if missing input camera (PR #1691).

  • fix unwisebrightblue PROGRAM=other not bright (PR #1694).

  • fix tsnr afterburner GOALTIME exp vs. tile consistency (PR #1694).

  • fix plot_spectra with astropy 5 (PR #1695).

0.51.8 (2022-02-13)

Bugfixes for Fuji; all impacted tiles/nights/healpix rerun with this tag, remaining tiles/nights/healpix are backwards compatible.

  • Set specmask BADFIBER only for impacted cameras, not all BRZ (PRs #1674 (master), #1678 (fuji))

  • Fix assemble_fibermap with input NaNs for astropy 5.0 (PR #1681).

  • Use only 120s flats for nightlyflat (PR #1682).

  • Add desi_purge_tilenight script (PR #1683).

  • Fix healpix input expid bookkeeping (PR #1684).

0.51.7 (2022-02-10)

  • fix tile-qa expid bookkeeping (PR #1670).

  • desi_tile_qa exposure/night bookkeeping fix (PR #1672).

  • Fix tsnr_afterburner exposure files SURVEY column (PR #1675).

0.51.6 (2022-02-09)

Used for Fuji healpix redshifts and cleanup of failed tile-qa. Backwards compatible with previously run steps.

  • Make tile-qa robust to missing cameras (PR #1665)

  • Refactor healpix redshifts workflow (PR #1668)

0.51.5 (2022-02-07)

Used for processing nightly biases for Fuji nights 20210331 and 20210422, and Guadalupe night 20210629. Backwards compatible with other nights.

  • Additional desi_compute_nightlybias options for flexibility on which ZEROs to use (PR #1662).

0.51.4 (2022-02-04)

Pipelining fix for Fuji; previously run impacted nights will be resubmitted.

  • Fix stdstar camword logic when input exposures have different cameras available (PR #1658).

0.51.3 (2022-01-31)

NOTE: this tag fixes a crash, but also produces slightly different humidity correction for a small set Fuji/Guadalupe exposures already run with an earlier tag. This note will be updated if those exposures are reprocessed with this tag.

  • Fix fiberflat humidity correction indexing bug when hear (but not at) upper limit of model humidity range (PR #1642).

0.51.2 (2022-01-27)

Fuji bug fixes (impacted nights will be re-run; nights run with earlier tags not impacted)

  • fix pipeline bug on nights with multiple 300s darks (PR #1635).

  • fix io.findfile(..., groupname='perexp') (PR #1637).

0.51.1 (2022-01-26)

Fuji bugfix tag made from the fuji branch mid-processing. These changes fix crashes but do not impact any data that were already successfully run.

  • Updated desi_find_badpos script to cross-reference flagged petals against existing bad-exposure tables.

  • Fix fiberflat crash when almost all input data are masked for a fiber (PR #1629).

  • Fix tile QA for cases when input fiberassign file is not gzipped (PR #1630).

  • Fix zcat stacking typo (PR #1633).

0.51.0 (2022-01-24)

This version will be used for Fuji.

Algorithm update:

  • Normalize fiberflat variation of each fiber for humidity correction (PR #1621).

Metadata tracking updates:

  • Add BADAMP[BRZ] bits to QAFIBERSTATUS (PR #1610).

  • specgroup metadata in spectra, coadd, zcat files (PR #1618).

New and fixed scripts / functions:

  • Add desispec.zcatalog.find_primary_spectra (PR #1609).

  • Add desispec.tile_qa.get_tilecov tile coverage plotting (PRs #1613, #1617).

  • Fix bookkeeping of nights and tiles in coadds (issue #1349) and enable coadding of previously coadded cframe files (issue #1359) (PR #1616).

  • Ensure tilepix.fits only contains healpixels with reduced data (issue #1374). Also fix issues #1373 and #1379 (PR #1614).

  • Add desi_find_badpos script to find exp-petals with catastrophic positioning (PR #1620).

0.50.1 (2022-01-20)

  • Modification extname in QN afterburner outputs. Add flag in desi_qso_catalog_maker for retro-compatibility (PR #1597).

  • Outlier rejection in skycor PCA (PR #1598).

  • Include FAILED jobs in resubmissions (PR #1602).

  • tile-qa updates for special tiles (PR #1603).

  • Better masking behavior for BADCOLUMN (PR #1605).

  • Add DEPNAM/DEPVER to fibermap HDUs (PR #1607).

  • desi_run_night enhancements and bug fixes (PR #1608).

0.50.0 (2022-01-16)

Used for the f5 test run part II.

  • Algorithmic changes:

    • Use sky fibers to correct for serial readout CTE problems (PR #1571).

    • Bug fixes for fiberflat_vs_humidity when near edge of model range (PRs #1589, #1594).

  • Miscellaneous:

    • Approximate for missing turbulence corrections in FIBER_RA/DEC during Dec 2021 (PR #1539).

    • nightqa improved selection of which 5min dark was used (PR #1584).

    • findfile(tileqapng) bugfix for tile/night QA (PR #1585).

    • night QA 5min DARK expid selection bug fix (PR #1586).

    • fix qprod non-empty fibermaps (PR #1587).

    • desi_update_specstatus –lastnight and –all options (PR #1588).

    • better desi_proc error tracking (PR #1590).

    • assemble_fibermap fail faster on invalid inputs (PR #1592).

0.49.1 (2022-01-10)

Used for the f5 test run part I.

  • Fix propagation of per-camera keywords into per-camera fibermap (commit #0c7aa720)

0.49.0 (2022-01-10)

  • Major algorithmic updates:

    • Fit CCD residual background between blocks of fiber traces (PR #1551, #1581).

    • Correction for fiberflat variations with humidity (PR #1565).

  • Other algorithmic updates:

    • Updated readnoise estimation when doing overscan per row (PR #1564).

    • Remove average of overscan cols to overscan rows (PR #1575).

    • Avoid false positive bad column mask on noisy input data (PR #1579).

    • Improvements to desi_interpolate_fiber_psf (PR #1557).

  • Miscellaneous:

    • run nightlybias on nights without a dark (PR #1553).

    • nightqa petal n(z) support programs with no tiles (PR #1549, #1554).

    • Fix GitHub tests fitsio/numpy incompatibility issues (PR #1566).

    • Bad readnoise PSF failure robustness (PR #1568).

    • Bad exposure bookkeeping for darks and biases (PR #1570).

    • don’t flag a tile as archived if archiving failed (PR #1572).

    • Tile QA plotting updates (PR #1577, )

    • Fix fitsverify errors when creating preproc files (PR #1582).

    • Added assemble_fibermap --tilepix option (PR #1583)

0.48.1 (2021-12-21)

Used for the f4 test run (albeit pre-tag).

  • Fix deprecation warnings for latest numpy (PR #1525).

  • Support astropy 5.x masked columns (PR #1526).

  • More robust when fiberassign file is in earlier expid (PR #1529, #1536).

  • Approximate FP coords if missing from coordinates file (PR #1532).

  • desi_run_night –surveys option (PR #1533).

  • night_qa v3 (PR #1535).

  • Update and standardize exposure tables (PR #1537).

  • Fix desi_proc spexec wrapper for single camera (PR #1540).

  • Fix desi_proc MPI logic bug if missing PSF input (PR #1542).

  • Change nightlybias OSTEP to be local instead of global diff (PR #1543).

  • Run nightlybias for all cameras (PR #1546).

  • Tune nightlybias running logic (PR #1547).

  • Add emlinefit afterburner (PR #1386).

0.48.0 (2021-12-10)

  • Fix TSNR afterburner “unknown” entries (PR #1495).

  • Tile QA skip n(x) comparison for backup program (PR #1497).

  • assemble_fibermap robust to missing guider EXPTIME (PR #1498).

  • update job dependencies to be afterok instead of afterany (PR #1502).

  • Add desi_night_qa (PR #1503, #1522).

  • desi_tile_vi only show main dark/bright tiles by default (PR #1505).

  • Add support for short QA flats in pipeline (PR #1507, #1517).

  • Revise size of KNL batch jobs for arc fits (PR #1508, #1521).

  • fix proctable entries with 0-length arrays (PR #1509).

  • flag FIBER_X/Y==0 as FIBERSTATUS MISSING (PR #1514).

  • Separate tile QA from tile archiving (PR #1519).

  • Improve pipeline operations on KNL (PR #1523).

0.47.1 (2021-11-24)

  • New desi_resubmit_queue_failures script (PR #1482).

  • fix CAMERA column name in stdstars INPUT_FRAMES HDU (PR #1484).

  • raise exception when no valid sky fibers (PR #1486, #1488).

  • Update qso_catalog_maker to include all targets (PR #1487).

  • Fix TSNR2 calculation and exposure QA for 2-amp readout (PR #1489).

  • Exclude masked pixels in stdstar RMS QA (PR #1490).

  • Support for very early fiberassign files in assemble_fibermap() (PR #1492).

  • desi_tile_vi –qastatus option (PR #1493).

0.47.0 (2021-11-11)

  • tsnr_afterburner support for old HDU names (PR #1403).

  • Tiles tables and QA cleanup (PRs #1406, #1407, #1409, #1410, #1430, #1442, #1445, #1449, #1458, #1475).

  • Support averaging PSFs with different wavelength ranges (PR #1411).

  • QSO afterburner add blank file when no targets to write (PR #1412).

  • Update airmass dependence for exposure quality cuts (PR #1413).

  • Only use good sky fibers for sky model (PR #1414).

  • Switch EFFTIME_SPEC to be based upon LRG instead of ELG (PR #1417).

  • specex job scheduler for improved performance (PR #1418).

  • Add desi_update_tiles_specstatus script (PR #1421).

  • Parallelize fiberflatnight and cleanup tempfiles (PR #1427).

  • Cleanup bad fiber tracking BROKENFIBERS, BADCOLUMNFIBERS, LOWTRANSMISSIONFIBERS (PR #1429).

  • Select calibration stars per exposure across petals (PR #1434).

  • QSO afterburner run on all targets, not just QSO (PR #1435).

  • Set OMP_NUM_THREADS in batch script (#1437).

  • Allow fiberassign svn to have different negative TARGETID (PR #1444).

  • Arc jobs restricted to <= 10 nodes even on KNL (PR #1450).

  • Sky model bug fix to avoid fake z~4.3 QSO (PR #1452).

  • desi_edit_exposure_table useability improvements (PR #1453).

  • switch io.read_fibermap to use fitsio to avoid astropy masked columns (PR #1454, #1479)

  • daily pipeline runner cache exposures after every new exp (PR #1455).

  • Hartmann doors analysis script (PR #1457).

  • Ensure consistent output from assemble_fibermap() (PR #1458).

  • New desi_compute_nightly_bias script (PR #1460).

  • Incorporate nightly bias and bad column identification into daily pipeline (PR #1463).

  • Add OSTEP metric for variation of overscan per row (PR #1464).

  • Add nightly bias and bad column flagging to pipeline (PR #1465, #1467)

  • Fix check_for_outputs bug and teach findfile about fit-psf (PR #1469).

  • Set fibermap.FIBERSTATUS BADREADNOISE and BADAMP[BRZ] (PR #1472).

  • Don’t use PSFs with bad amps (PR #1473).

  • Support 2-amp readout (PR #1476).

  • Use only offset traceshifts when amplifier is masked (PR #1477).

  • desi_archive_tilenight to archive nights after tile QA approval (PR #1478).

  • processing dashboard useability updates (PR #1480).

  • use desitarget.skybricks to check stuck sky locations (PR #1481).

0.46.1 (2021-09-03)

  • Restore desi_zcatalog backwards compatibility for inputs without a separate EXP_FIBERMAP (PR #1392).

  • tsnr_afterburner robustness to missing/different columns, e.g. from reprocessed daily exposures (PR #1396).

  • Save STDSTAR_FIBERMAP in fluxcalib file recording which stars were used (PR #1400).

  • stdstars robust to missing individual frames (PR #1402).

0.46.0 (2021-09-01)

  • Detect and flag sky fibers affected by a bright source (PR #1367)

  • Adjust spectral traces when computing CCD variance in preprocessing (PR #1368, #1378).

  • Detect bad CCD columns in nightly dark; use to mask impacted wavelengths of individual spectra (PR #1371).

  • Model CCD readnoise with Poisson noise from the zero exposure dark current (PR #1372).

  • Speed up coadd_cameras (PR #1377).

  • Improve sky subtraction with PCA model of wavelength and LSF sigma adjustments (PR #1381).

  • Swap fibers 3402 and 3429 if input fiber:location map is wrong (PR #1382).

  • Tile QA minor fixes (PR #1385).

  • Allow fiberassign SVN to override raw data fiberassign (PR #1387).

  • Add desi_run_night --tiles option (PR #1391).

0.45.3 (2021-07-29)

Everest cleanup PRs; in 21.7e software release.

  • Fix “unknown” entries in tsnr/exposures file (PR #1321).

  • desi_healpix_redshift options for extra memory (PR #1343).

  • desi_zcatalog for new fibermap format (PR #1347).

  • Include TILERA, TILEDEC, MJD in tiles/exposures output tables (PR #1348).

  • desi_group_spectra –healpix option (PR #1350).

  • desi_tile_redshifts –group cumulative bug fixes (PR #1353).

  • desi_map_tilepix script (PR #1358).

  • merge of above PRs into main/master (PR #1360).

0.45.2 (2021-07-20)

Everest bookkeeping update for sv1 spectra regrouping and sv3 redshift cleanup.

  • desi_group_spectra options to filter and group by survey and faprogram, including desispec.io.meta.faflavor2program to handle special cases for sv1 (PR #1341).

  • desi_healpix_redshifts options for memory usage (PR #1341).

0.45.1 (2021-07-15)

For Everest tile spectra+coadds+redshifts; in 21.7d

  • Recompute proccamword from exptable for proecessing dashboard (PR #1340).

  • Add support for healpix coadd+redshift jobs (PR #1341).

0.45.0 (2021-07-14)

For Everest tile spectra+coadds+redshifts; in 21.7c

Note: this is a major format change to coadd and redrock (formerly zbest) files:

  • FIBERMAP split into FIBERMAP (coadded) + EXP_FIBERMAP (per-exposure)

  • zbest files renamed redrock

Requires redrock >= 0.15.0

Details / PRs:

  • tune tile QA parameters

  • Add desi_qso_catalog_maker (PRs #1322, #1339).

  • QSO afterburner cross checks and bug fixes (PRs #1334, #1335)

  • Fix exposure table loading typo (PR #1337).

  • Rename zbest -> redrock (PR #1338).

  • Split coadd FIBERMAP into FIBERMAP + EXP_FIBERMAP (PR #1330).

  • QSO afterburners run in desi_tile_redshifts by default (PR #1330).

0.44.2 (2021-07-07)

Intended for Everest science exposures through cframes; in 21.7b.

  • Fixed data/qa/ installation.

0.44.1 (2021-07-07)

  • Add fibermap PLATE_RA,PLATE_DEC if missing from fiberassign file for sv3 and main survey data model consistency (PR #1331).

  • Set FIBERSTATUS POORPOSITION bit when positioner is offset by 30-100 microns. Don’t use these for stdstar fitting but otherwise process as normal. (PR #1333).

0.44.0 (2021-07-06)

First tag used for Everest arc/flat calibs; in 21.7a.

  • Add QSO afterburners for MgII and QuasarNet (PR #1312).

  • Spectra I/O for extra catalog (PR #1313).

  • Expand Spectra.select and .update functionality (PR #1319).

  • Add optional support for gpu_specter for extractions (PR #993).

  • Fix extra_catalog support for grouping by healpix (PR #1325).

  • Pipeline progress bug fixes and features (PRs #1326, #1329).

0.43.0 (2021-06-21)

This version was used for QA assessment of the first 315 main survey tiles released for unlocking overlapping tiles. That was done pre-tag under the development version “0.42.0.dev5412”.

  • Improved stitching of sky spectra from different cameras (PR #1273).

  • TSNR updates (PR #1274 and branch PRs #1275, #1277, #1279, #1282, #1283, #1285).

  • qproc robust to blank SEEING keyword (PR #1289).

  • update SV1-SV3 average throughtput (PR #1291).

  • fix x traceshift indexing bug (PR #1292).

  • desi_tile_redshifts –run_zqso option (PR #1293).

  • pre-write speclog when generating dark model scripts (PR #1300).

  • Add spectroscopic QA (PR #1302, #1316).

  • Improve pipeline metadata handling and implement QA cuts (PR #1304).

  • Check for completely masked fibers in qfiberflat (PR #1306).

  • Pipeline robustness when reading ETC values from raw data (PR #1309).

  • Adjust exposure QA cuts, cleanup outputs (PRs #1316, #1318).

  • Simplified tile QA (PR #1317).

  • zmtl using tile QA to set ZWARN bits (PR #1310).

  • Look for manifest files in nightly processing (PR #1320).

0.42.0 (2021-05-14)

Requires desiutil >= 3.2.1 for new dust extinction calculations.

  • Wrap bin/desi_proc and bin/desi_proc_joint_fit in functions to facilitate pipeline wrappers (PRs #1242 and #1244).

  • Use less restrictive gfaproc extension 2 instead of 3 for EFFTIME_GFA (PR #1245).

  • Add MPI to stdstar match_templates (PR #1248).

  • Updates to desi_average_flux_calibration (PR #1252).

  • desi_fit_stdstars --std-targetids option to override stdstars for testing and custom fields (PR #1257, #1259).

  • Launch redshifts automatically as part of pipeline (PR #1260).

  • Support stuck positioners assigned to sky locations (PR #1266).

  • Use desiutil.dust for extinction including Gaia (PR #1269).

  • Fix running instance checking in daily pipeline (PR #1270).

0.41.0 (2021-04-16)

Although most of the Denali production was run with tag 0.40.1, the following updates where made for final steps to recover some missing coadds and make the final tsnr and tiles files:

  • Exposure and tiles files updates, including merging GFA data. (PR #1226, #1232, #1236, plus commits directly to master on Apr 12).

  • Fix coadds with missing TSNR columns due to missing cameras (PR #1239).

Changes that also occured in the meantime but were not used for Denali processing (they impact earlier steps):

  • Flag fibers that are mis-positioned by >100 um as bad. (PR #1233).

  • Correct bit flagging and support split exposures with bin/assemble_fibermap (PR #1235).

  • Also write fibercorr to the fluxcalibration when using low S/N simplified calibration (direct fix to master).

0.40.1 (2020-04-01)

Installation and job submission fixes for Denali; no algorithmic changes.

  • fix data installation (PR #1221).

  • desi_tile_redshifts --batch-reservation fix for Denali run (PR #1222).

0.40.0 (2021-03-31)

First tag for 21.3/Denali run

  • Add fiber crosstalk correction (PR #1138).

  • Handle missing NIGHT in coadded fibermap (PR #1195).

  • Add desi_tiles_completeness script with TSNR2-based tile completeness calculations for survey ops (PR #1196, #1200, #1204, #1206, #1212).

  • TSNR2 camera coadd fix (PR #1197).

  • refactor desi_tile_redshifts for more flexibility (PR #1198, #1208, #1211).

  • processing dashboard: cache night info (PR #1199).

  • speed up sky calculation with different sparse matrices (PR #1209).

  • Check file outputs before submitting jobs (PR #1217).

  • improve noise of master dark model fit (PR #1219).

  • Add workflow hooks for KNL (PR #1220).

0.39.3 (2020-03-04)

Cascades update tag for final catalog creation.

Note: datamodel changes to coadd SCORES and FIBERMAP

  • Propagate TSNR2 into coadd SCORES; update coadd FIBERMAP columns (PR #1166)

  • bin/desi_tsnr_afterburner use pre-calculated TSNR2 from frame files unless requested to recalculate (PR #1167).

0.39.2 (2021-03-02)

Cascades update tag to fix coadd and tSNR crashes, and postfacto tag desi_spectro_calib version in desispec module file.

  • Processing dashboard useability updates (PR #1152).

  • Undo heliocentric correction in throughput analysis not used for production processing (PR #1154).

  • Fix coadd crash (PR #1163).

  • Fix tSNR alpha<0.8 crash (PR #1164).

  • Updated desi_spectro_calib version to 0.2.4.

0.39.1 (2021-02-23)

Cascades update tag to add functionality for using a queue reservation and for debugging, without algorithmically impacting what has already been run with the 0.39.0 tag.

  • Add desi_run_night --reservation option (PR #1145).

  • Fix desi_process_exposure --no-zero-ivar option (PR #1146).

0.39.0 (2021-02-16)

Initial tag for Cascades run.

Major updates:

  • Update exposure table formats and pipeline workflow (PR #1135, #1139).

  • Add template S/N (TSNR) depth calculations (PR #1136).

Smaller updates:

  • Propagate fiberassign HDU 0 keywords into fibermap header in addition to FIBERASSIGN (HDU 1) keywords (PR #1137).

  • desi_proc_joint_fit exit with error code if all cameras fail (PR #1140).

  • Frame units “electron/Angstrom” instead of “count/Angstrom” (PR #1142).

0.38.0 (2021-02-10)

  • Change how specex PSF fitting is called; requires specex>=0.7.0 (PR #1082)

0.37.0 (2021-02-10)

Major updates:

Smaller (but important) updates:

  • Standardize getting NIGHT from raw data headers (PR #1083, #1120).

  • Use acquisition guide file if full guide file isn’t available (PR #1084).

  • Updates to flux calibration averages used by nightwatch (PR #1085).

  • New read_tile_spectra and Spectra class slicing (PR #1107).

  • Add token to fix coverage tests (PR #1112).

  • Flux calibration robustness for low transmission exposures (PR #1116).

  • Apply heliocentric correction to fiberflat (PR #1118).

  • Robustness and feature updates to dark model generation (PR #1119, #1123)

  • More flexible CCD calibration configuration (PR #1121).

  • Processing dashboard useability updates (PR #1127).

  • NIGHT int vs. str bugfix in QA (PR #1129).

  • Support coaddition of fibermaps with different columns (PR #1130).

0.36.1 (2021-01-04)

  • Fix PSF traceshifts when a fiber is completely masked (PR #1080).

  • Robust to NaN in desi_average_flux_calibration (commit f1de1ac).

  • Increase arc and flat runtimes (commit 7cb294c).

0.36.0 (2020-12-23)

This is the primary tag for the Mt. Blanc spectro pipeline run.

  • Major updates:

    • Coadd fluxes in multi-exp standard stars before fitting (PR #1059).

    • New model of CCD pixel-level variance (PR #1062).

    • Adjust sky-line variance based on model chi2 (PR #1062).

  • Smaller (but important) updates:

    • Fixes assemble_fibermap for older data (PR #1047, bug introduced in PR #1045).

    • Use EBV instead of MW_TRANSMISSION_G/R/Z from fiberassign (PR #1048).

    • Fallback to using FA_TYPE if no stdstars in (SVn_)DESI_TARGET (PR #1050).

    • Use GitHub Actions for testing instead of Travis (PR #1053).

    • Fix stdstar absolute symlinks (PR #1056).

    • Adjust nodes per job (PR #1056 and #1068).

    • Workflow options for bad exposures and new end-of-cals manifests (PR #1057).

    • stdstar robustness if petal is disabled (PR #1060).

    • improved camera argument parsing (PR #1061).

    • Fix unphysical spike at edge of calibration vectors (PR #1065).

    • Add header keywords for input calib provenance (PR #1069).

    • More logging about stdstar selection cuts (PR #1070).

    • Only uses fiberassign .fits and .fits.gz (but not .fits.orig) (PR #1072).

    • Support “unpositioned” exposures; propagate FIBER_RA/DEC if present (PR #1073).

    • Use desi_spectro_calib tag 0.2.1

0.35.0 (2020-12-11)

  • Major updates:

    • New opts to model image variance and improve sky subtraction (PR #1008).

    • Refactor desi_proc and daily processing workflow (PRs #1012, #1014, #1030)

    • New bias+dark model (“non-linear dark y1D”) in desi_spectro_calib 0.2.0 (PR #1029)

  • Smaller (but important) updates:

    • etc/desispec.modules uses desi_spectro_calib 0.2.0

    • Default saturation 2**16-1; updated keywords (PR #1046).

    • Fix preproc header keyword propagation (PR #1045).

    • Add support for gzipped fiberassign files (PR #1042).

    • Fix tests on single-core machines (PR #1035).

    • desi_paste_preproc for future use combining short+long arcs (PR #1034).

    • desi_proc more robust to specex failures (PR #1033).

    • Add parallelism to desi_preproc (PRs #1032, #1036, #1038).

    • Fix specex empty path bug (PR #1031).

    • Better qproc warnings for test slit exposures (PR #1028).

    • desi_focus focus scan analysis (PR #1027).

    • Fix/add BUNIT header keyword (PR #1023).

    • Adds desi_compute_broadband_pixel_flatfield (PR #1022).

    • Update desi_proc timing logging (PR #1003, #1026).

    • desispec.module sets MPICH_GNI_FORK_MODE=FULLCOPY for MPI+multiprocessing (PR #1007).

    • Fix dark CCD calibration corrections (PR #1002).

0.34.7 (2020-09-01)

  • Switch desi_proc to use fitsio instead of astropy.io.fits to work around incompatibility between mpi4py and astropy 4 (PR #996).

0.34.6 (2020-08-04)

  • Extend runtime limit for spectra regrouping task (hotfix to master).

0.34.5 (2020-08-04)

  • Faster desi_zcatalog merging with target table (PR #994).

  • Python 3.8 support (PR #990).

  • Astropy 4.x support (PR #989).

  • Update CCD mask generation code (PR #987).

  • Update desispec.io.download to use data.desi.lbl.gov (PR #972).

  • Use middle of exposure for barycentric correction time (PR #971).

0.34.4 (2020-04-21)

  • Add desi_proc –batch-opts … option for specifying extras like queue reservation (direct push to master).

0.34.3 (2020-04-17)

  • Run desi_proc arc and flat jobs on max 10 nodes instead of 5 (PR #958).

0.34.2 (2020-04-16)

  • Include data/spec-arc-lamps.dat with installed data.

  • Mask high readnoise CCD amps (PR #957).

0.34.1 (2020-04-15)

  • Expanded scan range for y traceshifts from +-3 to +-10 A (commit 26279d8 direct to master)

  • Improved traceshift robusteness for very large shifts of arcs (PR #954).

  • Added scripts for creating bad pixels masks from darks (PR #946).

  • etc/desispec.module use desi_spectro_calib tag 0.1.1 (PR #955).

  • import specter only if needed to run, not requiring it just to import desispec.io (PR #955).

Note: python setup.py install of this version incorrectly doesn’t copy data/spec-arc-lamps.dat into the final installed data directory; that is fixed in next version, and was fixed by hand in NERSC 0.34.1 install.

0.34.0 (2020-04-13)

Compatibility notes:

  • Requires desiutil >= 2.0.3 (PR #951).

  • Backwards incompatible change to sky model format (PR #939.

Changes:

  • Refactor S/N fit for QA (PR #917)

  • Speed up QA (PR #917)

  • Don’t mask extreme mask fiberflat >2 or <0.1 in routine autocalib_fiberflat because the fiberflat includes the throughput difference between spectrographs (push to master to address issue #897).

  • Modify overscan methods. Default is to no longer analyze the ORSEC region (PR #838).

  • Fix sky subtraction with ivar=0 (PR #920).

  • Tweaks for logging nightly redshifts and srun (PR #921).

  • Added calib config management utilities (PR #926).

  • Coadd robustness when missing a camera (PR #927).

  • Shorter desi_proc job names (PR #928).

  • Set fiberstatus to mask fibers in bad regions of CCDs (PR #930).

  • Fix code generating fits reserved keyword warnings (PR #933, #935).

  • Try fibermap header if primary header doesn’t have RA,DEC (PR #934).

  • Force assemble_fibermap for nights before or during 20200310 (PR #936).

  • Don’t fit traceshifts in y for dome and twilight flats (PR #937).

  • Calculate sky model throughput corrections when making sky model instead of while applying model. Note: changes data model. (PR #939).

  • Improve averaging of fiberflats (PR #940).

  • Fix incorrect multiple calls to bary_corr depending upon MPI parallelism, and merge extract main and main_mpi (PR #943).

  • Propagate MJD to spectra fibermap (PR #944).

  • Generate spectra files by default and don’t coadd across cameras (PR #945).

  • Allow coadding across cameras of coadds (PR #948).

  • Implement fibermaps per camera (PR #949).

  • Use desiutil.iers.freeze_iers instead of desisurvey; requires desiutil>=2.0.3 (PR #951).

  • Module file users desi_spectro_calib tag 0.1

0.33.0 (2020-03-05)

  • Metadata bookkeeping for early CMX data (PR #857)

  • Improved PSF handling in desi_proc (PR #858)

  • Modeling scattered light (PR #859, #861, #862)

  • desi_proc –calibnight option (PR #860)

  • expanding flux calib stdstar bits (PR #862)

  • new assemble_fibermap script (PR #864, #902)

  • improved sky subtraction and flux calibration robustness (PR #865)

  • new desi_group_tileframes script; coadd frames directly (PR #866)

  • flux calibration improvements (PR #868, #871, #880, #898)

  • more efficient desi_proc –batch parallelism packing (PR #869)

  • new desi_proc_dashboard script (PR #870, #901)

  • new desi_dailyproc script (PR #872, #881, #895)

  • more robustness to missing inputs (PR #875, #876, #883)

  • groundwork for improving cosmics masking (PR #878)

  • enable barycentric correction in desi_proc (PR #879)

  • new plot_spectra script (PR #890)

  • new desi_nightly_redshifts script (PR #892)

  • Generate QA for a given night + QA bug fixes (PR #894)

  • coadd metadata propagation (PR #900)

  • don’t use FIBERSTATUS!=0 spectra in coadds (PR #903)

  • desi_proc more control options for minisv2 run (PR #904)

  • Two hotfixes to master to re-enable daily processing:

    • make assemble_fibermap more robust to missing input columns in the platmaker coordinates files.

    • better packing of extraction MPI ranks

0.32.1 (2019-12-27)

  • Integration test simulate past not current date to workaound pixsim header mismatch with DESI_SPECTRO_CALIB calibrations. (direct push to master).

0.32.0 (2019-12-22)

  • Adding more desi_proc options (PR #848, #850).

  • Support PSF bootstrapping with broken fibers (PR #849).

  • Hot fixes to desi_proc crashes (pushed directly to master).

  • Increase cframe task from 1 min to 2 min (direct to master).

  • Adapt to new spectrograph SMn naming (PR #853).

  • Workaround fitsio bug by setting blank keywords to None; adapt to new fiberassign file names (PR #855).

0.31.0 (2019-10-31)

First CMX release with bug fixes for on-sky data.

  • Use rrdesi –no-mpi-abort feature (PR #823).

  • Added code to generate pixflats (PR #824).

  • Support extractions of data without fibermaps (PR #825).

  • Propagate FIBERMAP into preproc files (not just frames) (PR #825 and #829).

  • Allow extraction wavelenghts slightly off CCD (PR #836).

  • PSF I/O pause before merging (PR #836).

  • Add bin/desi_proc single-exposure processing script (PR #837).

  • Use OBSTYPE instead of FLAVOR for desi_qproc (PR #839).

  • Bug fix for desi_proc double application of fiberflat (PR #841).

  • desi_proc options for non-default PSF and fiberflat (PR #842).

  • Correct fibermap to match what petal we are in (PR #843).

  • Update database loading to match current data model (PR #844).

  • Added desi_proc –batch option (PR #845).

0.30.0 (2019-10-17)

  • qproc updates (PR #787).

  • QL bias (PR #789).

  • Heliocentric corrections (PR #790).

  • Update photometric filter usages (PR #791).

  • Add gain output option to desi_compute_gain

  • Modify overscan subtraction algorithm in desi.preproc.preproc (PR #793).

  • Cleanup timing parameters (PR #794).

  • Pipeline docs (PR #797).

  • Correct for dark trail in raw images (PR #798).

  • yaml.load() to yaml.save_load() (PR #801).

  • help numba know the types (PR #802).

  • desi_pipe getready fix (PR #803).

  • Move raw data transfer scripts to desitransfer (PR #804).

  • spectra coaddition (PR #805).

  • memory constraints and load balancing (PR #806 and #809).

  • preproc header keywords CCDSEC1-4 vs. A-D (PR #807).

  • Add desi_pipe status command (PR #810).

  • Convert any expid input into an int in QA (PR #814).

  • Support new FIBERASSIGN_X/Y instead of DESIGN_X/Y (PR #821).

  • Added hostname and jobid to task logging (PR #822).

0.29.0 (2019-05-30)

  • Add HPSS backup to the raw data transfer script (PR #765).

  • Update desispec.database.redshift for latest changes in fiberassign tile file data model (PR #770).

  • Constants, docs, and test cleanup (PR #771, #773, #776).

  • Tune cosmics masking parameters (PR #775).

  • Add desi_compute_pixmask (PR #777).

  • qproc updates for more flexibility and exposure flavors (PR #778).

  • Better io.findfile camera checks (PR #780).

  • Support SV1_DESI_TARGET (PR #786).

0.28.0 (2019-02-28)

  • Update (non-essential) transfer script for spectrograph functional verification tests (PR #758).

  • New calibration data access (inc var. DESI_SPECTRO_CALIB replacing DESI_CCD_CALIBRATION_DATA) (PR #753).

  • Fix offline QA S/N vs. mag fits (PR #763).

0.27.1 (2019-01-28)

0.27.0 (2018-12-16)

  • DB loading targets columns PRIORITY_INIT and NUMOBS_INIT; requires desitarget/0.27.0 or later for DB loading (PR #747).

  • Fix S/N QA when inputs have NaNs (PR #746).

  • DB exposures table loading allows NaN entries for RA,DEC,SEEING,etc. for arc and flat calib exposures (PR #743).

  • Use new desiutil.dust.ext_odonnell function during flux-calibration (PR #736).

  • Add support for average flux calibration model in ccd_calibration_data repo (PR #735).

  • Support mockobs fibermap format with fewer columns (PR #733).

  • Upgrade data transfer script and add additional scripts (PR #732).

  • Fix desi_zcatalog RA_TARGET vs. TARGET_RA (PR #723).

  • Update redshift database data model and workaround a minor bad data problem (PR #722).

  • Refactor offline QA (S/N) to work with updated object typing

  • Drop contam_target DB truth column; no longer in truth files (one-line commit to master, no PR).

  • Bug fix in QA (S/N) + refactor exposure slurping (PR #746)

  • Refactor QA_Exposures, QA_Night, and QA_Prod; Generate new Prod QA (offline)

0.26.0 (2018-11-08)

Major non-backwards compatible changes:

  • Update to new fibermap format for consistency with targeting and fiber assignment (PR #717).

  • Include GAIN in preproc headers (PR #715).

  • Prototype data transfer status report webpage (PR #714).

  • Integrate qproc/qframe into quicklook (PR #713).

  • Quicklook flux calib and config edits (PR #707).

0.25.0 (2018-10-24)

  • QL algorithm, config, and format updates (PRs #699, #701, #702). (Includes non-backwards compatible changes).

0.24.0 (2018-01-05)

  • Quicklook updates (including non-backwards compatible changes)

    • New QL calibration QA metrics (PR #677).

    • Update QL to use xytraceset instead of custom PSF (PR #682).

    • Cleanup for robustness and maintainability (PR #693).

  • Offline QA updates

    • Integrates QL S/N QA into offline QA Frame object (PR #675).

    • Additional offline QA plots on S/N (PR #691).

  • Spectroscopic pipeline updates

    • Option to generate bash scripts instead of slurm scripts (PR #686).

    • new desi_pipe go –resume option (PR #687).

    • desi_pipe sync –force-spec-done option (PR #692)

  • Miscellaneous

    • Work-around bug that forbids opening memory-mapped files in update mode on some NERSC filesystems (PR #689).

    • Do not compress image masks (PR #696).

    • Ensure that FITS files specify FITS-standard-compliant units (PR #673).

    • Integration test fixes (PR #695).

0.23.1 (2018-08-09)

  • Support STD/STD_FSTAR/STD_FAINT bit names (PR #674).

0.23.0 (2018-07-26)

  • Adds qproc algorithms and QFrame class (PR #664).

  • Adds desi_pipe go for production running (PR #666).

  • Increase job maxtime for edison realtime queue (PR #667).

  • Updates for running desispec on BOSS data (PR #669).

  • Fix QL for list vs. array change in specter/master (PR #670).

0.22.1 (2018-07-18)

  • Update processing of QL metrics (PR #659).

  • Refactor pipeline and integration test (PR #660).

  • Update redshift database to handle changes to fiberassign data model (PR #662).

  • Allow rows to be filtered when loading the redshift database (PR #663).

0.22.0 (2018-06-30)

This is the version used for mock observing in June 2018. It includes an update to the directory substructure where raw data are found, grouping each exposure into a separate directory $DESI_SPECTRO_DATA/{YEARMMDD}/{EXPID}/.

  • Faster traceshift code; requires numba (PR #634).

  • Fixed integration tests (PR #635).

  • Give empty HDUs am EXTNAME (PR #636).

  • Update redshift database loading in integration test (PR #638).

  • Integration test DB loading (PR #640).

  • Move ccd_calibration.yaml to SVN repo (PR #641).

  • Logging QA metric status for QLF (PR #642).

  • Supporting both new and old fibermap via io.read_fibermap (PP #643).

  • Faster lower memory preproc using numba (PR #644)

  • ivar bugfix in resample_flux interpolation (PR #646).

  • Many QL updates from mock observing (PR #648).

  • Raw data in NIGHT/EXPID/. instead of NIGHT/. (PR #648).

  • Fix cosmics masking near masked saturated pixels (PR #649).

  • Update edison realtime queue config to 25 nodes (PR #650).

  • trace_shift code supports PSF formats without “PSF” HDU (PR #654).

  • Change keyword clobber to overwrite in functions from astropy.io.fits (PR #658).

0.21.0 (2018-05-25)

Major updates including non-backwards compatible changes to QL output format and pipeline updates for semi-realtime nightly processing.

  • Pipeline fix to allow redrock to use a full node per healpix (PR #585).

  • Update pipeline maxtime/maxnodes job calculation (PR #588).

  • Better sync of pixel tasks and DB sync bugfixes (PR #590).

  • Improved handling of errors in case of full job failure (PR #592).

  • QA speedups and improvements (PR #593)

    • Add ability to load Frame without reading Resolution matrix

    • Refactor offline QA to use qaprod_dir more sensibly

    • Include hooks in QA to previous fiberflat file location (calib2d)

    • Inhibit scatter plot in skyredidual QA

  • Pass MAG into output zbest file (PR #595)

  • Allow running multiple task types in a single job (PR #601).

  • Pipeline hooks for processing a single exposure (PR #604).

  • Override PSF file psferr to avoid masking bright lines. Requires specter > 0.8.1 (PR #606).

  • QL QA reorganization (PR #577, #600, #607, #613).

  • Integration test and QA fixes (PR #602 and #605).

  • New desi_night scripts for semi-realtime processing (PR #609).

  • Spectro teststand calibration/utility code updates (PR #610)

  • QL S/N vs. mag updates (PR #611)

  • QL resampling fixes (PR #615)

  • Merge database modules (PR #616).

  • Add flexure tests to QL (PR #617).

  • Added cori and edison realtime queue support (PR #618, #619, #624).

  • QL output format updates (PR #623).

0.20.0 (2018-03-29)

Multiple non-backwards compatible changes:

  • Astropy 2 compatibility (PR #519).

  • Update Travis tests to recent versions.

  • Integration test fixes (PR #552).

  • Adds pipeline db count_task_states (PR #552).

  • Standardize spectro filenames/locations (PR #545 and #559).

  • Complete rewrite of task pipelining (PR #520, #523, #536, #537, #538, #540, #543, #544, #547, )

  • QL format updates (#517, #554)

  • module file set DESI_CCD_CALIBRATION_DATA (#564).

  • Optionally include RA,DEC in merged zcatalog (#562).

  • QL updates to S/N calculations (#556).

  • fix BUNIT, HPXNSIDE, HPXPIXEL keywords (PR #566)

0.19.0 (2018-03-01)

  • Update DB loading for desitarget 0.19.0 targets; make DB loading API less specific to datachallenge directory structure (PR #516).

0.18.0 (2018-02-23)

  • Replace deprecated scipy.stats.chisqprob with scipy.stats.distributions.chi2.sf for compatibility with scipy 1.0. (PR #503)

  • Faster desi_group_spectra that also propagates SCORES table (PR #505 and #507 )

  • Add options for fitting spatially non-uniform sky (PR #506)

  • Fix logger redirection (PR #508)

  • Add hooks for MPI extraction timing benchmarks (PR #509)

  • QuickLook metric renaming (PR #512)

0.17.2 (2018-01-30)

  • Trace shift optimizations from analyzing teststand data (PR #482).

  • Minor QA edits to accommodate minitest (PR #489)

  • Additional QA edits including qaprod_root() method (PR #490)

  • Introduce QA_Night, QA_MultiExp and refactor QA_Prod accordingly (PR #491)

  • Add SCORES HDU to frame files (PR #492)

0.17.1 (2017-12-20)

  • Refactors spectral regouping to be faster and derive fibermap format from inputs (PR #473).

  • Removed deprecated Brick class, and unused coadds and redmonder zfind that were using Bricks (PR #473).

  • Adds skyline QA; fixes QA version usage (PR #458).

  • Fixes write_bintable bug if extname=None; fixes missing header comments

  • spectro DB database loading updates (PR #477).

  • trace shift updates for fiber flats (PR #479).

  • Pipeline scaling updates (PR #459 and #466).

0.17.0 (2017-11-10)

  • Enabled specter.extract.ex2d nsubbundles option for faster extractions. Requires specter 0.8.1 (PR #451).

  • Fixed bug in desispec.parallel.dist_discrete() (PR #446)

  • Tuned pipeline for scaling tests (PR #457)

  • Improved wavelength fitting (via specex update) and sky model error propagation (PR #459)

  • Added QL fiberflat, py3 fixes, updated algorithms and config

  • Many other QL updates (PR #462)

  • Enables MPI parallelism for desi_extract_spectra script (PR #448)

0.16.0 (2017-09-29)

  • Small fixes to desi_qa_prod and qa_prod

  • Removes a number of QL metrics from offline qa

  • Fixes integration tests for desisim newexp refactor

  • Removes spectra grouping by brick; nside=64 healpix grouping default

  • Add get_nights method to io.meta (PR #422)

  • Add search_for_framefile method to io.frame (PR #422)

  • Add desi_qa_frame script to generate frame QA (PR #424)

  • Add frame_meta to parameters (for slurping the Frame headers) (PR #425)

  • Add get_reduced_frames() method to io.meta (PR #425)

  • Modifies QA_Prod meta file output to be JSON (PR #425)

  • Add load_meta() method to QA_Exposure (PR #425)

  • Add time_series ploting to desi_qa_prod (PR #425)

  • Add several new plots for skysub residuals (PR #425)

  • Adds method to generate QA Table for Prod (PR #425)

  • Refactor of skysubresid script (PR #425)

  • Refactor QA files to sit in their own folder tree (PR #429)

  • Generate HTML files with links to QA figures (PR #429)

  • Enable generation of Exposure level QA (PR #429)

  • Normalize fiberflat QA by fiber area (PR #429)

  • Fixed exptime in fluxcalib ZP calculation (PR #429)

  • Added find_exposure_night() method (PR #429)

  • Add MED_SKY metric to QA and bright/dark flag in desi_qa_prod

  • Update pipeline code for specex and redrock (PR #439 and #440)

  • Adds code for adjusting trace locations to match sky lines (PR #433)

  • Updates to DB loading (PR #431)

  • Adds pixelflat code (PR #426)

0.15.2 (2017-07-12)

  • Make the loading of libspecex through ctypes more robust and portable.

  • QL configuration cleanup (PR #389).

  • Add extrapolate option to resample_flux (PR #415).

  • Sphinx and travis tests fixes.

0.15.1 (2017-06-19)

  • Fixed desispec.io.findfile() path for zbest and coadd (PR #411).

  • Add Notebook tutorial: introduction to reading and manipulating DESI spectra (PR #408, #410).

  • Update quicklook configuration (PR #395).

  • Rename Spectra.fmap attribute to Spectra.fibermap (PR #407).

  • Enable desi_group_spectra to run without pipeline infrastructure (PR #405).

  • Update desispec.io.findfile spectra path to match dc17a (PR #404).

  • Load redshift catalog data from healpix-based zbest files (PR #402).

0.15.0 (2017-06-15)

  • Refactor database subpackage and enable loading of both quicksurvey and pipeline outputs (PR #400).

  • Clean up pipeline script naming to be grouped by night.

  • Modify pipeline to use Spectra objects grouped by HEALPix pixels instead of bricks. Add entry point to regroup cframe data by pixel (PR #394).

  • Add a new class, Spectra, which encapsulates a grouping of 1D spectra in one or more bands. Includes selection, updating, and I/O.

  • Removed desispec.brick as it’s now in desiutil.brick (PR #392).

  • Added function to calculate brick vertices at a given location (PR #388).

  • Added function to calculate brick areas at a given location (PR #384).

  • Add scripts for submitting nightly job chains.

  • Production creation now correctly handles slicing by spectrograph.

  • Pipeline job concurrency now computed based on task run time and efficient packing.

  • Set default brick size to 0.25 sq. deg. in desispec.brick (PR #378).

  • Added function to calculate BRICKID at a given location (PR #378).

  • Additional LOCATION, DEVICE_LOC, and PETAL_LOC columns for fibermap (PR #379).

  • Create util.py in tests/ which is intended to contain methods to facilitate test runs

  • Add vette() method for Frame class (PR #386)

  • Began a desispec parameter file: data/params/desispec_param.yml

  • Flux calibration improvements (PR #390).

0.14.0 (2017-04-13)

  • Replace all instances of desispec.log with desiutil.log; get_logger() now prints a warning that users need to switch.

  • Working DTS delivery script and DTS simulator (PR #367).

  • Preproc updates for crosstalk and teststand data (PR #370).

  • Flux calibration algorithm updates (PR #371).

  • Adds quicklook integration test (PR #361).

  • Fixes brickname calculation (PR #373).

0.13.2 (2017-03-27)

  • Add framework for DTS delivery and nightly processing scripts (PR #365).

  • Force documentation errors to cause Travis errors (PR #364).

0.13.1 (2017-03-03)

  • Fix installation of data/ccd/ccd_calibration.yaml.

0.13.0 (2017-03-03)

  • Fix brick update corruption (PR #314).

  • Close PSF file after initializing PSF object.

  • Refactor desispec.io.database to use SQLAlchemy.

  • Fix graph_path() usage in workers.

  • Update desispec.io.raw.write_raw() to enable writing simulated raw data with new headers.

  • Allow test_bootcalib to run even if NERSC portal is returning 403 errors.

  • Add bricksize property to desispec.brick.Bricks; allow desispec.brick.Bricks.brickname to specify bricksize.

  • Do SVD inverses when cholesky decompositions fail in fiberflat, sky subtraction, and flux calibration.

  • Algorithm updates for teststand and BOSS data

  • pipeline updates for docker/shifter

  • quicklook updates

0.12.0 (2016-11-09)

  • Update integration test to use stdstar_templates_v1.1.fits.

  • Support asymmetric resolution matrices (PR #288).

  • Quicklook updates (PR #294, #293, #285).

  • Fix BUNIT and wavelength f4 versus f8.

  • Significant pipeline code refactor (PR #300 and #290).

  • fix docstrings for sphinx build (PR #308).

0.11.0 (2016-10-14)

  • Update template Module file to reflect DESI+Anaconda infrastructure.

  • Update redmonster wrapper for reproducibility.

  • desispec.io.brick.BrickBase.get_target_ids returns target IDs in the order they appear in input file.

  • Set BUNIT header keywords (PR #284).

  • Improved pipeline logging robustness.

  • MPI updates for robustness and non-NERSC operation.

  • More py3 fixes.

0.10.0 (2016-09-10)

PR #266 update for Python 3.5:

  • Many little updates to work for both python 2.7 and 3.5.

  • Internally fibermap is now a Table instead of FITS_rec table.

  • Bug fix for flux calibration QA.

  • Requires desiutil >= 1.8.0.

0.9.0 (2016-08-18)

PR #258 (requires specter >= 0.6.0)

  • Propagate pixel model goodness of fit to flag outliers from unmasked cosmics.

  • desi_extract_spectra –model option to output 2D pixel model

  • fix pipeline bug in call to desi_bootcalib (no –qafig option)

  • adds extraction tests

Misc:

  • desi_qa_skysub – plots residuals (PR #259)

  • More quicklook QA (PR #260 and #262)

  • Added support for template groups in redmonster (PR #255)

  • Lots more pipeline docs (PR #261)

0.8.1 (2016-07-18)

  • added QA_Prod

  • refactor of fluxcalib QA

  • fixed pipeline QA figure output (pdf vs. yaml)

0.8.0 (2016-07-14)

  • bootcalib robustness improvements

  • improved fibermap propagation

  • PRODNAME -> SPECPROD, TYPE -> SPECTYPE

  • meaningful batch job names for each step

  • better test coverage; more robust to test data download failures

  • more quicklook metrics

  • used for “oak1” production

0.7.0 and prior

  • No changes.rst yet