CASDA Release Notes

CASDA v1.25: 25 June 2024

Search

• Further UX enhancements to ATOA and ASKAP searches
• Add extra fields to ATOA search results
• ATOA – Allow search on array config, frontend and backend
• Link each file in search results to the data collection

Data Access

• Allow cutouts from Galactic cubes
• Update to Carta v4.1
• Add support for accessing data from multiple object stores
• Switch to CSIRO single sign-on for admin access
• ATOA – Make all data available to cone search
• Improve association of data access requests with the requesting user
• Correct cutout size in curated cutouts

VO Tools

• Add support for PostgreSQL v14
• Allow configuration of some datalink details

Deposit

• Add support for depositing data to multiple object stores
• Allow dynamic control of job concurrency limits
• Switch to CSIRO single sign-on for admin access
• Limit number of actively depositing observations at a time
• Speed up deposit of small files

CASDA v1.24: 22 February 2024

Search

• Group all astronomy searches under the Australia Telescope National Facility (ATNF) domain
• Add ATOA observation search interface (not available until data is available)
• Pulsar Observation Search & Results enhancements based on UX review
• Level 7 data products automatically shown as released when embargo ends
• Add facility to show data as unavailable

Data Access

• Record data access metrics

VO Tools

• obscore obs_id changed from sbid to telescope-sbid e.g. ASKAP-12345

Deposit

• Add support for multiple telescopes (for ATOA)
• Create per semester collections for ATCA, Parkes and LBA observations
• Apply 18 month embargo to all ATCA, Parkes and LBA observations
• Reject validation metric values with NaN values
• Output parsing errors to ERROR file
• Require comments when validating data as BAD and provide warning message

CASDA v1.23: 13 October 2023

Search

• Navigation Improvements
• Landing Page for CASDA
• Observation Search & Results enhancements based on UX review
• Skymap enhancements based on UX review
• Upgrade Aladin Lite to v3
• Add Obs Duration to search results

Data Access

• Visualisation of images and image cubes with CARTA
• Direct cutouts from the Pawsey object storage, Acacia

VO Tools

• Correct date format in TAP (UWS) job responses

Deposit

• Validation task enhancements based on UX review
• Preparation for depositing data from other telescopes
• Improvements to deposits of large derived data sets
• Improve handling of large numbers of spectra files

CASDA v1.22:  22 June 2023

Search

• Header navigation Improvements
• Offer sign-in on session timeout
• Improve navigation when sign-in required
• Update to ElasticSearch 7
• Show selected search criteria on the results page
• Correct column sorting
• Allow access again to info dialog for rejected data products
• Add RACS-mid as a background in the SkyMap page

Data Access

• Append beam and other extra tables in images to generated cutouts
• Parallel processing of cutouts in montage

VO Tools

• Improvements to robustness of configuration
• Allow results directory to be configured

Deposit

• Validation – allow quality selection by type and/or sub-type
• Validation – allow adjustment of quality and notes to be added after release
• Validation – general usability improvements
• Add default contributor to observational collections
• Advise admins of failed deposits
• Containerisation of the deposit functions
• Move of deposit functions to Setonix application node
• Allow deposits from Setonix /scratch
• Set quality of all derived data to be good

CASDA v1.21:  21 February 2023

Search

• Observation search – Allow positive declination values without explicit “+” sign
• Observation Search – Fix sort by type for spectra

Data Access

• Skymap – Add RACS and WALLABY catalogues
• Cutout Service – UI improvements
• Image Cutout – FoV too big for cube which crosses the prime meridian
• Fix failure to cancel cutout job under specific circumstances

VO Tools

• SIA2 – cone search – Add distance from image cube centre
• Cleanup UWS error logs
• Cleanup old VO Tools result files
• Allow configuration of the results directory
• SIA1 – correct validation errors

Deposit

• Improve loading and saving speed of validation tasks
• Deposit – Allow an unreleased observation to be deleted
• Establish policies to make data public in S3
• Improvement to metadata sent to DataCite for DOIs
• Fix an issue where duplicate release events could be created
• Fix an issue where validation tasks might not be generated for all projects in a multi-commensal SBID

CASDA v1.20:  21 Sep 2022

Search

• Add new CASDA Cutout Service page for producing a cutout from curated survey data
• Observation search – fix error when creating cutout from a cone search result
• Observation search – cutout service now uses RA, Dec and cutout radius, instead of box corners

Data Access

• Add support for Simple Image Access v1
• Update checksum file names to match data file names

VO Tools

• Add support for Simple Image Access v1

Deposit

• Provide extra metadata to Datacite when registering DOIs
• Move from /group to /askapbuffer for observational data deposits
• Add support for unicode characters in derived data deposits
• Remove use of NGAS

Server Migration

• Move database to new Nimbus virtual machines
• Move Data Access and VO Tools to new Nimbus virtual machines

CASDA v1.19:  2 June 2022

Search

• Show derived images on the images tab
• Correct search by velocity

Data Access

• Fix for CASDA incorrectly showing as unavailable after outage

Deposit

• Stability improvements to the derived data deposit process
• Add support for linked files in derived data
• Carry files forward from the earlier version when a derived collection is updated
• Add support for NaN values in derived catalogues
• Verify files in observation.xml are not directories

S3 Data Migration

• Completion of migration of all data to the Acacia object storage system at Pawsey

CASDA v1.18:  10 February 2022

Search:

• Show catalogue filenames in search result (delayed for DAP 2.34 release)

Data Access

• Add rejected events to the observation event list showing when an entire observation is not released
• Hide rejected data from SSA
• Bug fixes for accessing data in object storage

Deposit

• Make the level 7 deposit process more robust (delayed for DAP 2.34 release)
• Fix issue where updated observation events were not being recorded for some redeposits
• Populate start and stop time fields (t_min and t_max) in obscore for image cubes

S3 Data Migration

• Correct indexing of data after migration
• Add support for multiple servers to migrate data

VO Tools:

• Update log4j to v2.17.1

CASDA v1.17:  16 December 2021

VO Tools:

• Register the RACS catalogues and their SCS interfaces in our VO registry entries
• Datalink – Correct handling of future release dates
• Datalink – Ensure Astroquery uses public services, not Pawsey only ones
• Correct handling of column metadata for columns with a different name in TAP vs database
• Added a CASDA VO Tools server header
• Improve TAP standards compliance

Deposit

• Complete support for depositing data to object storage

Data Access

• Complete support for accessing data in object storage
• Tidy up validation report URLs
• Skymap – correct previews for data with spectral axes

S3 Data Migration

• Add support for migrating low priority data from Banksia to Acacia
• Add support for increased storage for high-priority data for initial transfer to Acacia

CASDA v1.16:  21 October 2021

Search:

• Make the cut-out service more obvious in the new UI.
• Include directions on creating OPAL accounts.
• Quick access to larger images from search results.
• Mark files that are in the process of migration to S3 as temporarily unavailable for download.
• Improve paging in cone search results

Deposit:

• Enable multiple migration jobs to run in parallel.
• Level 7 – Update release dates when a metadata only update is published.

CASDA v1.15:  6 September 2021

Deposit:

• Implement migration process to migrate all CASDA data to Pawsey’s new object storage system Acacia
• Commence preparing high priority data for migration
• Allow unicodeChar type in metrics files
• Fix issue where duplicate encapsulation files could be created
• Fix issue where redeposits would fail if catalogues from the original deposit were no longer on the lustre filesystem
• Preparation for depositing files into object storage

Data Access:

• Preparation for access to files in object storage

VO Tools:

• Correct validation errors: E_TSTD – TAP_SCHEMA column not flagged Std

CASDA v1.14:  1 July 2021

Search:

• Add commensal project search option

Deposit:

• Support for recording commensal projects for data products in ‘formally commensal’ observations
• Add support for deposit of large derived catalogues
• Add partial release option to the release task – this will reject bad data and release all other data

Data Access:

• Add RACS as the default layer to the skymap display
• Enable selected DOI to be easily copied when on the data tab of a CASDA collection
• Correct zoom on skymap object preview

VO Tools:

• Correct SIA2 capabilities endpoint missing accessURL

CASDA v1.13: 12 Nov 2020

Search:

• Correct issue where data products were still showing as unreleased after their embargo period had ended
• Only show published derived data products in search results

Deposit:

• Move validate and release tasks for observations to the new user interface
• Correct issue where older catalogues would fail to be released
• Add new background image types
• Link catalogue to their image cube by file name only, ignore the path listed
• Provide script to allow bulk validation and release of scheduling blocks

Data Access:

• Provide a new observation event list and query facility
• Ensure images and cubes for derived collections are shown on the images tab only

VO Tools:

• Correct output of large calculated numeric values

CASDA v1.12:  9 July 2020

Search:

• Highlight primary image data products in search results
• Add derived catalogues and data products to the skymap search
• Allow multiple facet values to be selected to filter search results
• Hide rejected data products by default
• Access validation reports from data product info page
• Allow both released and unreleased data to be displayed in a search
• Accept decimal and sexagesimal coordinates in all entry locations

Deposit:

• Public web hosting of validation reports produced by the ASKAPSoft pipeline
• Allow derived (level 7) collections to contain catalogues, images and spectral data products in the one collection
• Speed up deposit of many small files
• Only add catalogues to global catalogues when released
• Handle deposits with an unknown project code more gracefully
• Add lower limit for RESTFREQ from ASKAPsoft
• Allow plus signs in file names
• Hide already rejected data products on validation and release pages
• Correct interaction of observation metadata refresh and redeposit
• Correct problem handling spectra files with very small footprints

Data Access:

• Extend CASDA astroquery module to support staging and download of data products
• Add channel selection for SODA
• Download original catalogues rather than regenerate from global catalogues
• Update to Montage v6 for cutout processing
• Show the file name on the image generation page
• Improve error messages for SODA requests
• Correct issue preventing old level 7 data products from being downloaded via the UI or sync SODA download requests

VO Tools:

• Correct s_resolution values for image cubes in obscore
• Correct error in tap_tables config preventing new installations

CASDA v1.11a:  30 August 2019

Search:

• ASVO common authentication – can now login to DAP using AAO Data Central Lens account
• Restore coverage map to new user interface
• Fix problems viewing previews of sources in the tucana image.

Data Access:

• Astroquery module for querying ASKAP data
• cutouts_channels.py script in CASDA github repository, which divides whole ASKAP cube in to channels before downloading the sub-cubes
• Resolved issue with job queue stalling

CASDA v1.11:  30 July 2019

Search:

• Include visibilities in cone search results
• Correct issue with rest frequency selection
• Fix error revising cone search criteria

Deposit:

• Allow project admins to manage membership of their project teams
• Use original names for files on the validation pages
• Hide already released data products on validation and release pages

Data Access:

• Use original names for downloaded data files
• Increased the download limit per job to 500GB
• Allow panning on cutout selection map
• Allow cutouts from cubes crossing 0 RA
• Allow cutouts of entire planes of a cube
• Snap selected cutout region to bounds of cube
• Evaluation files for download now separate into i) ASKAP pipeline logs and calibration table data and ii) quality assessment pipeline output.
• Update UCD for SODA ID parameter (SODA v1.0 erratum #1)

VO Tools:

• Add support for ObsCore v1.1
• Provide more flexibility for the locations of the VO tools config
• Add actual file name to obscore
• Add support for embargoed catalogues
• Encrypt admin password when stored