HPC-Cloud-Subsetting

Project tracking repository for the NOAA HPC Subsetting Model Data in the Cloud project

Targets for End of Project

xarray_subset_grid repository is fully functional for ROMS and FVCOM datasets, contains examples and tests that prove the subsettings method is accurate.
xreds subset API is capable of delivering subsets 500 MB and below as netCDF, and any size as zarr.

Ingest NFOGS2
- Add mesh_topology to the zarr datasets so they can be loaded consistently with ugrid and cf
Ingest WCOFS

Create example subsetting notebooks for FVCOM and ROMS using NODD data with Xarray
- FVCOM
- ROMS
Create python package for subsetting model data with their native grids using Xarray
- https://github.com/asascience-open/xarray-subset-grid
Create Xpublish plugin API to subset datasets on demand in the cloud
Create custom Xpublish based service including all required API plugins (including the subsetting plugin)
Create simple GUI for creating spatiotemporal subsets using the service
Deploy demo functionality with existing XREDS service
- https://nextgen-dev.ioos.us/xreds/subset_export
Create example workflow scripts and notebooks utilizing the service
Create API for reusing slicing logic
Create test and validation suites for python package
Benchmark subetting performance using test and validation cases
Create full package documentation using python best practices
Add CI-CD
Vertical Subset API
- Subset Level and Levels
- Top Vertical Accessor
- Bottom Vertical Accessor
- Surface Vertical Accessor
Add python package deployment integration with github with deployment to pypi
Optimize performance in subsetting package
- Dataset subsetting methodology and algorithm performance
Expand to other grid types
- ADCIRC
- SELFE (Untested, but should work, CREOFS)
- SCHISM
- HYCOM (RTOFS)
- Regular Grid (Test with GFS NOMADS OpenDAP)
- 2D Grid (Test with NAM Conus OpenDAP)