Tracking issue for finished migration to public GCS bucket.

[jbusecke]

Once we moved the ingestion to the public GCS bucket the instructions in the README should be updated and this issue closed.

Some steps to track the evolution of this milestone. We are getting VERY CLOSE.

• Implement mover stage (Add mover/copy stage #67)

• Test access to public bucket from mover stage (Change move bucket #87)

• Refactor all database content to a single bigquery table:

  • Dump and backup various csv files from temp table (Refactor csv creation from single bigquery table #89)
  • Test ingestion logging to consolidated table (Consolidate database into single BQ table #90)
    • Tested manually
    • Tested with subset of ingestion (Consolidate database into single BQ table #90 merged)
  • Implement retractions to temp table
    • Developed way faster prototype 🚀
    • New action and script (Add retraction workflow + script #91) This just found 28k+ newly retracted stores and did not take 4 hours 🤪. It also generates clickable html reports about which stores were retracted ✨

• Test and copy LEAP stores (one time action, info -> 'leap-pangeo.cmip6_pgf_ingestion.leap_legacy'

• Test PANGEO legacy stores (one time action, info -> 'leap-pangeo.cmip6_pgf_ingession.pangeo_legacy'

• Add all legacy entries to beta test table "leap-pangeo.testcmip6.cmip6_consolidated_manual_testing"

• Set up revised json catalog files
  Beta version of catalog available here:

  import intake
  url = "https://storage.googleapis.com/cmip6/cmip6-pgf-ingestion-test/catalog/catalog.json" # Only stores that pass current tests
  # url = "https://storage.googleapis.com/cmip6/cmip6-pgf-ingestion-test/catalog/catalog_noqc.json" # Only stores that fail current tests
  # url = "https://storage.googleapis.com/cmip6/cmip6-pgf-ingestion-test/catalog/catalog_retracted.json" # Only stores that have been retracted by ESGF
  col = intake.open_esm_datastore(url)
  

• Get some folks to beta test.

  • Announcement added to the README

• Rebase on leap-data-management-utils release (Use new release of leap-data-management-utils #85)

• Implement consolidated coordinates (Test consolidates dim coords from PR #88)

• Optional/Later: Consolidate coordinates of LEAP legacy stores?

• Setup final table and repeat ingestion from legacy bq tables.

  • Change all target prefixes to the final location
  • Ingest data from 'leap-pangeo.cmip6_pgf_ingestion.leap_legacy'
  • Ingest data from 'leap-pangeo.cmip6_pgf_ingession.pangeo_legacy'
  • Redirect to proper csv in json catalog files.

• Delete and clean temp table/directories and populate final table from scratch.

  • Save/move reports
  • cmip6/cmip6-pgf-ingestion-test/catalogs/
  • cmip6/cmip6-pgf-ingestion-test/
  • cmip6/CMIP6_LEAP_legacy_test
  • Delete all the permanent stores on LEAP buckets

• Remove retraction logic from https://github.com/pangeo-data/pangeo-cmip6-cloud/blob/master/create_filtered_catalog.py and document new location there.

• Adopt new catalog in xMIP

• Write a Blogpost about this

• Dance in the streets 🕺

• (Optional/Later) Implement regular retesting of all stores

  • Tune dask performance
  • Speed up testing with coiled?

[jbusecke]

#67 Made substantial progress towards this goal.