Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create report on pangeo-forge bakery vs. regular rechunking workflow #383

Closed
amsnyder opened this issue Oct 17, 2023 · 4 comments
Closed
Assignees

Comments

@amsnyder
Copy link
Contributor

  • How does the workflow for each compare? (standardization, level of effort, format, etc.)
  • What cloud resources will we be using to deploy the bakery? (this can help us estimate cost)
  • What code/repos do we need to build this? How will we need to adapt/modify each of these?
@amsnyder
Copy link
Contributor Author

amsnyder commented Nov 9, 2023

pangeo-forge bakery has standardized template for use to fill in to rechunk a dataset

@thodson-usgs
Copy link
Member

thodson-usgs commented Nov 13, 2023

In principle, the pangeo-forge tooling seems very useful if we can jump the starting hurdle.
I'm working on a recipe to process SSEBOP data for @kjdoore.
The recipe was easy but I'm still working through issues

@thodson-usgs
Copy link
Member

Still stuck, so I create a recipe PR to pangeo-forge. I'll follow up in their coordination meeting on Monday.

@thodson-usgs
Copy link
Member

The SSEBOP recipe seems to be working now, thanks to some assistance from the pangeo-forge community.

All-in-all the recipes are fairly straightforward. Debugging Beam pipelines was challenging but I suspect that will get easier with time.

Pros: community of experts willing to help (esp if we contribute); recipes run a variety of "runners"; easier to manage large projects

Cons: Beam is harder to use than a notebook; Pangeo-forge documentation is incomplete; may create some technical lock-in or necessitate external support

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants