Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attempt at fixing aiohttp timeouts #3

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Attempt at fixing aiohttp timeouts #3

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions feedstock/recipe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import aiohttp
import pandas as pd
from pangeo_forge_recipes.patterns import FilePattern, ConcatDim
from pangeo_forge_recipes.patterns import ConcatDim, FilePattern
from pangeo_forge_recipes.recipes import XarrayZarrRecipe


Expand All @@ -9,11 +10,12 @@ def ics_wind_speed_direction(ds, fname):
direction for the u and v components in the specified product. Dask arrays are
created for delayed execution.
"""
from datetime import datetime

import dask
import dask.array as da
from datetime import datetime
from metpy.calc import wind_direction, wind_speed
import xarray as xr
from metpy.calc import wind_direction, wind_speed

@dask.delayed
def delayed_metpy_fn(fn, u, v):
Expand Down Expand Up @@ -136,9 +138,12 @@ def make_url(time):
# Daily products with 6-hourly values
NITEMS_PER_FILE = 4

# need to override the aiohttp timeout
# # https://github.com/pangeo-forge/eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock/issues/2
pattern = FilePattern(
make_url,
ConcatDim(name="time", keys=dates, nitems_per_file=NITEMS_PER_FILE),
fsspec_open_kwargs={"client_kwargs": {"timeout": aiohttp.ClientTimeout(total=1800)}},
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martindurant, i could be wrong but it appears that pangeo-forge-recipes doesn't know how to serialize this

Oct 31 17:09:48 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892001432727571&selected=1523892001432727571) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892001432727571&selected=1523892001432727571) 2022-11-01 00:09:47,835 DEBUG - orchestrator - Running command: ['pangeo-forge-runner', 'bake', '--repo=https://github.com/pangeo-forge/eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock', '--ref=1fd4c64a07d3e6f5caf247ca60784774c45e8740', '--json', '--prune', '--Bake.recipe_id=eooffshore_ics_ccmp_v02_1_nrt_wind', '-f=/tmp/tmpz2sirbhz.json']
Oct 31 17:09:50 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892012337860611&selected=1523892012337860611) [heroku/web.1](https://my.papertrailapp.com/events?q=program%3Aheroku%2Fweb.1&focus=1523892012337860611&selected=1523892012337860611) Process running mem=517M(101.1%)
Oct 31 17:09:50 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892012337860612&selected=1523892012337860612) [heroku/web.1](https://my.papertrailapp.com/events?q=program%3Aheroku%2Fweb.1&focus=1523892012337860612&selected=1523892012337860612) Error R14 (Memory quota exceeded)
Oct 31 17:09:53 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892025315627008&selected=1523892025315627008) [app/heroku-postgres](https://my.papertrailapp.com/events?q=program%3Aapp%2Fheroku-postgres&focus=1523892025315627008&selected=1523892025315627008) source=DATABASE addon=postgresql-fitted-67704 sample#current_transaction=2721 sample#db_size=9392687bytes sample#tables=4 sample#active-connections=11 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99991 sample#table-cache-hit-rate=0.99996 sample#load-avg-1m=0.02 sample#load-avg-5m=0.015 sample#load-avg-15m=0.025 sample#read-iops=0 sample#write-iops=0.076336 sample#tmp-disk-used=543600640 sample#tmp-disk-available=72435191808 sample#memory-total=8038316kB sample#memory-free=3837384kB sample#memory-cached=3643416kB sample#memory-postgres=22084kB sample#wal-percentage-used=0.06477745905179602
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692232&selected=1523892050944692232) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692232&selected=1523892050944692232) 2022-11-01 00:09:59,645 ERROR - orchestrator - Recipe run  failed with: Traceback (most recent call last):
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692233&selected=1523892050944692233) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692233&selected=1523892050944692233)   File "/usr/local/bin/pangeo-forge-runner", line 8, in <module>
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692234&selected=1523892050944692234) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692234&selected=1523892050944692234)     sys.exit(main())
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692235&selected=1523892050944692235) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692235&selected=1523892050944692235)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/cli.py", line 28, in main
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692236&selected=1523892050944692236) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692236&selected=1523892050944692236)     app.start()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692237&selected=1523892050944692237) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692237&selected=1523892050944692237)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/cli.py", line 23, in start
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692238&selected=1523892050944692238) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692238&selected=1523892050944692238)     super().start()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692239&selected=1523892050944692239) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692239&selected=1523892050944692239)   File "/usr/local/lib/python3.9/dist-packages/traitlets/config/application.py", line 464, in start
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692240&selected=1523892050944692240) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692240&selected=1523892050944692240)     return self.subapp.start()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692241&selected=1523892050944692241) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692241&selected=1523892050944692241)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/commands/bake.py", line 125, in start
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692242&selected=1523892050944692242) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692242&selected=1523892050944692242)     recipes = feedstock.parse_recipes()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692244&selected=1523892050944692244) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692244&selected=1523892050944692244)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/feedstock.py", line 58, in parse_recipes
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692245&selected=1523892050944692245) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692245&selected=1523892050944692245)     recipes[r["id"]] = self._import(r["object"])
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692246&selected=1523892050944692246) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692246&selected=1523892050944692246)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/feedstock.py", line 43, in _import
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692247&selected=1523892050944692247) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692247&selected=1523892050944692247)     exec(compile(source=f.read(), filename=filename, mode="exec"), ns)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692248&selected=1523892050944692248) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692248&selected=1523892050944692248)   File "/tmp/tmp7c2oaxz7/feedstock/recipe.py", line 151, in <module>
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692249&selected=1523892050944692249) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692249&selected=1523892050944692249)   File "<string>", line 22, in __init__
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692250&selected=1523892050944692250) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692250&selected=1523892050944692250)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 798, in __post_init__
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692251&selected=1523892050944692251) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692251&selected=1523892050944692251)     super().__post_init__()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692252&selected=1523892050944692252) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692252&selected=1523892050944692252)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/recipes/base.py", line 22, in __post_init__
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692253&selected=1523892050944692253) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692253&selected=1523892050944692253)     self.sha256 = dataclass_sha256(self, ignore_keys=self._hash_exclude_)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692254&selected=1523892050944692254) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692254&selected=1523892050944692254)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 75, in dataclass_sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692255&selected=1523892050944692255) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692255&selected=1523892050944692255)     return dict_to_sha256(d)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692256&selected=1523892050944692256) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692256&selected=1523892050944692256)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 36, in dict_to_sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692257&selected=1523892050944692257) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692257&selected=1523892050944692257)     b = dumps(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692258&selected=1523892050944692258) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692258&selected=1523892050944692258)   File "/usr/lib/python3.9/json/__init__.py", line 234, in dumps
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692260&selected=1523892050944692260) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692260&selected=1523892050944692260)     return cls(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692261&selected=1523892050944692261) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692261&selected=1523892050944692261)   File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692262&selected=1523892050944692262) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692262&selected=1523892050944692262)     chunks = self.iterencode(o, _one_shot=True)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692263&selected=1523892050944692263) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692263&selected=1523892050944692263)   File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692264&selected=1523892050944692264) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692264&selected=1523892050944692264)     return _iterencode(o, 0)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692265&selected=1523892050944692265) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692265&selected=1523892050944692265)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 19, in either_encode_or_hash
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692266&selected=1523892050944692266) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692266&selected=1523892050944692266)     elif hasattr(obj, "sha256"):
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692267&selected=1523892050944692267) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692267&selected=1523892050944692267)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/patterns.py", line 242, in sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692268&selected=1523892050944692268) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692268&selected=1523892050944692268)     return pattern_blockchain(self).pop(-1)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692269&selected=1523892050944692269) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692269&selected=1523892050944692269)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/patterns.py", line 313, in pattern_blockchain
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692270&selected=1523892050944692270) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692270&selected=1523892050944692270)     root_sha256 = dict_to_sha256(root_drop_empty)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692271&selected=1523892050944692271) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692271&selected=1523892050944692271)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 36, in dict_to_sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692272&selected=1523892050944692272) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692272&selected=1523892050944692272)     b = dumps(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692273&selected=1523892050944692273) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692273&selected=1523892050944692273)   File "/usr/lib/python3.9/json/__init__.py", line 234, in dumps
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692274&selected=1523892050944692274) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692274&selected=1523892050944692274)     return cls(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692275&selected=1523892050944692275) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692275&selected=1523892050944692275)   File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692276&selected=1523892050944692276) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692276&selected=1523892050944692276)     chunks = self.iterencode(o, _one_shot=True)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692277&selected=1523892050944692277) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692277&selected=1523892050944692277)   File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692278&selected=1523892050944692278) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692278&selected=1523892050944692278)     return _iterencode(o, 0)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692279&selected=1523892050944692279) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692279&selected=1523892050944692279)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 25, in either_encode_or_hash
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692280&selected=1523892050944692280) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692280&selected=1523892050944692280)     raise TypeError(f"object of type {type(obj).__name__} not serializable")
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692281&selected=1523892050944692281) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692281&selected=1523892050944692281) TypeError: object of type ClientTimeout not serializable
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692282&selected=1523892050944692282) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692282&selected=1523892050944692282) [2022-11-01 00:09:59 +0000] [66] [ERROR] Exception in ASGI application
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886528&selected=1523892050948886528) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886528&selected=1523892050948886528) Traceback (most recent call last):
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886529&selected=1523892050948886529) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886529&selected=1523892050948886529)   File "/opt/app/pangeo_forge_orchestrator/routers/github_app.py", line 619, in run
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886530&selected=1523892050948886530) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886530&selected=1523892050948886530)     out = subprocess.check_output(cmd)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886531&selected=1523892050948886531) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886531&selected=1523892050948886531)   File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886532&selected=1523892050948886532) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886532&selected=1523892050948886532)     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886533&selected=1523892050948886533) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886533&selected=1523892050948886533)   File "/usr/lib/python3.9/subprocess.py", line 528, in run
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886534&selected=1523892050948886534) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886534&selected=1523892050948886534)     raise CalledProcessError(retcode, process.args,
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886535&selected=1523892050948886535) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886535&selected=1523892050948886535) subprocess.CalledProcessError: Command '['pangeo-forge-runner', 'bake', '--repo=https://github.com/pangeo-forge/eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock', '--ref=1fd4c64a07d3e6f5caf247ca60784774c45e8740', '--json', '--prune', '--Bake.recipe_id=eooffshore_ics_ccmp_v02_1_nrt_wind', '-f=/tmp/tmpz2sirbhz.json']' returned non-zero exit status 1.
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886537&selected=1523892050948886537) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886537&selected=1523892050948886537) During handling of the above exception, another exception occurred:

let me know if any hints/workarounds come to mind

Copy link

@derekocallaghan derekocallaghan Nov 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @andersy005, this looks like a similar problem to the recent process_input and process_chunk serialization issue.

Two possible workarounds:

  1. Although the aiohttp.ClientSession read_timeout and conn_timeout parameters are deprecated in favour of timeout used in this PR commit, maybe they can be used as a short-term workaround as they accept floats (rather than aiohttp.ClientTimeout used for timeout), and should be serializable.
  2. FilePattern has its own sha256() implementation, and although fsspec_open_kwargs is currently included, perhaps this could be excluded (similar to the pattern format_function and combine dims currently excluded), or particular kwargs (e.g. any aiohttp instances) could be excluded from fsspec_open_kwargs prior to serialization. I guess a timeout value is something that isn't necessary for a recipe hash?

https://github.com/pangeo-forge/pangeo-forge-recipes/blob/3441d94a290d296b8f62638df15bb60993e86b1d/pangeo_forge_recipes/patterns.py#L299

    # we exclude the format function and combine dims from ``root`` because they determine the
    # index:filepath pairs yielded by iterating over ``.items()``. if these pairs are generated in
    # a different way in the future, we ultimately don't care.
    root = {
        "fsspec_open_kwargs": pattern.fsspec_open_kwargs,
        "query_string_secrets": pattern.query_string_secrets,
        "file_type": pattern.file_type,
        "nitems_per_file": {
            op.name: op.nitems_per_file  # type: ignore
            for op in pattern.combine_dims
            if op.name in pattern.concat_dims
        },
    }

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @andersy005, it looks like @yuvipanda is having a similar aiohttp hash problem in the daymet staged recipe as he's using aiohttp.BasicAuth in fsspec_open_kwargs["client_kwargs"]. More details here.

)

target_chunks = {"time": 8000, "latitude": -1, "longitude": -1}
Expand Down