Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attempt at fixing aiohttp timeouts #3

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Attempt at fixing aiohttp timeouts #3

wants to merge 1 commit into from

Conversation

andersy005
Copy link
Member

@andersy005
Copy link
Member Author

/run eooffshore_ics_ccmp_v02_1_nrt_wind

andersy005
andersy005 commented Nov 1, 2022
View reviewed changes
feedstock/recipe.py
pattern = FilePattern(
make_url,
ConcatDim(name="time", keys=dates, nitems_per_file=NITEMS_PER_FILE),
fsspec_open_kwargs={"client_kwargs": {"timeout": aiohttp.ClientTimeout(total=1800)}},
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martindurant, i could be wrong but it appears that pangeo-forge-recipes doesn't know how to serialize this

Oct 31 17:09:48 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892001432727571&selected=1523892001432727571) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892001432727571&selected=1523892001432727571) 2022-11-01 00:09:47,835 DEBUG - orchestrator - Running command: ['pangeo-forge-runner', 'bake', '--repo=https://github.com/pangeo-forge/eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock', '--ref=1fd4c64a07d3e6f5caf247ca60784774c45e8740', '--json', '--prune', '--Bake.recipe_id=eooffshore_ics_ccmp_v02_1_nrt_wind', '-f=/tmp/tmpz2sirbhz.json']
Oct 31 17:09:50 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892012337860611&selected=1523892012337860611) [heroku/web.1](https://my.papertrailapp.com/events?q=program%3Aheroku%2Fweb.1&focus=1523892012337860611&selected=1523892012337860611) Process running mem=517M(101.1%)
Oct 31 17:09:50 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892012337860612&selected=1523892012337860612) [heroku/web.1](https://my.papertrailapp.com/events?q=program%3Aheroku%2Fweb.1&focus=1523892012337860612&selected=1523892012337860612) Error R14 (Memory quota exceeded)
Oct 31 17:09:53 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892025315627008&selected=1523892025315627008) [app/heroku-postgres](https://my.papertrailapp.com/events?q=program%3Aapp%2Fheroku-postgres&focus=1523892025315627008&selected=1523892025315627008) source=DATABASE addon=postgresql-fitted-67704 sample#current_transaction=2721 sample#db_size=9392687bytes sample#tables=4 sample#active-connections=11 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99991 sample#table-cache-hit-rate=0.99996 sample#load-avg-1m=0.02 sample#load-avg-5m=0.015 sample#load-avg-15m=0.025 sample#read-iops=0 sample#write-iops=0.076336 sample#tmp-disk-used=543600640 sample#tmp-disk-available=72435191808 sample#memory-total=8038316kB sample#memory-free=3837384kB sample#memory-cached=3643416kB sample#memory-postgres=22084kB sample#wal-percentage-used=0.06477745905179602
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692232&selected=1523892050944692232) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692232&selected=1523892050944692232) 2022-11-01 00:09:59,645 ERROR - orchestrator - Recipe run  failed with: Traceback (most recent call last):
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692233&selected=1523892050944692233) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692233&selected=1523892050944692233)   File "/usr/local/bin/pangeo-forge-runner", line 8, in <module>
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692234&selected=1523892050944692234) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692234&selected=1523892050944692234)     sys.exit(main())
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692235&selected=1523892050944692235) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692235&selected=1523892050944692235)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/cli.py", line 28, in main
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692236&selected=1523892050944692236) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692236&selected=1523892050944692236)     app.start()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692237&selected=1523892050944692237) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692237&selected=1523892050944692237)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/cli.py", line 23, in start
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692238&selected=1523892050944692238) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692238&selected=1523892050944692238)     super().start()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692239&selected=1523892050944692239) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692239&selected=1523892050944692239)   File "/usr/local/lib/python3.9/dist-packages/traitlets/config/application.py", line 464, in start
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692240&selected=1523892050944692240) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692240&selected=1523892050944692240)     return self.subapp.start()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692241&selected=1523892050944692241) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692241&selected=1523892050944692241)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/commands/bake.py", line 125, in start
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692242&selected=1523892050944692242) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692242&selected=1523892050944692242)     recipes = feedstock.parse_recipes()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692244&selected=1523892050944692244) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692244&selected=1523892050944692244)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/feedstock.py", line 58, in parse_recipes
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692245&selected=1523892050944692245) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692245&selected=1523892050944692245)     recipes[r["id"]] = self._import(r["object"])
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692246&selected=1523892050944692246) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692246&selected=1523892050944692246)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_runner/feedstock.py", line 43, in _import
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692247&selected=1523892050944692247) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692247&selected=1523892050944692247)     exec(compile(source=f.read(), filename=filename, mode="exec"), ns)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692248&selected=1523892050944692248) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692248&selected=1523892050944692248)   File "/tmp/tmp7c2oaxz7/feedstock/recipe.py", line 151, in <module>
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692249&selected=1523892050944692249) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692249&selected=1523892050944692249)   File "<string>", line 22, in __init__
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692250&selected=1523892050944692250) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692250&selected=1523892050944692250)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 798, in __post_init__
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692251&selected=1523892050944692251) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692251&selected=1523892050944692251)     super().__post_init__()
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692252&selected=1523892050944692252) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692252&selected=1523892050944692252)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/recipes/base.py", line 22, in __post_init__
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692253&selected=1523892050944692253) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692253&selected=1523892050944692253)     self.sha256 = dataclass_sha256(self, ignore_keys=self._hash_exclude_)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692254&selected=1523892050944692254) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692254&selected=1523892050944692254)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 75, in dataclass_sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692255&selected=1523892050944692255) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692255&selected=1523892050944692255)     return dict_to_sha256(d)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692256&selected=1523892050944692256) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692256&selected=1523892050944692256)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 36, in dict_to_sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692257&selected=1523892050944692257) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692257&selected=1523892050944692257)     b = dumps(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692258&selected=1523892050944692258) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692258&selected=1523892050944692258)   File "/usr/lib/python3.9/json/__init__.py", line 234, in dumps
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692260&selected=1523892050944692260) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692260&selected=1523892050944692260)     return cls(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692261&selected=1523892050944692261) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692261&selected=1523892050944692261)   File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692262&selected=1523892050944692262) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692262&selected=1523892050944692262)     chunks = self.iterencode(o, _one_shot=True)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692263&selected=1523892050944692263) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692263&selected=1523892050944692263)   File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692264&selected=1523892050944692264) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692264&selected=1523892050944692264)     return _iterencode(o, 0)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692265&selected=1523892050944692265) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692265&selected=1523892050944692265)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 19, in either_encode_or_hash
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692266&selected=1523892050944692266) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692266&selected=1523892050944692266)     elif hasattr(obj, "sha256"):
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692267&selected=1523892050944692267) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692267&selected=1523892050944692267)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/patterns.py", line 242, in sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692268&selected=1523892050944692268) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692268&selected=1523892050944692268)     return pattern_blockchain(self).pop(-1)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692269&selected=1523892050944692269) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692269&selected=1523892050944692269)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/patterns.py", line 313, in pattern_blockchain
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692270&selected=1523892050944692270) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692270&selected=1523892050944692270)     root_sha256 = dict_to_sha256(root_drop_empty)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692271&selected=1523892050944692271) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692271&selected=1523892050944692271)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 36, in dict_to_sha256
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692272&selected=1523892050944692272) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692272&selected=1523892050944692272)     b = dumps(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692273&selected=1523892050944692273) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692273&selected=1523892050944692273)   File "/usr/lib/python3.9/json/__init__.py", line 234, in dumps
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692274&selected=1523892050944692274) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692274&selected=1523892050944692274)     return cls(
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692275&selected=1523892050944692275) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692275&selected=1523892050944692275)   File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692276&selected=1523892050944692276) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692276&selected=1523892050944692276)     chunks = self.iterencode(o, _one_shot=True)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692277&selected=1523892050944692277) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692277&selected=1523892050944692277)   File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692278&selected=1523892050944692278) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692278&selected=1523892050944692278)     return _iterencode(o, 0)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692279&selected=1523892050944692279) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692279&selected=1523892050944692279)   File "/usr/local/lib/python3.9/dist-packages/pangeo_forge_recipes/serialization.py", line 25, in either_encode_or_hash
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692280&selected=1523892050944692280) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692280&selected=1523892050944692280)     raise TypeError(f"object of type {type(obj).__name__} not serializable")
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692281&selected=1523892050944692281) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692281&selected=1523892050944692281) TypeError: object of type ClientTimeout not serializable
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050944692282&selected=1523892050944692282) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050944692282&selected=1523892050944692282) [2022-11-01 00:09:59 +0000] [66] [ERROR] Exception in ASGI application
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886528&selected=1523892050948886528) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886528&selected=1523892050948886528) Traceback (most recent call last):
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886529&selected=1523892050948886529) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886529&selected=1523892050948886529)   File "/opt/app/pangeo_forge_orchestrator/routers/github_app.py", line 619, in run
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886530&selected=1523892050948886530) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886530&selected=1523892050948886530)     out = subprocess.check_output(cmd)
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886531&selected=1523892050948886531) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886531&selected=1523892050948886531)   File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886532&selected=1523892050948886532) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886532&selected=1523892050948886532)     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886533&selected=1523892050948886533) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886533&selected=1523892050948886533)   File "/usr/lib/python3.9/subprocess.py", line 528, in run
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886534&selected=1523892050948886534) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886534&selected=1523892050948886534)     raise CalledProcessError(retcode, process.args,
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886535&selected=1523892050948886535) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886535&selected=1523892050948886535) subprocess.CalledProcessError: Command '['pangeo-forge-runner', 'bake', '--repo=https://github.com/pangeo-forge/eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock', '--ref=1fd4c64a07d3e6f5caf247ca60784774c45e8740', '--json', '--prune', '--Bake.recipe_id=eooffshore_ics_ccmp_v02_1_nrt_wind', '-f=/tmp/tmpz2sirbhz.json']' returned non-zero exit status 1.
Oct 31 17:10:00 [pangeo-forge-api-prod](https://my.papertrailapp.com/systems/pangeo-forge-api-prod/events?focus=1523892050948886537&selected=1523892050948886537) [app/web.1](https://my.papertrailapp.com/events?q=program%3Aapp%2Fweb.1&focus=1523892050948886537&selected=1523892050948886537) During handling of the above exception, another exception occurred:

let me know if any hints/workarounds come to mind

Copy link

@derekocallaghan derekocallaghan Nov 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @andersy005, this looks like a similar problem to the recent process_input and process_chunk serialization issue.

Two possible workarounds:

  1. Although the aiohttp.ClientSession read_timeout and conn_timeout parameters are deprecated in favour of timeout used in this PR commit, maybe they can be used as a short-term workaround as they accept floats (rather than aiohttp.ClientTimeout used for timeout), and should be serializable.
  2. FilePattern has its own sha256() implementation, and although fsspec_open_kwargs is currently included, perhaps this could be excluded (similar to the pattern format_function and combine dims currently excluded), or particular kwargs (e.g. any aiohttp instances) could be excluded from fsspec_open_kwargs prior to serialization. I guess a timeout value is something that isn't necessary for a recipe hash?

https://github.com/pangeo-forge/pangeo-forge-recipes/blob/3441d94a290d296b8f62638df15bb60993e86b1d/pangeo_forge_recipes/patterns.py#L299

    # we exclude the format function and combine dims from ``root`` because they determine the
    # index:filepath pairs yielded by iterating over ``.items()``. if these pairs are generated in
    # a different way in the future, we ultimately don't care.
    root = {
        "fsspec_open_kwargs": pattern.fsspec_open_kwargs,
        "query_string_secrets": pattern.query_string_secrets,
        "file_type": pattern.file_type,
        "nitems_per_file": {
            op.name: op.nitems_per_file  # type: ignore
            for op in pattern.combine_dims
            if op.name in pattern.concat_dims
        },
    }

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @andersy005, it looks like @yuvipanda is having a similar aiohttp hash problem in the daymet staged recipe as he's using aiohttp.BasicAuth in fsspec_open_kwargs["client_kwargs"]. More details here.

@derekocallaghan derekocallaghan mentioned this pull request Nov 8, 2022
Draft
This was referenced Nov 16, 2022
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@derekocallaghan derekocallaghan derekocallaghan left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@andersy005 @derekocallaghan