Avoid serialization blues by computing + caching the hash.

[alxmrs]

• Fixes Generating a recipe's hash fails when using process_chunk and process_input functions #427 by computing the hash at recipe construction time instead of pipeline runtime. This avoids the need for serializing functions.
• Supersedes Use bytecode for hashes in *some* cases #428

[andersy005]

@alxmrs, should we go ahead and merge this?

[alxmrs]

Sure thing! I'll do the honors :)

[cisaacstern]

Thanks @alxmrs! What an elegant solution to this problem.

[derekocallaghan]

Hi @cisaacstern and @alxmrs, fyi recipes that use aiohttp objects in file_pattern.fsspec_open_kwargs["client_kwargs"] are currently failing as the hash can't be computed:

• eooffshore_ics_ccmp_v02_1_nrt_wind-feedstock: uses aiohttp.ClientTimeout, details and suggested workarounds here.
• daymet staged recipe: uses aiohttp.BasicAuth, details here.

FilePattern has its own sha256() implementation, and although fsspec_open_kwargs (and client_kwargs) is currently included, perhaps this could be excluded (similar to the pattern format_function and combine dims currently excluded), or particular kwargs (e.g. any aiohttp instances) could be excluded from fsspec_open_kwargs prior to serialization. I guess timeouts/credentials values are something that aren't necessary for a recipe hash?

pangeo-forge-recipes/pangeo_forge_recipes/patterns.py

Line 299 in 3441d94

root = {

    # we exclude the format function and combine dims from ``root`` because they determine the
    # index:filepath pairs yielded by iterating over ``.items()``. if these pairs are generated in
    # a different way in the future, we ultimately don't care.
    root = {
        "fsspec_open_kwargs": pattern.fsspec_open_kwargs,
        "query_string_secrets": pattern.query_string_secrets,
        "file_type": pattern.file_type,
        "nitems_per_file": {
            op.name: op.nitems_per_file  # type: ignore
            for op in pattern.combine_dims
            if op.name in pattern.concat_dims
        },
    }

[cisaacstern]

Thanks for flagging this, @derekocallaghan. And also for becoming so engaged these last few months! The project is benefiting so much from your contributions.

This issue of not being able to serialize aiohttp objects is important, and we should fix it. Currently it seems like your deep familiarity with this issue appears to be spread across a few different threads (here, and in a couple of staged-recipes discussions you link to above).

Could I ask that you summarize this problem in a new, standalone issue here on pangeo-forge-recipes? Something brief that links to your prior reports will suffice, and if you could also provide the shortest possible MRE that shows how the relevant aiohttp objects throw errors when passed to

pangeo-forge-recipes/pangeo_forge_recipes/serialization.py

Lines 10 to 25 in fe5d1f7

	def either_encode_or_hash(obj: Any):
	"""For objects which are not serializable with ``json.dumps``, this function defines
	type-specific handlers which extract either a serializable value or a hash from the object.
	:param obj: Any object which is not serializable to ``json``.
	"""
	if isinstance(obj, Enum): # custom serializer for FileType, CombineOp, etc.
	return obj.value
	elif hasattr(obj, "sha256"):
	return obj.sha256.hex()
	elif inspect.isfunction(obj):
	return inspect.getsource(obj)
	elif isinstance(obj, bytes):
	return obj.hex()
	raise TypeError(f"object of type {type(obj).__name__} not serializable")

that'd be great.

My guess is that we can resolve this by simply adding an additional conditional block(s) to that function. Open to all other suggestions, though. However we fix it, having a dedicated issue will be helpful to frame the discussion. Thanks so much!

[derekocallaghan]

Thanks @cisaacstern, I'll create a new standalone issue for the aiohttp serialization.

[cisaacstern]

Thanks @derekocallaghan!