Unable to use run_deployment across versions

[ir3456]

Bug summary

There is an incompatibility issue when using prefect across major versions, specifically calling run_deployment using prefect 3.X to run a deployment that was deployed using prefect 2.X.

When doing this the following happens:
After calling run_deployment using prefect 3.X, a flow run is created of the deployment (2.X). The run_deployment call blocks as expected for the duration of the flow run, however, once the flow run finishes, regardless of its state, we encounter the following error.

Task run failed with exception: TypeError('No constructor defined') - Retries are exhausted
Traceback (most recent call last):
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/results.py", line 1156, in __new__
    subcls = lookup_type(cls, dispatch_key=kwargs["type"])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/utilities/dispatch.py", line 199, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 'unpersisted' in registry for type 'BaseResult'."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/task_engine.py", line 763, in run_context
    yield self
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/task_engine.py", line 1323, in run_task_sync
    engine.call_task_fn(txn)
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/task_engine.py", line 786, in call_task_fn
    result = call_with_parameters(self.task.fn, parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/utilities/callables.py", line 206, in call_with_parameters
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/dr/projects/prefect-sandbox/prefect_sandbox/tasks.py", line 72, in task_1
    run_prefect_deployment_check_successful(
  File "/Users/isaac/dr/projects/prefect-sandbox/prefect_sandbox/tasks.py", line 29, in run_prefect_deployment_check_successful
    flow_run_result = run_deployment(
                      ^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 399, in coroutine_wrapper
    return run_coro_as_sync(ctx_call())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 243, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 312, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 182, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/.pyenv/versions/3.11.7/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 383, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 225, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 389, in ctx_call
    result = await async_fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/client/utilities.py", line 103, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/deployments/flow_runs.py", line 172, in run_deployment
    flow_run = await client.read_flow_run(flow_run_id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/client/orchestration.py", line 2039, in read_flow_run
    return FlowRun.model_validate(response.json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/pydantic/main.py", line 596, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Library/Caches/pypoetry/virtualenvs/prefect-sandbox-qGp-k3h3-py3.11/lib/python3.11/site-packages/prefect/results.py", line 1158, in __new__
    raise ValidationError(errors=[exc], model=cls)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: No constructor defined

The following error has also been observed when working across versions. This is from an Oct 10 post in the ask-community prefect Slack channel. I initially had the same error below and diagnosed the issue in this thread.

File "C:\Web Apps\Prefect2.0-Flows\flows\Python\Snowflake\ff_gl_stats_sf_load.py", line 22, in get_completed_flow_runs_with_flow_name
    flow_runs = await client.read_flow_runs(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Web Apps\Prefect2.0-Flows\venv\Lib\site-packages\prefect\client\orchestration.py", line 2148, in read_flow_runs
    return pydantic.TypeAdapter(List[FlowRun]).validate_python(response.json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Web Apps\Prefect2.0-Flows\venv\Lib\site-packages\pydantic\type_adapter.py", line 135, in wrapped
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Web Apps\Prefect2.0-Flows\venv\Lib\site-packages\pydantic\type_adapter.py", line 366, in validate_python
    return self.validator.validate_python(object, strict=strict, from_attributes=from_attributes, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for list[FlowRun]
0.state.data.BaseResult
  Value error, Invalid type: unpersisted [type=value_error, input_value={'type': 'unpersisted'}, input_type=dict]

Version info (prefect version output)

3.0.4 and 3.0.8

Additional context

This issue was created in response to a thread in the ask-community prefect Slack channel posted on Oct 10.