Wrong kubernetes version in requirements?

[cagantomer]

Describe the issue:
Due the listed version of kubernetes dependency (v12.0.1, see here), in certain condition, pip install dask_kubernetes would install older versions of kubernetes (e.g. 17.17.0) which do not yet have V1PodDisruptionBudget class / attribute. This will result in the following error when trying to start a cluster:

Traceback (most recent call last):
   File "/usr/local/lib/python3.9/site-packages/distributed/deploy/spec.py", line 309, in _start
     self.scheduler = await self.scheduler
   File "/usr/local/lib/python3.9/site-packages/distributed/deploy/spec.py", line 64, in _
     await self.start()
   File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 218, in start
     self.pdb = await self._create_pdb()
   File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 269, in _create_pdb
     self.pdb_template = clean_pdb_template(make_pdb_from_dict(pdb_template_dict))
   File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/objects.py", line 241, in make_pdb_from_dict
     return SERIALIZATION_API_CLIENT.deserialize(dict_, client.V1PodDisruptionBudget)
 AttributeError: module 'kubernetes.client' has no attribute 'V1PodDisruptionBudget'

Minimal Complete Verifiable Example:

Use the following docker file to build an image (docker build -t dask-deps .)

FROM debian:buster-slim

# Install system dependencies
RUN apt-get -q update && \
    DEBIAN_FRONTEND=noninteractive apt-get -q install -y --no-install-recommends \
    libdw-dev curl git python3-pip \
    `#required for pyenv` \
    make build-essential libssl-dev zlib1g-dev \
    libbz2-dev libreadline-dev libsqlite3-dev \
    libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
    gcc g++ \
    && apt-get autoremove

# Install Python with shared libraries
RUN curl -sS https://pyenv.run | bash
RUN env PYTHON_CONFIGURE_OPTS="--enable-shared" ~/.pyenv/plugins/python-build/bin/python-build 3.9.5 /usr/local

Run the container (docker run --rm -it dask-deps bash),
and manually install kubectl and configure it (create relevant ./kube/config)

Then, install the dependencies:

pip install kubernetes==17.17.0 dask_kuberenetes==2022.10.1

The following code would generate the error (vanilla creation of KubeCluster)

from distributed import Client
from dask_kubernetes.classic import KubeCluster, make_pod_spec

image = "your image here"
pod_spec = make_pod_spec(
    image=image,
    memory_limit="4G",
    memory_request="4G",
    cpu_limit=1,
    cpu_request=1,
)

cluster = KubeCluster(
    pod_spec,
    namespace="research",
    n_workers=2,
)

This should throw an error:

RuntimeError: Cluster failed to start: module 'kubernetes.client' has no attribute 'V1PodDisruptionBudget'

The fix is trivial - need to update requirements for the repo/package to the first version where 'V1PodDisruptionBudget' was introduced - seems that was introduced in v21.7.0. (I can try to make a PR later)

Anything else we need to know?:
With plain pip install dask_kubernetes, a later version (24) of kubernetes is install and the bug won't happen. It seems that the project I work on somehow introduce and cause version 17 to be installed...

Environment:

• Dask version: ~=2022.10.0
• Dask kubernetes version: ~=2022.10.0
• Python version: 3.9.5
• Operating System: debian buster
• Install method (conda, pip, source): pip

[jacobtomlinson]

Thanks for raising such a detailed issue. If you have an interest in making the PR to bump the minimum up that would be awesome! Feel free to ping me directly for review.

[cagantomer]

What version should I change to - first that introduced V1PodDisruptionBudget? Latest available?
More importantly, how do I test this change doesn't break anything?

[jacobtomlinson]

I think the first that introduced the V1PodDisruptionBudget will be fine.

We currently don't test minimum versions (which is how this happened in the first place) and I don't expect we will start now. If problems like this start coming up more often then of course we can reassess that.

If the CI passes with your PR then I'm happy to merge it. If you want to test it more robustly locally I recommend you create a fresh environment and then downgrade the kubernetes package to the minimum allowed and ensure tests run locally.

[jacobtomlinson]

The classic KubeCluster was removed in #890. All users will need to migrate to the Dask Operator. Closing.