[Feature request] Python 3.12 support

[bkmartinjr]

Add support for Python 3.12 (release Oct 2023).

[bkmartinjr]

while we are at it, it would be good to remove 3.8 support. Lots of our upstream packages have already done so (e.g. numba, pandas, etc)

[johnkerl]

Status as of 2024-02-05: numba 0.58 is not yet ready for Python 3.12.

$ docker run --rm -it --entrypoint=/bin/bash python:3.12-bookworm
root@20d715ac5ab7:/# pip install numba==0.58
Collecting numba==0.58
  Downloading numba-0.58.0.tar.gz (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 32.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-417bk_qg/numba_1954fc6d020f4c6087042050c24f72c3/setup.py", line 51, in <module>
          _guard_py_ver()
        File "/tmp/pip-install-417bk_qg/numba_1954fc6d020f4c6087042050c24f72c3/setup.py", line 48, in _guard_py_ver
          raise RuntimeError(msg.format(cur_py, min_py, max_py))
      RuntimeError: Cannot install on Python version 3.12.1; only versions >=3.8,<3.12 are supported.
      [end of output]

$ docker run --rm -it --entrypoint=/bin/bash python:3.11-bookworm
root@d8f7153f52c6:/# pip install numba==0.58
Collecting numba==0.58
  Obtaining dependency information for numba==0.58 from https://files.pythonhosted.org/packages/f7/b6/8800604a3380aa94af830c30839db93077c7fe2812a48ce7d4400d75ba9c/numba-0.58.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
...
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.41.1 numba-0.58.0 numpy-1.25.2

[bkmartinjr]

Numba 0.59, with Python 3.12 support has shipped. Pandas, Scipy and others also have support for it. Worth getting on a sprint plan IMHO

[johnkerl]

@bkmartinjr let me re-try now what failed for me earlier as shown above ...

[johnkerl]

@bkmartinjr this succeeds for me! :)

ubuntu@anzo[prod][][~]$ docker run --rm -it --entrypoint=/bin/bash python:3.12-bookworm
root@62dc2a187e2d:/# pip install numba==0.59
...

I'll re-try my #2108 probe ...

[johnkerl]

Work items detailed on #2140; more info coming here momentarily

[johnkerl]

Problem:

• I put up PR [python] Support Python 3.12 #2108 and it was green so I merged it
• Now PRs [python] Outgest additional X layers to AnnData #2119 [python] Remove a temp comment, and complete Python 3.12 checks #2136 and anything else new is failing MacPS Python 3.12 CI fails
  • Unable to install wheel for pyarrow
  • https://github.com/single-cell-data/TileDB-SOMA/actions/runs/7896650237/job/21550972931?pr=2138 for example
• Another issue: TileDB-Py doesn't have wheels for 3.12
  • https://pypi.org/project/tiledb/#files

To do:

• Leave 3.12 support in place -- don't entirely revert [Feature request] Python 3.12 support #1849
• Don't fail CI for 3.12 until the above is resolved -- [python/ci] Leave 3.12 support provisional #2140
• Leave this issue open for tracking

See also: #1938

It seems that maybe:

• We get a build-from-source for pyarrow < 13 on MacOS, which fails as above
• pyarrow >= 13 on MacOS led to [python] Work around bug with aws-sdk-cpp and pyarrow on MacOS #1938 -- we should re-try [python] Work around bug with aws-sdk-cpp and pyarrow on MacOS #1938 but for Python 3.12 ...

[johnkerl]

while we are at it, it would be good to remove 3.8 support. Lots of our upstream packages have already done so (e.g. numba, pandas, etc)

@bkmartinjr I will file a separate tracking task (#2141) -- bringing up the leading edge and deprecating the trailing edge (for all stakeholders involved) are two very different problems

[johnkerl]

Please also see TileDB-Inc/TileDB-Py#1865

[sc-37726]

[bkmartinjr]

@johnkerl - FYI, I believe tiledb-py is now supported on Python 3.12 (in reference to TileDB-Inc/TileDB-Py#1865).

[johnkerl]

@bkmartinjr thanks! On the discussion list for the next sprint ✌️

[johnkerl]

Pending TileDB-Inc/TileDB-Py#1865

[johnkerl]

Huge progress: TileDB-Inc/TileDB#5223 is near completion in core. This should go into a core 2.26, which we'd include in a tiledbsoma 1.14.

[johnkerl]

Current unsolved challenges:

• Latest Python 3.12 is 3.12.5
  • As debug I tried Python 3.12.3 which worked as debug, but isn't a good option for shipping
• TileDB-SOMA's current typeguard version is 4.2.0
• Python 3.12.4 added a new required argument to evaluate_forward_ref within Python's built-in typing module which broke many packages including typeguard
• As a result, our TileDB-SOMA CI for PR 3001 fails like this
  • TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard'
• typeguard 4.3.0 contains a fix for this problem
  • https://github.com/agronholm/typeguard/releases
• But this surfaces another problem which is:

E       typeguard.TypeCheckError: item 0 of argument "coords" (tuple) did not match any element in the union:
E         NoneType: is not an instance of NoneType
E         int: is not an instance of int
E         somacore.types.Slice[int]: is not compatible with the Slice protocol because it has no attribute named '__orig_bases__'

/home/ubuntu/.pyenv/versions/3.12.5/lib/python3.12/site-packages/typeguard/_checkers.py:433: TypeCheckError

single-cell-data/SOMA#222

Once that works, we'll need to:

• Merge this PR
• Tag a release of this repo (somacore)
• Depend on 1.0.next of somacore within the TileDB-SOMA repo

I've currently got the following setup on an EC2 instance:

• somacore locally with pip install -e with this PR checked out
• tiledbsoma locally with pip install -e with [python] Python 3.12 support #3001 checked out

[johnkerl]

Current status:

• ✅ Build the AWS SDK with hidden symbols. TileDB-Inc/TileDB#5223 went into core 2.26 -- resolving the Arrow/MacOS issue
• ❌ We have another blocker
  • Python 3.12.4 and above has a new required argument in evaluate_forward_ref within Python's built-in typing module which broke many packages including typeguard
  • typeguard has this breakage as of 4.2.1 (which we currently use)
  • Moving up to typeguard 4.3.0 reveals another typeguard bug which @thetorpedodog has analyzed here: Protocol checking against typing_extenstion.Protocol fails agronholm/typeguard#479 (comment)
  • See also [python] Python 3.12 support #3001

We will attempt to work around these bugs in order to deliver Python 3.12 support in TileDB-SOMA 1.14.2 or 1.15.0, whichever comes first.