[python] Python 3.12 support #3001

johnkerl · 2024-09-16T18:29:36Z

For issue #1849 -- which has a long history.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.65%. Comparing base (928c281) to head (b7db9ed).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3001      +/-   ##
==========================================
+ Coverage   83.54%   83.65%   +0.11%     
==========================================
  Files          51       51              
  Lines        5434     5434              
==========================================
+ Hits         4540     4546       +6     
+ Misses        894      888       -6

Flag	Coverage Δ
python	`83.65% <ø> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	`83.65% <ø> (+0.11%)`	⬆️
libtiledbsoma	`∅ <ø> (∅)`

thetorpedodog

right now leaving comments as I go through each commit. apologies if this is already resolved in a later one

thetorpedodog · 2024-09-17T13:49:32Z

.github/workflows/python-ci-single.yml

+ git clone [email protected]:single-cell-data/SOMA.git soma
+ cd ./SOMA
+ git checkout kerl/python-3.12
+ pip install ./
+ python -c 'import somacore; print(somacore.__version__)'


pip install git+https://github.com/single-cell-data/SOMA.git@kerl/python-3.12 might be able to replace this with a one-liner that leaves your venv in a better state?

@thetorpedodog thanks! This stanza was my copy-paste of another time someone wanted to run tiledbsoma CI against a non-merged somacore mod ... I'll save off your advice in my cheat-sheet for next time -- thank you!

thetorpedodog · 2024-09-17T14:00:19Z

From the build log:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.10/site-packages/tiledbsoma/__init__.py", line 103, in <module>
    from ._collection import Collection
  File "/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.10/site-packages/tiledbsoma/_collection.py", line 3[7](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10895777089/job/30234556256#step:18:8), in <module>
    from ._dataframe import DataFrame
  File "/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.10/site-packages/tiledbsoma/_dataframe.py", line 17, in <module>
    from tiledbsoma import _new_shape_feature_flag_enabled
ImportError: cannot import name '_new_shape_feature_flag_enabled' from partially initialized module 'tiledbsoma' (most likely due to a circular import) (/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.[10](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10895777089/job/30234556256#step:18:11)/site-packages/tiledbsoma/__init__.py)

This is because at the point this line is executed, the symbol _new_shape_feature_flag_enabled is not yet available. The line

TileDB-SOMA/apis/python/src/tiledbsoma/__init__.py

Line 138 in a96c72d

_new_shape_feature_flag = os.getenv("SOMA_PY_NEW_SHAPE") is not None

needs to be moved above all the imports. (Alternately, it might make sense to create a tiledbsoma._flags module so that there’s no circular dependency between submodules and the tiledbsoma/__init__.py file.

johnkerl · 2024-09-17T14:02:26Z

needs to be moved above all the imports

@thetorpedodog thank you!!! I got some 'new style lint' here -- I don't know why, I didn't change any linter versions -- and was forced to move imports :(

There's something weird to debug here -- I'll see if I can make everything in the CI toolchain happy simultaneously -- thank you!!!

Also I'll try the _flags module idea -- this sounds promising -- thank you! :)

johnkerl · 2024-09-17T15:08:55Z

@thetorpedodog

And re

There's something weird to debug here -- I'll see if I can make everything in the CI toolchain happy simultaneously -- thank you!!!

I have traced that to here:

https://github.com/single-cell-data/TileDB-SOMA/pull/3004/files#r1763424424

iosonofabio · 2024-09-23T06:46:26Z

Hi guys, thanks for your amazing package.

I understand this is in flux as we write - as a power user I would kindly like to ask if there's anything I can do to help push this through - I really need it and workarounds are a decent pain.

(As I understand, now there is a buggy typechecker stalling CI stalling 3.12 adoption, which itself was released 12 months ago. If inaccurate feel free to correct)

johnkerl · 2024-09-23T13:06:24Z

@iosonofabio thank you! Your assessment is correct -- current status is here: #1849 (comment) and @ryan-williams will be moving this forward

ryan-williams · 2024-10-21T14:53:42Z

I can't add you as a reviewer since you opened this PR, @johnkerl, but I would like your review on the latest changes that I've pushed here 🙏

Depending on typeguard@HEAD may trip up SOMA developers who have existing typeguard 4.2.1 or 4.3.0 installations, but I'm not sure what the other option is. As soon as agronholm/typeguard#490 makes it into a release, we can move to that. Could also try to only depend on typeguard@HEAD in Python 3.12, but I thought that was more complex than it was worth.

johnkerl

LGTM!! :)

johnkerl · 2024-10-21T14:56:21Z

apis/python/tests/test_dataframe.py

@@ -783,7 +783,9 @@ def make_multiply_indexed_dataframe(
 "domain": [[-1000, 1000]],
 "coords": [{"bogus": True}],
 "A": None,
- "throws": TypeError,
+ # Disable Typeguard while asserting this error, otherwise a typeguard.TypeCheckError is


Let's hvae a GitHub issue tracking the 'real' typeguard release, and let's put its URL here as a to-do comment

This change isn't due to using Typeguard@HEAD; it will remain relevant when we move back to a proper Typeguard release.

The reason for it is: in 3.12, better type-checking was causing a TypeCheckError to occur before the TypeError we were verifying here.

I worked around it by disabling type-checking while triggering this error, so we can continue to verify what we were previously.

johnkerl · 2024-10-21T14:56:39Z

apis/python/tests/test_dataframe.py

+ throws = io.get("throws", None)
+ if throws:
+ if isinstance(throws, tuple) and not throws[1]:
+ # Disable Typeguard, verify actual runtime error type (avoid


Ditto the above to-do URL/comment

Same as above, todo related to upgrading Typeguard is not applicable here.

johnkerl · 2024-10-21T14:57:29Z

Depending on typeguard@HEAD may trip up SOMA developers who have existing typeguard 4.2.1 or 4.3.0 installations, but I'm not sure what the other option is

Agreed. And it only affects folks on 3.12, so any dev who really wants to avoid this can use 3.11.

ryan-williams · 2024-10-21T18:30:28Z

apis/python/tests/test_dataframe.py

@@ -783,7 +783,9 @@ def make_multiply_indexed_dataframe(
 "domain": [[-1000, 1000]],
 "coords": [{"bogus": True}],
 "A": None,
- "throws": TypeError,
+ # Disable Typeguard while asserting this error, otherwise a typeguard.TypeCheckError is


This change isn't due to using Typeguard@HEAD; it will remain relevant when we move back to a proper Typeguard release.

The reason for it is: in 3.12, better type-checking was causing a TypeCheckError to occur before the TypeError we were verifying here.

I worked around it by disabling type-checking while triggering this error, so we can continue to verify what we were previously.

ryan-williams · 2024-10-21T18:31:25Z

apis/python/tests/test_dataframe.py

+ throws = io.get("throws", None)
+ if throws:
+ if isinstance(throws, tuple) and not throws[1]:
+ # Disable Typeguard, verify actual runtime error type (avoid


Same as above, todo related to upgrading Typeguard is not applicable here.

ryan-williams · 2024-10-21T18:32:31Z

apis/python/requirements_dev.txt

-typeguard==4.2.1
+# Python 3.12 support requires https://github.com/agronholm/typeguard/pull/490.
+# Use Typeguard @ HEAD until that PR is included in a release.
+typeguard @ git+https://github.com/agronholm/typeguard


This would be the place for a TODO / issue link related to upgrading Typeguard. Do you want me to change this comment?

Filed #3216 and referenced it here

for agronholm/typeguard#490, for Python 3.12 tests

johnkerl force-pushed the kerl/python-3.12-support branch from dc19af6 to f1a3909 Compare September 16, 2024 19:10

johnkerl force-pushed the kerl/python-3.12-support branch from 9616847 to f1a3909 Compare September 16, 2024 21:36

This was referenced Sep 16, 2024

Work around various issues with Python 3.12 single-cell-data/SOMA#222

Closed

[python] Try 3.12 again [WIP] #2692

Closed

johnkerl force-pushed the kerl/python-3.12-support branch from beed527 to a96c72d Compare September 17, 2024 03:02

johnkerl mentioned this pull request Sep 17, 2024

[Feature request] Python 3.12 support #1849

Open

thetorpedodog reviewed Sep 17, 2024

View reviewed changes

johnkerl force-pushed the kerl/python-3.12-support branch from 8ff4ec3 to f3674a9 Compare September 17, 2024 14:19

This was referenced Sep 17, 2024

TileDB-SOMA 1.14 #2999

Closed

[python] Move temporary feature flag into a module #3005

Merged

[python] Remove pyarrow pin on MacOS #3004

Merged

johnkerl force-pushed the kerl/python-3.12-support branch 2 times, most recently from 9790ab0 to db17d38 Compare September 17, 2024 20:03

ryan-williams mentioned this pull request Sep 23, 2024

[python] Add --maxfail=50 to GHA pytest invocations #3034

Merged

johnkerl force-pushed the kerl/python-3.12-support branch from db17d38 to 6fdbd75 Compare October 16, 2024 22:12

johnkerl mentioned this pull request Oct 16, 2024

[python/ci] Extend Python 3.12 CI coverage #3184

Closed

ryan-williams force-pushed the kerl/python-3.12-support branch 2 times, most recently from 63b6d5e to 8c5f857 Compare October 18, 2024 19:36

ryan-williams marked this pull request as ready for review October 18, 2024 20:48

johnkerl requested a review from ryan-williams October 21, 2024 14:55

johnkerl commented Oct 21, 2024

View reviewed changes

ryan-williams approved these changes Oct 21, 2024

View reviewed changes

ryan-williams mentioned this pull request Oct 21, 2024

[python] Upgrade Typeguard to >4.3.0 (once it is available) #3216

Open

johnkerl and others added 4 commits October 21, 2024 14:39

[python] Python 3.12 support

3f5af11

python-ci-packaging: consistent dep order

ad613ae

use Typeguard@HEAD

8b45396

for agronholm/typeguard#490, for Python 3.12 tests

rm soma@kerl/python-3.12 CI pin

b7db9ed

ryan-williams force-pushed the kerl/python-3.12-support branch from 8c5f857 to b7db9ed Compare October 21, 2024 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python] Python 3.12 support #3001

[python] Python 3.12 support #3001

johnkerl commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

thetorpedodog left a comment

thetorpedodog Sep 17, 2024

johnkerl Sep 17, 2024

thetorpedodog commented Sep 17, 2024

johnkerl commented Sep 17, 2024 •

edited

Loading

johnkerl commented Sep 17, 2024

iosonofabio commented Sep 23, 2024

johnkerl commented Sep 23, 2024

ryan-williams commented Oct 21, 2024

johnkerl left a comment

johnkerl Oct 21, 2024

ryan-williams Oct 21, 2024

johnkerl Oct 21, 2024

ryan-williams Oct 21, 2024

johnkerl commented Oct 21, 2024

ryan-williams Oct 21, 2024

ryan-williams Oct 21, 2024

ryan-williams Oct 21, 2024

ryan-williams Oct 21, 2024

[python] Python 3.12 support #3001

Are you sure you want to change the base?

[python] Python 3.12 support #3001

Conversation

johnkerl commented Sep 16, 2024 • edited Loading

codecov bot commented Sep 16, 2024 • edited Loading

Codecov Report

thetorpedodog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thetorpedodog commented Sep 17, 2024

johnkerl commented Sep 17, 2024 • edited Loading

johnkerl commented Sep 17, 2024

iosonofabio commented Sep 23, 2024

johnkerl commented Sep 23, 2024

ryan-williams commented Oct 21, 2024

johnkerl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnkerl commented Oct 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnkerl commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

johnkerl commented Sep 17, 2024 •

edited

Loading