Support Python 3.12 #186

johnkerl · 2024-02-13T22:31:06Z

Note: I ran python-spec/update-requirements-txt

bkmartinjr · 2024-02-13T22:36:13Z

Question: if you ran `update-requirements-txt, with the mods to include python 3.11 and 3.12, shouldn't there be two additional files included in this PR? (requirements-py3.11.txt and requirements-py3.12.txt)?

bkmartinjr

Also, is it worth updating the python-somacore.yaml to include python 3.11 and 3.12?

johnkerl · 2024-02-13T22:38:38Z

Question: if you ran `update-requirements-txt, with the mods to include python 3.11 and 3.12, shouldn't there be two additional files included in this PR? (requirements-py3.11.txt and requirements-py3.12.txt)?

Yes and I watched myself typing git add of them ... 🤔

johnkerl · 2024-02-13T22:38:54Z

Also, is it worth updating the python-somacore.yaml to include python 3.11 and 3.12?

Thanks for seeing what I did not!

thetorpedodog · 2024-02-13T22:42:54Z

This is way bigger of a change than what I would expect. Was this run on a Linux machine or Mac OS? Running it right now on Linux I get the diff in change 7e3adfe.

johnkerl · 2024-02-13T22:44:04Z

@thetorpedodog I ran it on an Ubuntu EC2 instance.

I'm happy to commit whatever. Just trying to do what I thought was right.

johnkerl · 2024-02-13T22:50:10Z

@thetorpedodog 4551a83

thetorpedodog · 2024-02-13T22:51:39Z

I'm happy to commit whatever. Just trying to do what I thought was right.

I am puzzled. I’m sure you did everything right, and everything else looks normal, but the reason that Python 3.10 (and only Python 3.10???) has (a) so many additional packages and (b) a bunch of downgrades is beyond me. The other thing I can think of: is this an AMD64 machine, or AArch64? (But even then, why would it be so different??????)

johnkerl · 2024-02-13T22:53:12Z

@thetorpedodog I don't know the right thing to do to merge this PR.

johnkerl · 2024-02-13T22:54:31Z

@thetorpedodog I applied your diff in @thetorpedodog 4551a83

Does that unblock us?

thetorpedodog

What you’ve got now is good. I remain bewildered by the difference. (Architecture? OS version?)

For comparison, I have:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm
$ uname -a
Linux tiledebian 6.4.0-0.deb12.2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.4.4-3~bpo12+1 (2023-08-08) x86_64 GNU/Linux

johnkerl · 2024-02-13T22:57:24Z

I am puzzled. I’m sure you did everything right, and everything else looks normal, but the reason that Python 3.10 (and only Python 3.10???) has (a) so many additional packages and (b) a bunch of downgrades is beyond me. The other thing I can think of: is this an AMD64 machine, or AArch64? (But even then, why would it be so different??????)

@thetorpedodog here are the specs:

ubuntu@wihti[prod][][~]$ python --version
Python 3.10.12

ubuntu@wihti[prod][][~]$ uname -a
Linux ip-172-31-85-136 6.2.0-1018-aws #18~22.04.1-Ubuntu SMP Wed Jan 10 22:54:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@wihti[prod][][~]$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

That machine has a 3.10 already, and tiledbsoma already, before my running your script.

Should I add a comment that python-spec/update-requirements-txt should be run from a clean Docker image perhaps?

thetorpedodog · 2024-02-13T22:58:49Z

That machine has a 3.10 already, and tiledbsoma already, before my running your script.

Should I add a comment that python-spec/update-requirements-txt should be run from a clean Docker image perhaps?

It shouldn’t need to, at least as far as I am aware, because I thought Conda provided enough isolation. I’ll see what happens when running in a couple different Dockers.

johnkerl · 2024-02-13T23:00:57Z

Thanks @thetorpedodog but I do know your time is very valuable. Debugging this is not high-pri for me. Just to be very clear what I'm not asking of your time.

thetorpedodog · 2024-02-13T23:47:37Z

I made this little script and put it at playground/run-in-container.sh:

#!/bin/sh

apt-get update
apt-get -y install python-is-python3 python3-pip git wget
apt-get -y upgrade
cd /tmp
wget -O miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x ./miniconda.sh
./miniconda.sh -b
eval "$(/root/miniconda3/bin/conda shell.bash hook 2>/dev/null)"
git clone -b original-py3.12-branch https://github.com/single-cell-data/SOMA.git
cd SOMA
./python-spec/update-requirements-txt
echo "####################################################################"
echo
echo
git --no-pager diff
echo
echo
echo "####################################################################"

original-py3.12-branch refers to commit 67e9c40

and ran it like:

$ export DISTRO=ubuntu:22.04
$ docker run -it --rm --mount type=bind,source="$(pwd)/playground/run-in-container.sh",target=/opt/do-stuff,readonly "$DISTRO" /opt/do-stuff

and the diff it output looked like this, on both ubuntu:22.04 and debian:12 (there may have been some slight differences in what versions were installed in the end but the list of + and − packages was the same):

diff --git a/python-spec/requirements-py3.10.txt b/python-spec/requirements-py3.10.txt
index 52fb7e9..ec1f979 100644
--- a/python-spec/requirements-py3.10.txt
+++ b/python-spec/requirements-py3.10.txt
@@ -1,37 +1,19 @@
-anndata==0.9.1
-attrs==23.1.0
-cloudpickle==2.2.1
-contourpy==1.1.0
-cycler==0.11.0
-fonttools==4.40.0
-h5py==3.9.0
-joblib==1.2.0
-kiwisolver==1.4.4
-llvmlite==0.39.1
-matplotlib==3.7.1
+anndata==0.10.5.post1
+array_api_compat==1.4.1
+attrs==23.2.0
+exceptiongroup==1.2.0
+h5py==3.10.0
+llvmlite==0.42.0
 natsort==8.4.0
-networkx==3.1
-numba==0.56.4
-numpy==1.23.5
-packaging==23.1
-pandas==1.5.3
-patsy==0.5.3
-Pillow==9.5.0
-pyarrow==12.0.1
+numba==0.59.0
+numpy==1.26.4
+packaging==23.2
+pandas==2.2.0
+pyarrow==15.0.0
 pyarrow-hotfix==0.6
-pynndescent==0.5.10
 python-dateutil==2.8.2
 pytz==2024.1
-scanpy==1.9.3
-scikit-learn==1.2.2
-scipy==1.10.1
-seaborn==0.12.2
-session-info==1.0.0
+scipy==1.12.0
 six==1.16.0
-statsmodels==0.14.0
-stdlib-list==0.9.0
-tblib==1.7.0
-threadpoolctl==3.1.0
-tqdm==4.65.0
-typing_extensions==4.6.3
-umap-learn==0.5.3
+typing_extensions==4.9.0
+tzdata==2024.1
diff --git a/python-spec/requirements-py3.12.txt b/python-spec/requirements-py3.12.txt
index b23699f..2bb7667 100644
--- a/python-spec/requirements-py3.12.txt
+++ b/python-spec/requirements-py3.12.txt
@@ -13,8 +13,8 @@ pyarrow-hotfix==0.6
 python-dateutil==2.8.2
 pytz==2024.1
 scipy==1.12.0
-setuptools==69.0.3
+setuptools==68.2.2
 six==1.16.0
 typing_extensions==4.9.0
 tzdata==2024.1
-wheel==0.42.0
+wheel==0.41.2
diff --git a/python-spec/requirements-py3.7.txt b/python-spec/requirements-py3.7.txt
index fcdd4b3..7980211 100644
--- a/python-spec/requirements-py3.7.txt
+++ b/python-spec/requirements-py3.7.txt
@@ -1,5 +1,6 @@
 anndata==0.8.0
 attrs==23.2.0
+certifi @ file:///croot/certifi_1671487769961/work/certifi
 h5py==3.8.0
 importlib-metadata==6.7.0
 llvmlite==0.39.1

…which is to say that, even within an empty container, I couldn’t get the same package combinations you were getting and the end result is closer to what I had gotten by running it on my own regular machine. Does the result end up looking any different if you run it in a container on yours?

johnkerl · 2024-02-14T04:08:38Z

@thetorpedodog full log: https://gist.github.com/johnkerl/2dd8f2a19db99c601865e42e242d29b9

Just the diff part: https://gist.github.com/johnkerl/39c1829377265bf6dc41f691167bea24

Run python-spec/update-requirements-txt

ac95a6a

johnkerl requested review from thetorpedodog and bkmartinjr February 13, 2024 22:31

bkmartinjr reviewed Feb 13, 2024

View reviewed changes

John Kerl added 2 commits February 13, 2024 22:39

more

4625598

more

67e9c40

Use files from pfish/python-3.12 branch

4551a83

johnkerl requested a review from bkmartinjr February 13, 2024 22:51

thetorpedodog approved these changes Feb 13, 2024

View reviewed changes

johnkerl merged commit 76b2d2b into main Feb 13, 2024
8 checks passed

johnkerl deleted the kerl/python-3.12 branch February 13, 2024 22:58

ryan-williams mentioned this pull request Mar 12, 2024

[python/ci] Test matrix should include Python 3.11 #176

Closed

ryan-williams mentioned this pull request Oct 21, 2024

Work around various issues with Python 3.12 #222

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Python 3.12 #186

Support Python 3.12 #186

johnkerl commented Feb 13, 2024 •

edited

Loading

bkmartinjr commented Feb 13, 2024

bkmartinjr left a comment

johnkerl commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog left a comment

johnkerl commented Feb 13, 2024 •

edited

Loading

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 14, 2024 •

edited

Loading

Support Python 3.12 #186

Support Python 3.12 #186

Conversation

johnkerl commented Feb 13, 2024 • edited Loading

bkmartinjr commented Feb 13, 2024

bkmartinjr left a comment

Choose a reason for hiding this comment

johnkerl commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog left a comment

Choose a reason for hiding this comment

johnkerl commented Feb 13, 2024 • edited Loading

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 13, 2024

thetorpedodog commented Feb 13, 2024

johnkerl commented Feb 14, 2024 • edited Loading

johnkerl commented Feb 13, 2024 •

edited

Loading

johnkerl commented Feb 13, 2024 •

edited

Loading

johnkerl commented Feb 14, 2024 •

edited

Loading