Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python #1138

Open
suzuki-shunsuke opened this issue Dec 6, 2021 · 13 comments
Open

Add python #1138

suzuki-shunsuke opened this issue Dec 6, 2021 · 13 comments
Labels
enhancement New feature or request python

Comments

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented Dec 6, 2021

#858
https://github.com/danhper/asdf-python
https://www.python.org/

@borissmidt
Copy link
Contributor

borissmidt commented Aug 13, 2024

@suzuki-shunsuke
There doesn't seem to be support for python yet in aqua,
Maybe it would be a good idea to support the following builds?

I suspect it isn't possible with the current version of aqua but maybe this type of releases could be used instead:
https://github.com/indygreg/python-build-standalone
https://github.com/indygreg/python-build-standalone/releases/tag/20240726

The issue with python seems to be that it has a lot of system dependencies and that python then has to be rebuild quite regularly.
So the tags are date based and the version is on of the release files.

These builds are used by rye, bazel and some other build tools to download a binary python so it's an unofficial standard.

@suzuki-shunsuke
Copy link
Member Author

Hi, thank you for your suggestion.
I didn't know that project.
I'll take a look.

@suzuki-shunsuke
Copy link
Member Author

https://github.com/indygreg/python-build-standalone/releases/tag/20240726

Hmm. Each release has assets for multiple Python versions (3.10.14, 3.11.9, etc).
aqua can't specify the Python version for now.

e.g.

- name: indygreg/python-build-standalone@20240726 # Can't specify python version

My idea is to enable aqua to specify parameters.

e.g.

- name: indygreg/python-build-standalone@20240726
  with:
    python_version: "3.10.14"

@boris-smidt-klarrio
Copy link
Contributor

It seems rye indexes the releases to know which version it can download from which release.
Each release has to currently maintained version and its latest patch available.

So when updating the release the 'python_version' should also be checked to see if it needs to be updated.

https://github.com/astral-sh/rye/blob/main/rye-devtools/src/rye_devtools/find_downloads.py

@suzuki-shunsuke
Copy link
Member Author

Related:

@suzuki-shunsuke
Copy link
Member Author

📝 After we support parameters, aquaproj/aqua#3052
to update parameters such as python_version by aqua update command, we need additional implementation.
I'm not sure if we will address it.
We would add the setting updating python_version to aqua-renovate-config.

@mab
Copy link

mab commented Aug 22, 2024

Maybe you should look into https://github.com/jdx/mise. To me mise is to specify Node, Phyton, Java versions - so languages coming with an own dependency management and Aqua for single binary tools. I see aqua as version manager, I don't see that I could also put configuration next to the version E.g. for python https://mise.jdx.dev/lang/python.html

@boris-smidt-klarrio
Copy link
Contributor

@mab I currently use mise and it is sometimes flaky to get new developers to pick it up. Because its based on ASDF plugins it is hard to add new tools. And some plugins are a bit brittle. Like checkov needed jq installed to work.

To me aqua seems simpler since it only downloads binaries and better on the front of security and managing the 'registry'.

Ideally mise would use the aqua repository by default and have several special install scripts baked in for python, java, ruby and others where it isn't trivial to install these tools.

@suzuki-shunsuke
Copy link
Member Author

WIP

export PIP_TARGET=$(aqua root-dir)/pip-global # location where pip installs tools
export PATH=$PIP_TARGET/bin:$PATH
export PYTHONPATH=$PIP_TARGET # Without this setting, tools installed by pip didn't work
pip install mkdocs
mkdocs -V

I found an issue that even if I change the Python version, seems like mkdocs -V refers to old Python.

  1. Install mkdocs with Python 3.11.9
  2. Change the Python version to 3.10.14
  3. Run mkdocs -V, then it outputs Python 3.11
$ mkdocs -V
mkdocs, version 1.6.0 from /Users/shunsukesuzuki/.local/share/aquaproj-aqua/pip-global/mkdocs (Python 3.11)

I'm not familiar with Python, but Python creates cache *.pyc.

$ ls -lh /Users/shunsukesuzuki/.local/share/aquaproj-aqua/pip-global/mkdocs/__pycache__ 
total 84K
-rw-r--r-- 1 shunsukesuzuki staff  237  8 23 09:02 __init__.cpython-311.pyc
-rw-r--r-- 1 shunsukesuzuki staff  20K  8 23 09:02 __main__.cpython-311.pyc
-rw-r--r-- 1 shunsukesuzuki staff 2.4K  8 23 09:02 exceptions.cpython-311.pyc
-rw-r--r-- 1 shunsukesuzuki staff 4.7K  8 23 09:02 localization.cpython-311.pyc
-rw-r--r-- 1 shunsukesuzuki staff  35K  8 23 09:02 plugins.cpython-311.pyc
-rw-r--r-- 1 shunsukesuzuki staff 8.8K  8 23 09:02 theme.cpython-311.pyc

@suzuki-shunsuke
Copy link
Member Author

I found an issue that even if I change the Python version, seems like mkdocs -V refers to old Python.

This is a bit confusing, but doesn't so matter.

@suzuki-shunsuke
Copy link
Member Author

suzuki-shunsuke commented Aug 24, 2024

📝 This isn't official but aqua can get the list of Python versions from this repository.

https://github.com/containerbase/python-prebuild

Python provides API, but aqua doesn't support getting versions from API.

https://www.python.org/api/v2/downloads/release/

ref. https://docs.renovatebot.com/modules/datasource/python-version/

@suzuki-shunsuke
Copy link
Member Author

suzuki-shunsuke commented Aug 24, 2024

Remaining tasks

@suzuki-shunsuke
Copy link
Member Author

Hmm... Python works on macOS, but doesn't work on Linux container.

Set up a container according to https://aquaproj.github.io/docs/tutorial/#docker

docker run --rm -ti debian:bookworm-20231009 bash
apt update
apt install -y curl vim

mkdir ~/workspace
cd ~/workspace
export PATH="${AQUA_ROOT_DIR:-${XDG_DATA_HOME:-$HOME/.local/share}/aquaproj-aqua}/bin:$PATH"
curl -sSfL -O https://raw.githubusercontent.com/aquaproj/aqua-installer/v3.0.1/aqua-installer
echo "fb4b3b7d026e5aba1fc478c268e8fbd653e01404c8a8c6284fdba88ae62eda6a  aqua-installer" | sha256sum -c

chmod +x aqua-installer
./aqua-installer

Update aqua to v2.31.0-2.

aqua upa v2.31.0-2

aqua.yaml

registries:
- type: standard
  ref: bcc7a2c5ae39a84af41b7ca559ac619e3f85e8ed
packages:
- name: indygreg/python-build-standalone@20240726
  vars:
    python_version: 3.11.9
aqua i -l
python -V
root@bc0be027deb3:~/workspace# python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.11'
  sys._base_executable = '/root/.local/share/aquaproj-aqua/bin/python'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/root/.local/share/aquaproj-aqua/bin/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python311.zip',
    '/install/lib/python3.11',
    '/install/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000ffff82890020 (most recent call first):
  <no Python frame>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request python
Projects
None yet
Development

No branches or pull requests

4 participants