Configure GitHub Actions to build binary wheels and upload to pypi #117

Akuli · 2022-08-19T23:13:37Z

Discussed on #103. ~~Still needs more work.~~

Carreau · 2022-09-07T08:34:36Z

I'd love to have this as well, I'm working on a project that uses tree-sitter, and some users have had problem building tree-sitter.

Is there anything I can do to help?

Akuli · 2022-09-07T08:57:28Z

For now, you can use https://github.com/Akuli/py-tree-sitter-builds which installs tree_sitter and tree_sitter_languages without requiring a C compiler.

maxbrunsfeld · 2022-09-07T16:03:13Z

This awesome! I can add PyPi credentials to the org if they are needed for publishing. When you get a chance, could you write up what you think still needs to be done?

Akuli · 2022-09-07T16:23:38Z

I think a good next step would be including wheels for the tree_sitter module, but not the language binaries (currently tree_sitter_languages), for a couple reasons:

It is not needed to get tree-sitter working on systems without a C compiler. The existing tree-sitter-languages package depends on tree-sitter, and would install without requiring a C compiler if tree-sitter didn't require a C compiler.
Some languages are licensed under MIT. Others use Apache 2.0. This repo uses MIT license so we can't just bundle something that is Apache 2.0 licensed.
Including the languages makes the build slower.

To get this working:

We need to agree on when this workflow runs. Currently releasing is done by a script that is ran locally and pushes a tag. Should this run automatically when a tag is pushed, or be triggered manually from GitHub's UI, or both?
Someone from the tree-sitter org needs to add a PyPI api token as a secret to github actions. The secret should be named PYPI_API_TOKEN (although this doesn't really matter, just has to match the .yml file). It should have permission to upload to the tree-sitter project on pypi. Generating a token on pypi.org is quite straight-forward.
I need to copy my newer config from tree-sitter-builds to here, and delete the tree_sitter_languages related parts.
I currently use a patch file with git apply to make this project's tests work in cibuildwheel. I will need to figure out a cleaner way to do that, maybe an environment variable that is set when running under cibuildwheel or including the Python version in the language binary file name.
I need some way to test this pull request, after making all the changes. I believe there is a test version of pypi, so I can use that.

lunixbochs · 2022-09-07T16:28:10Z

This repo uses MIT license so we can't just bundle something that is Apache 2.0 licensed

You can certainly ship a wheel that has both MIT and Apache 2.0 code in it, just need to make sure all of the license files are present with correct attribution.

maxbrunsfeld · 2022-09-07T16:40:02Z

We need to agree on when this workflow runs. Currently releasing is done by a script that is ran locally and pushes a tag. Should this run automatically when a tag is pushed, or be triggered manually from GitHub's UI, or both?

Personally, I think just doing it based on any tag that starts with v and then a digit (or something like that) is a good approach.

Someone from the tree-sitter org needs to add a PyPI api token as a secret to github actions. The secret should be named PYPI_API_TOKEN (although this doesn't really matter, just has to match the .yml file). It should have permission to upload to the tree-sitter project on pypi. Generating a token on pypi.org is quite straight-forward.

I can take care of that.

I currently use a patch file with git apply to make this project's tests work in cibuildwheel. I will need to figure out a cleaner way to do that, maybe an environment variable that is set when running under cibuildwheel or including the Python version in the language binary file name.

I'm not familiar with cibuildwheel, maybe @lunixbochs can advise on that issue.

I need some way to test this pull request, after making all the changes. I believe there is a test version of pypi, so I can use that.

Maybe we just test by creating a tag with a qualifier like v0.20.2-beta0, -beta1 etc, so that we won't spam our PyPi package with bogus releases?

maxbrunsfeld · 2022-09-07T16:43:32Z

Ok, the PYPI_API_TOKEN secret has been added, so it should be accessible to actions.

Akuli · 2022-09-07T18:51:13Z

I'm not familiar with cibuildwheel

This is more of a thing with the tests than with cibuildwheel. The tests use Language.build_library() which looks at the modified time to check whether the library needs to be rebuilt. This works great, except that when cibuildwheel runs the tests with multiple Python versions, both 32-bit and 64-bit, the languages binary from the previous Python version is still there and so the tests fail to load it.

A few ways how I could work around it, not sure yet which I will choose:

Just delete the languages binary before running tests
Add an environment variable that forces the binary to be rebuilt, if it is set
Use an existing environment variable for this, probably some variable that is always set when running in cibuildwheel

This reverts commit d670a46.

…uild" This reverts commit 66bb76f.

Carreau · 2022-09-07T20:13:04Z

need some way to test this pull request, after making all the changes. I believe there is a test version of pypi, so I can use that.

You can just use GH upload action, and manually point pip to the locally downloaded wheels (or even download the wheel in another action).

Carreau · 2022-09-07T20:17:16Z

tests/test_tree_sitter.py

+LIB_PATH = path.join(project_root, "build", "languages.so")
+if os.getenv("PY_TREE_SITTER_TESTS_FORCE_REBUILD_LANGUAGES"):
+    try:
+        os.remove(LIB_PATH)
+    except FileNotFoundError:
+        pass
+


Would it be worth it to have a .so name that contain the python version/platform in the same way that pyc files ends with for example cpython-310 ? That would avoid any conflicts.

It actually has to be something like cpython-310-amd64, because 32-bit and 64-bit wheels are built in the same environment. I'll change this.

Sure, and like also sys.platform to be sure ?

Turns out that just placing the languages binary to current working directory works. This seems to be how cibuildwheel is meant to be used: it makes a new working directory for each wheel build just so that you can make a mess in it to be cleaned up before the next build.

This reverts commit a4b0ebb.

This reverts commit 5ed541e.

Akuli · 2022-09-07T20:22:26Z

It is working: https://test.pypi.org/project/tree-sitter-test/

Carreau · 2022-09-07T20:32:22Z

It is working: https://test.pypi.org/project/tree-sitter-test/

Working for me on mac M1 and Linux x86_64 for Python 3.10.

$ pip install tree-sitter-test --extra-index-url https://test.pypi.org/simple

If there are lurkers that don't know how to point pip at test-pypi

This reverts commit 90c8921.

This reverts commit 82cf80d.

Akuli · 2022-09-07T21:00:04Z

Ready for review!

maxbrunsfeld

I just left one minor question.

.github/workflows/pypi.yml

leotrs · 2022-12-31T19:00:28Z

Would absolutely love to see this merged soon. It looks like it is ready to be merged?

EDIT: I have now been able to deploy a python app using tree-sitter-test from @Akuli's repository, whereas I had not been able to compile this repo's source on my docker image.

Configure GitHub Actions to build binary wheels for pypi

d45ef1e

Akuli mentioned this pull request Aug 27, 2022

Microsoft Visual C++ 14.0 requirement #103

Closed

Akuli added 5 commits September 7, 2022 21:53

Merge remote-tracking branch 'origin/master' into pypiwheels

c65192f

Copy config from other repository

139c2c5

Delete old publish script

a8ce724

A pile of changes. Still need to test all this

4cdf587

Update wheels.yml

bc28dd1

Akuli marked this pull request as draft September 7, 2022 19:21

Akuli added 9 commits September 7, 2022 22:22

TEMPORARILY delete ci.yml

5ed541e

TEMPORARILY replace slow things with a zip file of previous build

66bb76f

Try dis

d670a46

Revert "Try dis"

66d9af3

This reverts commit d670a46.

TEMPORARILY change package name

a4b0ebb

fix

636b144

Revert "TEMPORARILY replace slow things with a zip file of previous b…

ae9e669

…uild" This reverts commit 66bb76f.

Give names to confusing pypi jobs

2964029

Style nit

ca73f3e

Carreau reviewed Sep 7, 2022

View reviewed changes

Akuli added 2 commits September 7, 2022 23:21

Revert "TEMPORARILY change package name"

faa25ba

This reverts commit a4b0ebb.

Revert "TEMPORARILY delete ci.yml"

c856d52

This reverts commit 5ed541e.

Akuli added 3 commits September 7, 2022 23:24

TEMPORARILY change package name

82cf80d

TEMPORARILY bump version

90c8921

Name binary better

eab8834

Let's try this

eff5fda

Akuli mentioned this pull request Sep 7, 2022

README: should we mention py-tree-sitter-languages? #120

Closed

Akuli added 4 commits September 7, 2022 23:55

Revert "TEMPORARILY bump version"

c45c96f

This reverts commit 90c8921.

Revert "TEMPORARILY change package name"

61b634a

This reverts commit 82cf80d.

Cleanups

9e85a80

Rename

fa50952

Akuli marked this pull request as ready for review September 7, 2022 20:59

Move comment

17e9b40

Akuli changed the title ~~Configure GitHub Actions to build binary wheels for pypi~~ Configure GitHub Actions to build binary wheels and upload to pypi Sep 7, 2022

maxbrunsfeld reviewed Sep 7, 2022

View reviewed changes

.github/workflows/pypi.yml Outdated Show resolved Hide resolved

maxbrunsfeld requested a review from lunixbochs September 7, 2022 22:08

Carreau approved these changes Sep 8, 2022

View reviewed changes

One or more digits

f963f48

amaanq approved these changes Aug 2, 2023

View reviewed changes

amaanq merged commit 9239f3d into tree-sitter:master Aug 2, 2023
17 checks passed

ofek mentioned this pull request Aug 29, 2023

New tag and PyPi release? #149

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure GitHub Actions to build binary wheels and upload to pypi #117

Configure GitHub Actions to build binary wheels and upload to pypi #117

Akuli commented Aug 19, 2022 •

edited

Loading

Carreau commented Sep 7, 2022

Akuli commented Sep 7, 2022

maxbrunsfeld commented Sep 7, 2022

Akuli commented Sep 7, 2022 •

edited

Loading

lunixbochs commented Sep 7, 2022

maxbrunsfeld commented Sep 7, 2022 •

edited

Loading

maxbrunsfeld commented Sep 7, 2022

Akuli commented Sep 7, 2022 •

edited

Loading

Carreau commented Sep 7, 2022

Carreau Sep 7, 2022

Akuli Sep 7, 2022

Carreau Sep 7, 2022

Akuli Sep 7, 2022

Akuli commented Sep 7, 2022

Carreau commented Sep 7, 2022

Akuli commented Sep 7, 2022

maxbrunsfeld left a comment

leotrs commented Dec 31, 2022 •

edited

Loading

Configure GitHub Actions to build binary wheels and upload to pypi #117

Configure GitHub Actions to build binary wheels and upload to pypi #117

Conversation

Akuli commented Aug 19, 2022 • edited Loading

Carreau commented Sep 7, 2022

Akuli commented Sep 7, 2022

maxbrunsfeld commented Sep 7, 2022

Akuli commented Sep 7, 2022 • edited Loading

lunixbochs commented Sep 7, 2022

maxbrunsfeld commented Sep 7, 2022 • edited Loading

maxbrunsfeld commented Sep 7, 2022

Akuli commented Sep 7, 2022 • edited Loading

Carreau commented Sep 7, 2022

Carreau Sep 7, 2022

Choose a reason for hiding this comment

Akuli Sep 7, 2022

Choose a reason for hiding this comment

Carreau Sep 7, 2022

Choose a reason for hiding this comment

Akuli Sep 7, 2022

Choose a reason for hiding this comment

Akuli commented Sep 7, 2022

Carreau commented Sep 7, 2022

Akuli commented Sep 7, 2022

maxbrunsfeld left a comment

Choose a reason for hiding this comment

leotrs commented Dec 31, 2022 • edited Loading

Akuli commented Aug 19, 2022 •

edited

Loading

Akuli commented Sep 7, 2022 •

edited

Loading

maxbrunsfeld commented Sep 7, 2022 •

edited

Loading

Akuli commented Sep 7, 2022 •

edited

Loading

leotrs commented Dec 31, 2022 •

edited

Loading