Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use uv as a pip alternative #1088

Open
wants to merge 1 commit into
base: nightly
Choose a base branch
from

Conversation

MoisesGSalas
Copy link
Contributor

@MoisesGSalas MoisesGSalas commented Jul 1, 2024

uv is a new package installer/resolver written in rust by the same people that wrote ruff.

This is a small test to assess the viability of using uv instead of pip. It presents it self as a drop in replacement for pip, and for the most part it is, I only had to apply a single change due to limitations in editable VCS packages.

I included the change in the edx-platform requirements as a patch, but I think it makes sense to upstream it, I don't think there's any reason to use an editable VCS requirement for proctortrack (there's even the discussion about droping it from the requirements altogether).

I used a build argument to make it possible to use pip instead of uv in case people have incompatible requirements in one of their forks/patches.

Tests

Using --target=python-requirements:

command --no-cache time
uv X 05:10.26
pip X 11:15.94
uv - 00:15.98
pip - 01:17.16

During testing I added a RUN touch /tmp/1 instruction right before the pip install -r /openedx/edx-platform/requirements/edx/base.txt command to bust layer cache.

@MoisesGSalas MoisesGSalas changed the title feat: use uv as a pip alternative [WIP] feat: use uv as a pip alternative Jul 1, 2024
@MoisesGSalas MoisesGSalas marked this pull request as draft July 1, 2024 14:05
@regisb
Copy link
Contributor

regisb commented Sep 23, 2024

Hey Moises, these figures are getting everyone excited :) Do you think you'll be able to take this PR out of draft state? Or do you need some review?

@MoisesGSalas
Copy link
Contributor Author

Hi @regisb, sorry for leaving this abandoned. I can work on this, but first I wanted to know people's opinion about the tool.

IMO the tool is pretty awesome and slots in almost perfectly, the same applies to the other tool developed by the same team ruff. But considering it still has not reached v1.0.0 I wonder if some people may be worried about that.

If the overall feeling is that it's worth a shot, I can clean up the PR and have it ready for review.

@regisb
Copy link
Contributor

regisb commented Sep 24, 2024

Yes, ruff is awesome, to the point that I'm considering migrating the entire code base from pylint/black to ruff. But let's talk about this another day. For now I'd be curious to see this PR brought to completion.

@DawoudSheraz
Copy link
Contributor

@MoisesGSalas What are the plans for this? We can target this for nightly so that it becomes part of sumac cutoff next week.

@MoisesGSalas MoisesGSalas changed the base branch from master to nightly October 15, 2024 22:22
@MoisesGSalas MoisesGSalas changed the title [WIP] feat: use uv as a pip alternative feat: use uv as a pip alternative Oct 15, 2024
@MoisesGSalas MoisesGSalas marked this pull request as ready for review October 15, 2024 22:49
@MoisesGSalas
Copy link
Contributor Author

MoisesGSalas commented Oct 15, 2024

@DawoudSheraz, @regisb, sorry I couldn't find the time to retake this.

I think is ready for review. I wasn't quite sure what mechanism to use to fallback to the standard pip command if needed so I ended up using a build argument because it seemed to be the simpler one.

I rebased on top of nightly and targeted that branch. Build the image and imported a demo course just to check the platform launched correctly. I also mounted plugin using the platform-plugin-* automount and launched the dev environment and everything was working fine.

@@ -51,8 +55,10 @@ RUN git config --global user.email "[email protected]" \
{{ patch("openedx-dockerfile-git-patches-default") }}
{%- elif EDX_PLATFORM_VERSION == "master" %}
# Patches in nightly node
RUN curl -fsSL https://github.com/openedx/edx-platform/commit/3642cab3ac61ddfd360e7ceb7463b52be8c4deb0.patch | git am
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this patch needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that uv doesn't support editable dependencies is a blocking issue, I think. I suggest we wait until the upstream issue is resolved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they are not going to implement it in the near future. Note that it refers to editable remote requirements, it works fine with editable requirements in a local path.

AFAIK the only use for an editable remote requirement is that it copies all the files, but you can fix that using the MANIFEST.in.

Copy link
Contributor

@regisb regisb Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that. The, using uv means we will no longer be able to support adding editable dependencies to edx-platform. I'm not sure I'm ready to make that breaking change, as I expect that many people (Axim included) might add editable dependencies there.

Can we consider making an upstream PR to support that feature?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment gave me the impression they don't want to maintain code related to this feature:

As of now we intentionally don't support this. It adds a lot of complexity and we actually don't re-clone at all upon re-running the install command (we have a cache that contains a copy of each repository, and we checkout-and-install as needed with incremental fetches).

Maybe we can keep standard pip as the default with the possibility to swap to uv if your fork of edx-platform doesn't contain any editable dependency.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can keep standard pip as the default with the possibility to swap to uv if your fork of edx-platform doesn't contain any editable dependency.

That's already the case. People can install uv in their own Docker image and alias pip='uv pip', all with a Tutor plugin.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that wouldn't work for the RUN commands in the Dockerfile, right? The main appeal would be to use it in the python-requirements layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

3 participants