Move dependencies to pyproject toml and create ci/cd install and import test

[leifdenby]

The aim of this PR is to move dependencies from requirements.txt to pyproject.toml (as has been proposed for v0.2.0 on the roadmap) and discussed on #27 (comment).

Requirements:

• dependencies should still be installable with pip and pyproject.toml can be updated by hand
• dependencies can be handle with pdm
• ci/cd test that runs the pip install and runs a test that imports neural-lam to ensure that the required packages have correctly been installed.

[leifdenby]

I will copy here my motivation for allow using of pdm from #27 (comment):

For someone not using pdm the process of setting up a new development environment would be:

1. Clone repo git clone http://github.com/<username>/neural-lam and change directory cd neural-lam
2. Optionally create your virtualenv with your favourite tool, maybe mkvirtualenv ... and then activate ...
3. Install neural-lam editable with pip so that changes you make are reflected in the site-packages version (actually this is simply a link) pip install -e .
4. Edit and run neural-lam: python -m neural_lam.train_model --config-file config.yaml
5. Manually install a dependency with pip and edit pyproject.toml by hand, i.e. python -m pip install ... and edit the pyproject.toml file
6. Stage, commit and push!

I like using a package manager like pdm (other options are pipenv, poetry, uv) because it makes the development process easier by 1) automatically updating package versions in pyproject.toml to ensure versions work together when I add a new package, 2) allowing me to "dry run" any dependency modification without clobbering my current env (this has hurt me quite a few times with pip) and 3) handling creating and activating virtuals envs

So with pdm the development process would be:

1. Clone repo git clone http://github.com/<username>/neural-lam and change directory cd neural-lam
2. Create virtual env pdm venv create and either activating it pdm venv activate ... or making it the default when using pdm run python ... with pdm use ...
3. Install package pdm install
4. Add dependency pdm add ... (this is where you can do pdm add --dry-run ... to see what would change before you install a package), or remove one with pdm remove .... You can also add dev dependencies separately (which won't be included in wheels) with pdm add --dev ... or dependency groups with pdm add --group visualisation matplotlib for example (if we didn't want visualisation tools installed by default)
5. Edit and run neural-lam: python -m neural_lam.train_model --config-file config.yaml (if using activated virtualenv) or pdm run python -m neural_lam.train_model --config-file config.yaml
6. Stage, commit and push!

[leifdenby]

I see. Yes, I think using an optional extras group is preferable then, since it would work smoothly with both pip and pdm.

Cool. Yes I think you're right. I would imagine @sadamov, @SimonKamuk and @khintz are happy with this too. If I don't hear any complaints I will change that this morning and update the PR (README and pyproject.toml)

I will take that is your review so far @joeloskarsson 😀

[joeloskarsson]

I have looked over all the changes and I think it looks good. Want to do a full install on my end and see that everything works, but will wait with that until you've made that final change. I have not looked in detail on any of the github workflows, as I don't know that stuff so well. But I trust others to go over that.

Does this go in before or after #32 ?

[leifdenby]

I have looked over all the changes and I think it looks good. Want to do a full install on my end and see that everything works, but will wait with that until you've made that final change. I have not looked in detail on any of the github workflows, as I don't know that stuff so well. But I trust others to go over that.

Great, thanks!

Does this go in before or after #32 ?

After I would say. The other PR is about moving files around to make neural-lam into a package. Once we've done that I can make sure that package installs and runs correctly for both cpu and gpu support by adding the changes in this PR

[sadamov]

Looking great! Thanks, really also for the work on the github runners which must have been quite a hassle to set up 🚀

I have tried installation with pdm (4m30s) and pip (6m54s) (latest torch version) on my HPC system running CUDA 12.0 and can report that for both envs neuralLAM-training was successful. Tested both on cpu and gpu.

After capping the numpy version this can be merged, from my side. I trust that the --dev install will work after the change discussed above.

[joeloskarsson]

Change the numpy version spec and the dev group name, then this is all good to go!

Something I realized, that's really good, is that we don't actually need any of the optional PyG dependencies atm. So this getting rid of them is great, and also leads to less complications with torch CPU/CUDA versions. We can just re-introduce them later if some will be needed.

[leifdenby]

Ok @sadamov and @joeloskarsson, I've moved the dev dependencies to the optional dependency groups and updated the ci/cd setups and README to reflect the change.

If you have time for a last quick look then I think this is ready to go in!

[joeloskarsson]

• Now this includes ( a bit hidden) the upgrade to numpy 2.0. Can we make this a bit more transparent? For example by adding this specifically to the changelog, as it now just contains the entry for capping it to <2.0.
• If we now are moving to allow numpy 2.0, should we actually make this >=2.0.0, so we do not need to make sure to support both numpy 1 and 2 in the future?

[leifdenby]

• Now this includes ( a bit hidden) the upgrade to numpy 2.0. Can we make this a bit more transparent? For example by adding this specifically to the changelog, as it now just contains the entry for capping it to <2.0.

• If we now are moving to allow numpy 2.0, should we actually make this >=2.0.0, so we do not need to make sure to support both numpy 1 and 2 in the future?

Ah! Sorry, that was an oversight. I had somehow convinced myself we now did support numpy>=2.0.0, but of course #67 capped the version. I'm glad you noticed this. I will make another commit to cap the version as it was before in requirements.txt

[leifdenby]

Ok, I fixed that issue now @joeloskarsson

[joeloskarsson]

Oh no, sorry for the confusion. We want to remove the cap to <2.0 here, just make it clear that we do that. The code works with numpy 2, see #67

[sadamov]

I installed the dev envs with both pip and pdm now, it works!
One model training was also succesful with new envs.

I agree to remove the numpy cap and mention it in the changelog of this PR.

Happy merging 🎉

[leifdenby]

Oh no, sorry for the confusion. We want to remove the cap to <2.0 here, just make it clear that we do that. The code works with numpy 2, see #67

Ah I see! Ok, so I will include a note in the CHANGELOG to say that we removed the cap to numpy<2.0.0 because it works. I'll draft that changelog entry now and revert the changes

[joeloskarsson]

Great! Now everything looks good to me.