Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apptainer environment override, Python drop/bump, pyhit-wasp recipe marriage, versioner QOL #28847

Open
wants to merge 18 commits into
base: next
Choose a base branch
from
Open

Apptainer environment override, Python drop/bump, pyhit-wasp recipe marriage, versioner QOL #28847

wants to merge 18 commits into from

Conversation

milljm
Copy link
Member

@milljm milljm commented Oct 14, 2024 •
edited
Loading

This PR adds the following:

  • An ${HOME}/.bashrc_apptainer user override to be applied as they enter our containers if the file exists.
  • Python 3.8 drop (EOL).
  • Python 3.12 addition.
  • Combines PyHit Conda package into WASP's.
  • versioner.py quality of life improvment, by providing a summary of what should be appended to versioner_hashes.yaml when bumping an influential file: 
    $ ./versioner.py --summary
8e39fa181b8abba62f117006603d3a1dddea642d: #PR
tools: 0c9ab93
mpi: 80648d3
petsc: 9664f0c
libmesh: cf37955
wasp: 98d4d49
moose-dev: 5410b50

Closes #28813 #27690

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 1ecb994 wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 2f892f4 wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 6a3f65b wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 41d7ffe wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 770787a wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 68e3cfa wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 19ef3c1 wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config check on 21c813f wanted to post the following:

A change of the following file(s) triggered this check:

conda/peacock

The following file(s) are unchanged:

conda/mpi/conda_build_config.yaml

The conda build config was not changed but one or more of its dependencies have changed

@moosebuild
Copy link
Contributor

moosebuild commented Oct 16, 2024 •
edited
Loading

Job Documentation, step Docs: sync website on 0e89a6f wanted to post the following:

View the site here

This comment will be updated on new commits.

@moosebuild
Copy link
Contributor

moosebuild commented Oct 16, 2024 •
edited
Loading

Job Coverage, step Generate coverage on 6e0a8a2 wanted to post the following:

Framework coverage

Coverage did not change

Modules coverage

Coverage did not change

Full coverage reports

Reports

This comment will be updated on new commits.

milljm
milljm commented Oct 16, 2024
View reviewed changes
python/MooseDocs/test/extensions/test_versioner.py
'peacock',
'pprof',
'pyhit',
'seacas']
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@loganharbour I am not familiar with these things not having a 'conda version'. Especially the new tracking additions. But without these, I do get errors. I want to make sure I am doing this right.

error:

Traceback (most recent call last):
  File "/Users/milljm/projects/moose/python/MooseDocs/test/extensions/test_versioner.py", line 80, in testCodeRender
    conda_version = self.version_meta[package]['conda']['version_and_build']
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
KeyError: 'version_and_build'

I suppose I need to see how we're building this dictionary...

@milljm milljm mentioned this pull request Oct 17, 2024
Open
@milljm milljm force-pushed the 28813-add-user-override branch 2 times, most recently from 1ad08df to eb82471 Compare October 21, 2024 20:06
@milljm milljm changed the title 28813 add user override Apptainer environment override, Python drop/bump, pyhit-wasp recipe marriage, versioner QOL Oct 22, 2024
@milljm milljm force-pushed the 28813-add-user-override branch 2 times, most recently from 55d54af to d9e1699 Compare October 28, 2024 14:09
@milljm milljm mentioned this pull request Oct 29, 2024
Open
milljm and others added 16 commits October 29, 2024 08:48
@milljm
Add user-override singularity/env/ file
b28a56f 
Adjust versioner.yaml as suggested in next update.
Bump everything. Again.

pin xmltodict below 0.14 (issues discovered).

Closes idaholab#28813
@milljm
drop Python 3.8, support 3.12
5c7e301
@milljm
Droping Python 3.8 allowing 3.12
52d846c 
setuptools <60 is not available for Python >=3.12. We pinned
setuptools long ago. Lets see if we don't have to anymore.
@milljm
pyhit transition to idaholab pyhit
92e18f8 
Move away from using my own private pyhit repo.
Needed now that pyhit must be built, to conform to our new Python
version stack.

Because wasp/pyhit are closely intwined, it makes sense to have
pyhit as an output variant of wasp.

Modify versioner.yaml influential files to match changes.

Closes idaholab#27690
@milljm
consolidate CC CXX variables, fix unittest
c502f26 
Consolidate the compilers used for building Wasp and MPI. Looks
like these can now diverge on conda-forge.

Add additional packages to be ignored due to not having a conda
version... meh, I am not familiar with the why of this, as the
libraries added, are being tracked, and have a conda version.
TODO: ask Logan!
@milljm
Add retry functionality
7fe6572 
Add a retry function for all packages that can support it.
@milljm
Make the build script more like it was
78f981b 
I cannot fathom what I might be doing wrong. upstream/deve builds
seacas just fine. Try to make the way I am building seacas more
like it was, each step of the way.
@milljm
Pin compiler everywhere
de58a6d 
Everywhere we were allowing use of jinja2 compilers from conda-forge,
supply a pinned default instead. This insures our packages are
compatible with each other.

When moving away from jinja templated compilers, I found we are not
including enough. *compilers*_arch package is necessary. For example:

```pre
conda install gcc
cmake dosomething
```

...may fail with Cmake not able to discover where/how to use GCC.
SEACAS contrib example:

```pre
-- The C compiler identification is unknown
CMake Error at cmake/tribits/core/package_arch/TribitsGlobalMacros.cmake:1462 (enable_language):
  No CMAKE_C_COMPILER could be found.
```

`conda install gcc gcc_linux-64`, is the proper way (or rather, more
proper than without it). As with all things conda-build, I am still
learning.

SEACAS build.sh change:

Supply lib directory when building SEACAS. There seems to now be an
issue with contribs being installed in `lib64` while Conda still
expects things in `lib`. This is the bit I am referring to when I say
"or rather, more proper than without it" above. Before this change,
telling Cmake were libs go was not necessary...
@milljm
add runtime compiler libs
e13e0d3 
Seems tests fail now without it.
@milljm
adjust versions due to incoming changes
3871584 
Bump versions.

Fix VTK_PREFIX being declared outside environment. We are getting
lucky with the other builds. TODO: clean those up. Remove exports
where possible.

Clean up the retry_build function.
@milljm
Fix unbound variables
c07f5e8
@milljm
add ability to print versioner summary
d891e03 
Allow the printing of a summary of what to append to versioner_hashes.yaml

```bash
./versioner.py -s||--summary
```
@milljm
do not print 'app'
b0be2e6
@milljm
rebase on next to bring in libMesh update
e95b3e4
@milljm
Fix perms on ./singularity.d/envs/*
e593ba8 
Forgot to add the 99-user-overrides.sh file...
chmod +rx on *.sh

remove deprecated/TODO entry in versioner.py
@milljm
Update documentation tags for Python support
bdf27cd 
Drop Python 3.8, add 3.12.
Remove what seems no longer used: conda_version
@milljm
Fix Unittests, restrict python version in peacock
a663da6 
Restrict Python veresion in peacock.
Bring unittests inline with Python 3.12 requirements.
Fix non-raw strings.
@milljm
bump hashes
0e89a6f
@moosebuild
Copy link
Contributor

Job Precheck, step Conda build config notification on 0e89a6f wanted to post the following:

The following file(s) are changed:

conda/mpi/conda_build_config.yaml

The conda configuration has changed; ping @idaholab/moose-dev-ops

@moosebuild
Copy link
Contributor

Job Precheck, step Conda PETSc build config notification on 0e89a6f wanted to post the following:

The following file(s) are changed:

conda/petsc/meta.yaml

The PETSc conda configuration has changed; ping @idaholab/moose-dev-ops

@moosebuild
Copy link
Contributor

Job Precheck, step Conda libmesh build config notification on 0e89a6f wanted to post the following:

The following file(s) are changed:

conda/libmesh/meta.yaml

The libmesh conda configuration has changed; ping @idaholab/moose-dev-ops

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@loganharbour loganharbour loganharbour requested changes

@cticenhour cticenhour Awaiting requested review from cticenhour cticenhour is a code owner

Requested changes must be addressed to merge this pull request.

Assignees

@milljm milljm

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

create a 99-end-user.sh override ability in our Apptainer containers
3 participants
@milljm @moosebuild @loganharbour