Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running singularity workflow on other HPC platforms #17

Open
SorooshMani-NOAA opened this issue Jul 11, 2023 · 3 comments
Open

Running singularity workflow on other HPC platforms #17

SorooshMani-NOAA opened this issue Jul 11, 2023 · 3 comments
Assignees

Comments

@SorooshMani-NOAA
Copy link
Collaborator

So far the simple workflow implemented with Singularity containers and a Bash script is only tested on ParallelWorks platform. This needs to be tested on other HPC platforms as well. This Issue tracks problems related to that.

@SorooshMani-NOAA
Copy link
Collaborator Author

@WPringle I will soon share the Singularity images (.sif files) with you to test. Let's track all the issues (at high level) related to your tests here. Thanks again!

@SorooshMani-NOAA SorooshMani-NOAA self-assigned this Jul 11, 2023
@WPringle
Copy link

@SorooshMani-NOAA You can put images on "Ensemble research collection" globus shared directory. You should have permissions to write.

@SorooshMani-NOAA
Copy link
Collaborator Author

@WPringle I have transferred the singularity images to your shared directory. In order to use them you need to download the https://github.com/oceanmodeling/ondemand-storm-workflow repo and switch to feature/singularity branch.

There is then a directory called singularity at the top level which includes image definitions as well as a scripts directory. In that scripts directory there is a workflow.shand ainput.conffiles. In theory you should only modifyinput.confand runworkflow.sh` and the rest should be taken care of.

In input.conf file make sure to update:

  • L_NWM_DATASET: Location of NWM hydrofabric (extracted) file (downloadable at https://www.nohrsc.noaa.gov/pub/staff/keicher/NWM_live/web/data_tools/NWM_channel_hydrofabric.tar.gz)
  • L_TPXO_DATASET: Directory containing TPXO .nc files for pyschism to use
  • L_DEM_HI: Directory containing high resolution DEMs (not used for subsetting)
  • L_DEM_LO: Directory containing lower res DEMs, e.g. GEBCO
  • L_MESH_HI: High resolution mesh file (e.g. .14) for subsetting
  • L_MESH_LO: Low resolution mesh file (e.g. .14) for subsetting
  • L_SHP_DIR: Directory containing precomputed domain shapes (not used for subsetting)
  • L_IMG_DIR: Directory containing singularity images (.sif files I shared)
  • L_SCRIPT_DIR: Path to the scripts directory where input.conf resides
  • SINGULARITY_BINDFLAGS: Location bind flag for singularity, on an HPC you usually need to bind your scratch directory in order to have access to it from within the container, e.g. on ParallelWorks for me this would be "--bind /lustre"
  • TMPDIR: Path to a location where temporary files can be stored by OCSMesh and PySCHISM while computing intermediate steps. Since home directory usually has limited space, I set this to some temp location on my scratch dir. For example on ParallelWorks I set it to /lustre/.tmp
  • L_SOLVE_MODULES Name of the openmpi module on your HPC of choice, e.g. "openmpi/4.1.2"

Note that you need to make sure the HPC you're using has singularity command available. If it doesn't by default it might be under some module, that can for example be loaded by module load singularity or something like that.

Please let me know how it goes if you get the change to test this. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants