Home

AUS Cross-Platform Build and Test

To summarize the current status of on-going AUS build and test runs for HPC-stack and UFS-SRW app across platforms.

Test Status

Build and test runs are classified into two approaches based on container and non-container.

Non-container

Apps	Orion	Cheyenne	AWS	GC	Azure	Ubuntu	MacOS	Redline cluster
hpc-stack	intel, gnu-9.3	intel, gnu-9.3	gnu-9.3	gnu-9.3		gnu-9.3
SRW	intel	intel	gnu-9.3		gnu-9.3

Container

Apps	Orion	Cheyenne	AWS	GC	Azure	Ubuntu	MacOS	Redline cluster
SRW	gnu-9.3	gnu-9.3	gnu-9.3	gnu-9.3		gnu-9.3

Containers

• Several containers currently available on AUS team (built on ubuntu20.04 container):

  docker://noaaepic/ubuntu20.04-gnu9.3
  
  docker://noaaepic/ubuntu20.04-hpc-stack
  
  docker://noaaepic/ubuntu20.04-epic-srwapp
  
  docker://noaaepic/ubuntu20.04-epic-mrwapp
  

Current documents and instructions

• docs/userguide on the ufs-srw-app forked branch:

  • https://github.com/jkbk2004/ufs-srweather-app/blob/feature/srw-docker-v1.0.1/docs/UsersGuide/source/Dockersrw.rst

• EPIC-Documentation Test on readthedocs (sphinx):

  • https://epic-documentation-test.readthedocs.io/en/latest/

Action items

• Daily updates:

  01-26-2022: prepared ubuntu20.04-epic container and followed up to add libtiff-dev package in it

  01-27-2022: completed SRW container and non-container build test on Cheyenne, progress on hpc-stack build within ubuntu20.04 container

  01-31-2022: fms/2020.04.03, cmake/3.20 (mapl needs cmake/3.21), fortran dialect issue in fv3 diag files

  02-01-2022: ip2 needs cmake3.20 but mapl needs cmake3.21

  02-04-2022: hpc-stack builds ok on orion/cheyenne with intel-2021-2, needs to follow up hpc-stack issue#378, needs to follow up with Jeff's request for s2s fully coupled ufs build option,cheyenne role accounts

• hpc-stack build steps - option I (based on a ubuntu linux container, this is an example for the use in cloud with userid Jong.Kim):

  singularity pull ubuntu20.04-gnu9.3.sif docker://noaaepic/ubuntu20.04-gnu9.3
  
  singularity build --sandbox ubuntu20.04-gnu9.3 ubuntu20.04-gnu9.3.sif
  
  cd to ubuntu20.04-gnu9.3 and mkdir contrib to mount /contrib to the container
  
  cd ..
  
  git clone -b feature/ubuntu20.04 https://github.com/jkbk2004/hpc-stack
  
  singularity shell -e --writable --bind /contrib:/contrib ubuntu20.04-gnu9.3
  
  cd to hpc-stack
  
  ./setup_modules.sh -p /contrib/Jong.Kim/hpc-stack/hpc-modules -c config/config_custom.sh
  
  Keyboard inputs: yes/yes/yes
  
  sed -i "10 a source /usr/share/lmod/6.6/init/bash" ./build_stack.sh
  
  sed -i "10 a export PATH=/usr/local/sbin:/usr/local/bin:$PATH" ./build_stack.sh
  
  sed -i "10 a export LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:$LD_LIBRARY_PATH" ./build_stack.sh
  
  ./build_stack.sh -p /contrib/Jong.Kim/hpc-stack/hpc-modules -c config/config_custom.sh -y stack/stack_noaa.yaml -m
  
  make sure cmake3.21 (or higher version) available to install mapl
  
  module use /contrib/Jong.Kim/hpc-stack/hpc-modules/modulefiles/stack
  
  module load hpc hpc-gnu hpc-openmpi
  
  module avail
  
  module load netcdf hdf5 bacio sfcio sigio nemsio w3emc esmf fms crtm g2 png zlib g2tmpl ip sp w3nco cmake gfsio wgrib2 upp
  
  ./build_stack.sh -p /contrib/Jong.Kim/hpc-stack/hpc-modules -c config/config_custom.sh -y stack/stack_mapl.yaml -m
  

• UFS-SRW model build and forecast setup on HPCs (container approach on ubuntu laptop):

  singularity pull ubuntu20.04-hpc-stack-0.1.sif docker://noaaepic/ubuntu20.04-hpc-stack:0.1
  
  singularity build --sandbox ubuntu20.04-hpc-stack-0.1 ubuntu20.04-hpc-stack-0.1.sif
  
  mkdir home inside sandbox
  
  singularity shell -e --writable --bind /home:/home ubuntu20.04-hpc-stack
  
  Clone the develop branch of the UFS-SRW weather application repository:
  
  git clone https://github.com/jkbk2004/ufs-srweather-app
  
  cd ufs-srweather-app
  
  Check out submodules:
  
  ./manage_externals/checkout_externals
  
  Set build environments and modules:
  
  ln -s /usr/bin/python3 /usr/bin/python
  
  source /usr/share/lmod/6.6/init/profile
  
  module use /opt/hpc-modules/modulefiles/stack
  
  module load hpc hpc-gnu hpc-openmpi hpc-python
  
  module load netcdf hdf5 bacio sfcio sigio nemsio w3emc esmf fms crtm g2 png zlib g2tmpl ip sp w3nco cmake gfsio wgrib2 upp
  
  make build
  
  cd build
  
  cmake .. -DCMAKE_INSTALL_PREFIX=..
  
  make -j 4
  

• UFS-SRW model build and forecast setup on HPCs: non-container approach:

  Clone the develop branch of the UFS-SRW weather application repository:
  
  git clone -b develop https://github.com/ufs-community/ufs-srweather-app
  
  cd ufs-srweather-app
  
  Check out submodules:
  
  ./manage_externals/checkout_externals
  
  Load build environments and modules:
  
  source env/build_hera_intel.env
  
  Build the system:
  
  mkdir build
  
  cd build
  
  cmake .. -DCMAKE_INSTALL_PREFIX=..
  
  make -j 8 >& build.out &
  
  Once all build steps are complete,a directory structure will be created as shown below in a simplified screen capture. Next steps are to set up and launch the UFS-SRW forecast experiment:
  
  cd ../regional_workflow/ush; cp config.community.sh config.sh (need to edit config.sh according to user’s experiment configuration:
  
  source ../../env/wflow_hera.env
  
  Set up the experiment directory:
  
  ./generate_FV3LAM_wflow.sh
  
  Once the experiment sets up, the procedures to run the forecast is described in:
  
  https://github.com/jkbk2004/ufs-srweather-app/blob/feature/srw-docker-v1.0.1/docs/UsersGuide/source/Dockersrw.rst
  

Other technical tips

Need to update but a quick collection of technical tips! (just start working on a feature branch and commit back the change to the branch. Some useful git commands will follow for additional jobs: forking, create new branch, etc.):

git clone -b feature/srw-docker-v1.0.1 https://github.com/jkbk2004/ufs-srweather-app
cd path to file to work modify file

check status and branch name again

git status

git branch

commit the change to the feature branch

git add -A

git commit -m"description of change"

git push origin feature/srw-docker-v1.0.1

export SINGULARITY_CACHEDIR=/contrib/Jong.Kim/singularity

export SINGULARITY_TMPDIR=/contrib/Jong.Kim/singularity/tmp

singularity pull ubuntu20.04-epic-srwapp.sif docker://noaaepic/ubuntu20.04-epic-srwapp

singularity build --sandbox ubuntu20.04-epic-srwapp ubuntu20.04-epic-srwapp.sif

sphinx usage:

`link 1`_.

`link 2`_.

git clone -b gspedits https://github.com/jkbk2004/ufs-srweather-app gspedits

git clone https://github.com/ufs-community/ufs-srweather-app

cd ufs-srweather-app

git remote -v

git remote rename origin upstream

git remote -v

git remote add origin https://github.com/gillian-gitid/ufs-srweather-app

git remote -v

git remote update

git remote -v

git checkout -b gspedits

git checkout gspedits

git branch

git add -A

git commit -m"add nice and short comments"

git push origin gspedits