Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a docker workflow replacement for extbuild.yml #274

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions .github/workflows/dockerextbuild.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# This is a workflow to compile the cmeps source without cime
name: extbuild
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
push
# :
# branches: [ master ]
# pull_request:
# branches: [ master ]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
build-cmeps:
runs-on: ubuntu-latest
strategy:
matrix:
openmpi: [4.1.2]
hdf5: [1.12]
netcdfc: [4.8.1]
netcdff: [4.5.4]
pnetcdf: [1.12.3]
env:
CC: mpicc
FC: mpifort
CXX: mpicxx
CPPFLAGS: "-I/usr/include -I/usr/local/include"
# Versions of all dependencies can be updated here
ESMF_VERSION: jedwards/pio_update2
# PIO version
PIO_VERSION: pio2_5_6

container:
image: jedwards4b/o${{ matrix.openmpi }}h${{ matrix.hdf5 }}nc${{ matrix.netcdfc }}nf${{ matrix.netcdff }}p${{ matrix.pnetcdf }}
options: --user 1001
steps:
- name: checkout cmeps
uses: actions/checkout@v3
with:
path: components/cmeps
- name: checkout cmake_fortran_utils
uses: actions/checkout@v3
with:
repository: CESM-Development/CMake_Fortran_utils
path: build/CMake_Fortran_utils
- name: Checkout PIO
uses: actions/checkout@v3
with:
repository: NCAR/ParallelIO
ref: ${{ env.PIO_VERSION }}
path: ParallelIO
- name: Checkout ESMF
uses: actions/checkout@v3
with:
repository: esmf-org/esmf
ref: ${{ env.ESMF_VERSION }}
path: ESMF
- name: Checkout genf90
uses: actions/checkout@v3
with:
repository: ParallelIO/genf90
path: build/genf90
- name: Cache PIO
id: cache-PIO
uses: actions/cache@v2
with:
path: ParalleIO
key: ${{ runner.os }}-${{ env.PIO_VERSION }}.pio

- name: build PIO
if: steps.cache-PIO.outputs.cache-hit != 'true'
run: |
cd build
export PATH=/usr/local/bin:$PATH
cmake -Wno-dev \
-DCMAKE_INSTALL_PREFIX=${GITHUB_WORKSPACE}/pio \
-DGENF90_PATH=${GITHUB_WORKSPACE}/build/genf90 \
-DNetCDF_PATH=/usr/local/ \
-DPnetCDF_PATH=/usr/local \
-DPIO_ENABLE_FORTRAN=ON \
-DPIO_ENABLE_TIMING=Off \
../ParallelIO
make VERBOSE=1
make install

- id: cache-esmf
uses: actions/cache@v2
with:
path: esmf
key: ${{ runner.os }}-${{ env.ESMF_VERSION }}-ESMF
- name: build ESMF
if: steps.cache-esmf.outputs.cache-hit != 'true'
env:
ESMF_COMPILER: gfortran
ESMF_BOPT: g
ESMF_PIO: external
ESMF_PIO_INCLUDE: ${GITHUB_WORKSPACE}/pio/include
ESMF_PIO_LIBPATH: ${GITHUB_WORKSPACE}/pio/lib
ESMF_NETCDF: nc-config
ESMF_PNETCDF: pnetcdf-config
ESMF_COMM: openmpi
ESMF_INSTALL_PREFIX: esmf
run: |
cd ESMF
export ESMF_DIR=`pwd`
echo $ESMF_DIR
make -j 2
make install

- name: Build CMEPS
env:
ESMFMKFILE: ${GITHUB_WORKSPACE}/esmf/lib/libg/Linux.gfortran.64.openmpi.default/esmf.mk
PIO: ${GITHUB_WORKSPACE}/pio
run: |
mkdir build-cmeps
cd build-cmeps
cmake -DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_Fortran_FLAGS="-g -Wall -ffree-form -ffree-line-length-none" ../components/cmeps
make VERBOSE=1

129 changes: 129 additions & 0 deletions .github/workflows/dockersrt.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# CIME scripts regression tests

name: scripts regression tests

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
push
# :
# branches: main
# pull_request:
# branches: main

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
strategy:
matrix:
mpich: [4.0.1]
hdf5: [1.12]
netcdfc: [4.8.1]
netcdff: [4.5.4]
pnetcdf: [1.12.3]
env:
CC: mpicc
FC: mpifort
CXX: mpicxx
CPPFLAGS: "-I/usr/include -I/usr/local/parallel/include"
# Versions of all dependencies can be updated here
PARALLELIO_VERSION: pio2_5_6
ESMF_VERSION: jedwards/pio_update2
CIME_MODEL: cesm
container:
image: jedwards4b/m${{ matrix.mpich }}h${{ matrix.hdf5 }}nc${{ matrix.netcdfc }}nf${{ matrix.netcdff }}p${{ matrix.pnetcdf }}

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- name: cime checkout
uses: actions/checkout@v3
with:
repository: ESMCI/cime

- name: share checkout
uses: actions/checkout@v3
with:
repository: ESCOMP/CESM_share
path: share

- name: cpl7 checkout
uses: actions/checkout@v3
with:
repository: ESCOMP/CESM_CPL7andDataComps
path: components/cpl7

- name: mct checkout
uses: actions/checkout@v3
with:
repository MCSclimate/MCT
path: libraries/mct

- name: parallelio checkout
uses: actions/checkout@v3
with:
repository NCAR/ParallelIO
path: libraries/parallelio

- name: Cache inputdata
id: cache-inputdata
uses: actions/cache@v2
with:
path: $HOME/cesm/inputdata
key: inputdata

- name: Checkout ESMF
uses: actions/checkout@v3
with:
repository: esmf-org/esmf
ref: ${{ env.ESMF_VERSION }}
path: ESMF

- id: cache-esmf
uses: actions/cache@v2
with:
path: esmf
key: ${{ runner.os }}-${{ env.ESMF_VERSION }}-ESMF
- name: build ESMF
if: steps.cache-esmf.outputs.cache-hit != 'true'
env:
ESMF_COMPILER: gfortran
ESMF_BOPT: g
ESMF_PIO: external
ESMF_PIO_INCLUDE: ${GITHUB_WORKSPACE}/pio/include
ESMF_PIO_LIBPATH: ${GITHUB_WORKSPACE}/pio/lib
ESMF_NETCDF: nc-config
ESMF_PNETCDF: pnetcdf-config
ESMF_COMM: openmpi
ESMF_INSTALL_PREFIX: esmf
run: |
cd ESMF
export ESMF_DIR=`pwd`
echo $ESMF_DIR
make -j 2
make install
#
# The following can be used to ssh to the testnode for debugging
# see https://github.com/mxschmitt/action-tmate for details
# - name: Setup tmate session
# uses: mxschmitt/action-tmate@v3

- name: scripts regression tests
run: |
mkdir -p ${HOME}/cesm/scratch
mkdir -p ${HOME}/cesm/inputdata
cd ${HOME}/work/CESM_share/CESM_share/scripts/tests
ls -l $HOME/work/CESM_share/CESM_share
export NETCDF=/usr/local/parallel
export PATH=$NETCDF/bin:$PATH
export LD_LIBRARY_PATH=$NETCDF/lib:$HOME/pnetcdf/lib:$LD_LIBRARY_PATH
./scripts_regression_tests.py --no-fortran-run --compiler gnu --mpilib mpich --machine ubuntu-latest

# the following can be used by developers to login to the github server in case of errors
# see https://github.com/marketplace/actions/debugging-with-tmate for further details
# - name: Setup tmate session
# if: ${{ failure() }}
# uses: mxschmitt/action-tmate@v3
2 changes: 1 addition & 1 deletion cime_config/buildexe
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ def _main_func():
if os.path.isfile(exename):
os.remove(exename)

cmd = "{} exec_se -j {} EXEC_SE={} MODEL=driver {} -f {} "\
cmd = "{} exec_se -j {} EXEC_SE={} CIME_COMP=driver {} -f {} "\
.format(gmake, gmake_j, exename, gmake_args, makefile)


Expand Down
33 changes: 21 additions & 12 deletions cime_config/config_component.xml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@
<desc>List of component classes supported by this driver</desc>
</entry>

<entry id="CONTAINER_ENVIRONMENT">
<type>char</type>
<valid_values></valid_values>
<default_value></default_value>
<group>case_comp</group>
<file>env_case.xml</file>
<desc>Container environment to invoke, if any</desc>
</entry>

<entry id="COMP_CPL">
<type>char</type>
<valid_values>cpl</valid_values>
Expand Down Expand Up @@ -648,7 +657,7 @@
<group>build_def</group>
<file>env_build.xml</file>
<desc>Output root directory for each machine.
Base directory for build and run directories.
Base directory for build and run directories.
</desc>
</entry>

Expand Down Expand Up @@ -997,9 +1006,9 @@
<file>env_run.xml</file>
<desc>
Determines what ESMF log files (if any) are generated when
USE_ESMF_LIB is TRUE.
USE_ESMF_LIB is TRUE.
ESMF_LOGKIND_SINGLE: Use a single log file, combining messages from
all of the PETs. Not supported on some platforms.
all of the PETs. Not supported on some platforms.
ESMF_LOGKIND_MULTI: Use multiple log files -- one per PET.
ESMF_LOGKIND_NONE: Do not issue messages to a log file.
By default, no ESMF log files are generated.
Expand Down Expand Up @@ -1040,8 +1049,8 @@
<group>run_flags</group>
<file>env_run.xml</file>
<desc>Turns on component barriers for component timing.
This variable is for testing and debugging only and should never
be set for a production run.
This variable is for testing and debugging only and should never
be set for a production run.
</desc>
</entry>

Expand Down Expand Up @@ -1249,7 +1258,7 @@
<group>run_domain</group>
<file>env_run.xml</file>
<desc>Latitude of grid location, in single column mode interpolate datasets to this location
in single point mode assume all datasets are at this location</desc>
in single point mode assume all datasets are at this location</desc>
</entry>

<entry id="PTS_LON">
Expand All @@ -1258,7 +1267,7 @@
<group>run_domain</group>
<file>env_run.xml</file>
<desc>Longitude of grid location, in single column mode interpolate datasets to this location
in single point mode assume all datasets are at this location</desc>
in single point mode assume all datasets are at this location</desc>
</entry>

<entry id="PTS_DOMAINFILE">
Expand All @@ -1267,10 +1276,10 @@
<group>run_domain</group>
<file>env_run.xml</file>
<desc>used only if if PTS_LAT and PTS_LON are greater than or
equal to 0. If this is the case then if PTS_DOMAINFILE is not
equal to UNSET a nearest neighbor search of PTS_DOMAINFILE using
PTS_LAT and PTS_LON will be done and the component mesh will have
this nearest neighbor value. </desc>
equal to 0. If this is the case then if PTS_DOMAINFILE is not
equal to UNSET a nearest neighbor search of PTS_DOMAINFILE using
PTS_LAT and PTS_LON will be done and the component mesh will have
this nearest neighbor value. </desc>
</entry>

<!-- ======================================================================= -->
Expand Down Expand Up @@ -1955,7 +1964,7 @@
<desc>pio rearranger communication max pending requests (io2comp) :
-2 implies that CIME internally calculates the value ( = 64),
-1 implies no bound on max pending requests
0 implies that MPI_ALLTOALL will be used
0 implies that MPI_ALLTOALL will be used
</desc>
</entry>

Expand Down