Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Unit Tests for CI Codes #2581

Closed
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
174 commits
Select commit Hold shift + click to select a range
a62a9f0
added development Jenkninsfile for dispatching from GitHub Actions
TerryMcGuinness-NOAA May 6, 2024
959edf5
Merge branch 'NOAA-EMC:develop' into feature/jenkins-dispatch
TerrenceMcGuinness-NOAA May 8, 2024
002c985
updated method for counting status cases from rocotostat
TerryMcGuinness-NOAA May 8, 2024
0243294
started a unit test for rocotostat
TerryMcGuinness-NOAA May 8, 2024
233c614
added more moc args
TerryMcGuinness-NOAA May 8, 2024
eb3589c
updated third arg to moc
TerryMcGuinness-NOAA May 8, 2024
060f2a8
removed moc stuff from test
TerryMcGuinness-NOAA May 8, 2024
2f9662d
added data for unit testing rocotostat
TerryMcGuinness-NOAA May 8, 2024
b01504d
moved tests dir for ci unit tests under scripts
TerryMcGuinness-NOAA May 8, 2024
a6508f5
added ci unit tests using pytest
TerryMcGuinness-NOAA May 8, 2024
ce9af99
chris H has install.h spelled INSTALL
TerryMcGuinness-NOAA May 8, 2024
060aed6
added path to rocotostat in GitHub runner
TerryMcGuinness-NOAA May 8, 2024
336cbc4
added path relative to sript for input data
TerryMcGuinness-NOAA May 8, 2024
0a3561d
did not put in path for artifacts on success to unittess
TerryMcGuinness-NOAA May 8, 2024
c36ef99
details for artifacts and new actions file
TerryMcGuinness-NOAA May 8, 2024
4e89645
syntax errors in ci unit tests actions yaml file
TerryMcGuinness-NOAA May 8, 2024
c59c997
added report in UI
TerryMcGuinness-NOAA May 8, 2024
cb3c3e5
trying another reportor
TerryMcGuinness-NOAA May 8, 2024
2dfe779
needed update permitions for report
TerryMcGuinness-NOAA May 8, 2024
ed29a28
added conclution to pytest ci
TerryMcGuinness-NOAA May 8, 2024
b22801a
indetation error in ci-pyunit acdtion file
TerryMcGuinness-NOAA May 8, 2024
f868628
removed conclution in messages
TerryMcGuinness-NOAA May 8, 2024
dfdcf7a
job_summary did not work
TerryMcGuinness-NOAA May 8, 2024
ca60b13
more publish and config report
TerryMcGuinness-NOAA May 8, 2024
c29acdf
more publish and config report path update
TerryMcGuinness-NOAA May 8, 2024
74fe7ee
added full path to xml output
TerryMcGuinness-NOAA May 8, 2024
10b0489
needed space for name
TerryMcGuinness-NOAA May 8, 2024
58ad853
needed space for name
TerryMcGuinness-NOAA May 8, 2024
ca985a2
had name wrong was file name up upload
TerryMcGuinness-NOAA May 8, 2024
316d2ee
took out upload
TerryMcGuinness-NOAA May 8, 2024
0cb4a6d
added docstrings to functions
TerryMcGuinness-NOAA May 8, 2024
2534fed
removing Jenkins dispatch file
TerryMcGuinness-NOAA May 8, 2024
89d6058
added verbose in the pytest argument to list each test
TerryMcGuinness-NOAA May 8, 2024
100c893
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA May 9, 2024
d540011
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA May 10, 2024
a53445b
moved test data specfic to rocotostat tests in test directory and neu…
TerryMcGuinness-NOAA May 10, 2024
6fc76b2
Merge branch 'ci_unit-tests' of github.com:TerrenceMcGuinness-NOAA/gl…
TerryMcGuinness-NOAA May 10, 2024
eafc6cc
cashed install of Rocoto in actions runner
TerryMcGuinness-NOAA May 10, 2024
6ed8e71
update cashe action to version 4
TerryMcGuinness-NOAA May 10, 2024
e81512b
added test data for rocotocheck stalled case
TerryMcGuinness-NOAA May 10, 2024
14b1622
added stalled and done as functions and added two more unit tests for…
TerryMcGuinness-NOAA May 10, 2024
5901d51
added doctrings to new functions in rocotstat
TerryMcGuinness-NOAA May 10, 2024
9fe4f62
updated pynorm spaces
TerryMcGuinness-NOAA May 10, 2024
2f3cd87
do not need to assert is set to true, it just is
TerryMcGuinness-NOAA May 10, 2024
9bb3b24
Merge pull request #286 from TerrenceMcGuinness-NOAA/ci_unit-tests_st…
TerrenceMcGuinness-NOAA May 10, 2024
7f7f705
cashe after intall or Rocoto silly
TerryMcGuinness-NOAA May 10, 2024
b201a6e
Merge branch 'ci_unit-tests' of github.com:TerrenceMcGuinness-NOAA/gl…
TerryMcGuinness-NOAA May 13, 2024
4566503
removded all rocoto test data and put them in S3 bucket and have them…
TerryMcGuinness-NOAA May 13, 2024
2ad0c1c
fix pynorms and install wget in runner to download test data
TerryMcGuinness-NOAA May 13, 2024
6659215
removed stub files and added .gitkeep and correspondingly update uint…
TerryMcGuinness-NOAA May 13, 2024
33de358
pycodestyle indenetation fix
TerryMcGuinness-NOAA May 13, 2024
2b4e774
cd into the ci test dir and run pytest there to resolve paths correctly
TerryMcGuinness-NOAA May 13, 2024
12ee2db
update path to test results for them to be published
TerryMcGuinness-NOAA May 13, 2024
73b0f11
put in full path to test resultes because the weill not read and move…
TerryMcGuinness-NOAA May 13, 2024
0fc0ec8
trying to debug output test results xml file
TerryMcGuinness-NOAA May 13, 2024
4ed0fa6
using relative path in Pub reslutes and updated test for cashed Rocot…
TerryMcGuinness-NOAA May 13, 2024
7282787
the cached section has to come before the install of Rocoto because i…
TerryMcGuinness-NOAA May 13, 2024
b63adea
moved Cache to before install of Rocoto because it had a duel purpose…
TerryMcGuinness-NOAA May 13, 2024
92971b0
Update .github/workflows/ci-unit_tests.yaml
TerrenceMcGuinness-NOAA May 14, 2024
8c5dbd6
Update .github/workflows/ci-unit_tests.yaml
TerrenceMcGuinness-NOAA May 14, 2024
0d68041
removed caps in docstrings and testing write permissions to pr in act…
TerryMcGuinness-NOAA May 14, 2024
6425410
made deep copies of objects in calls to --all and --summary are disti…
TerryMcGuinness-NOAA May 14, 2024
b5cbcda
Update .github/workflows/ci-unit_tests.yaml
TerrenceMcGuinness-NOAA May 15, 2024
4ae1854
Update .github/workflows/ci-unit_tests.yaml
TerrenceMcGuinness-NOAA May 20, 2024
fc7549e
added a attempt_multiple_times routine to the python rocotostat utili…
TerryMcGuinness-NOAA May 21, 2024
e6ad4fa
it is workflow_dispatch not dispatch
TerryMcGuinness-NOAA May 21, 2024
9581a14
merged develop
TerryMcGuinness-NOAA May 29, 2024
9ee2e6a
merged in changes to rocoto check from develop
TerryMcGuinness-NOAA May 29, 2024
656eb92
added docstring to attempt_multiple_times in the rocoso stat code
TerryMcGuinness-NOAA May 29, 2024
2adbcac
use wxflow on disk and added link_workflow and removed pip install wx…
TerryMcGuinness-NOAA May 29, 2024
d5f4103
misspelled link_workflow.sh
TerryMcGuinness-NOAA May 29, 2024
553404b
add link up wxflow because virt env was not current
TerryMcGuinness-NOAA May 29, 2024
05797cd
moved making of wxflow links into test section
TerryMcGuinness-NOAA May 29, 2024
2cc2fe2
removed pytest junit install
TerryMcGuinness-NOAA May 29, 2024
7407f4e
added PYTHONPATH for tests
TerryMcGuinness-NOAA May 29, 2024
a9de59d
added redundent wxflow as a pip install
TerryMcGuinness-NOAA May 29, 2024
1363144
no comma seperator in pip install
TerryMcGuinness-NOAA May 29, 2024
242f8b7
pip install wxflow from develop branch
TerryMcGuinness-NOAA May 29, 2024
bd7ca33
Merge branch 'develop' into ci_unit-tests
TerryMcGuinness-NOAA May 30, 2024
5dc1dc9
added unit test to test setup_expt and setup_xml
TerryMcGuinness-NOAA May 30, 2024
e81e260
added /work/noaa dir on running to make pass Host() as oroin
May 30, 2024
6fd980d
remove env from setup xml
TerryMcGuinness-NOAA May 30, 2024
62576d6
Merge branch 'ci_unit-tests' of github.com:TerrenceMcGuinness-NOAA/gl…
TerryMcGuinness-NOAA May 30, 2024
b3dff2c
pynorms and unset EXPDR in runner shell
TerryMcGuinness-NOAA May 30, 2024
c7809b4
added more pynorms
TerryMcGuinness-NOAA May 30, 2024
28278ba
shortend path to PSLOT
TerryMcGuinness-NOAA May 30, 2024
b20137b
removed push and made full paths to test data for tesgint setups
TerryMcGuinness-NOAA May 30, 2024
06ef59e
trying to remove EXPDIR from env
TerryMcGuinness-NOAA May 30, 2024
01d2295
trying to remove EXPDIR from runner env
TerryMcGuinness-NOAA May 30, 2024
848e220
added wxflow update to runner for test setups
TerryMcGuinness-NOAA May 30, 2024
2f7f1a2
Update ci/scripts/utils/rocotostat.py
TerrenceMcGuinness-NOAA May 30, 2024
f90eab1
Update ci/scripts/utils/rocotostat.py
TerrenceMcGuinness-NOAA May 30, 2024
f0c907a
use updated wxflow config in this test pr
TerryMcGuinness-NOAA May 30, 2024
c432eda
Merge branch 'ci_unit-tests' of github.com:TerrenceMcGuinness-NOAA/gl…
TerryMcGuinness-NOAA May 30, 2024
1aa135c
added ACCOUT to env for testing updates
TerryMcGuinness-NOAA May 30, 2024
cfc85d7
checklut output from setup exp in runner
TerryMcGuinness-NOAA May 30, 2024
d4d3933
echo EXPDIR and cat config.bas in runner
TerryMcGuinness-NOAA May 31, 2024
fc44f85
cleaned up path to RUNDIR data to PSLOT
TerryMcGuinness-NOAA May 31, 2024
bb76df3
made one function and added a pause just to see if it is ot on dixk y…
TerryMcGuinness-NOAA May 31, 2024
4132ee9
added debug outputs in setup xml
TerryMcGuinness-NOAA May 31, 2024
99f9ada
use develop wxflow
TerryMcGuinness-NOAA May 31, 2024
30b074d
narrowing in on issue that we are not getting base with my update
TerryMcGuinness-NOAA May 31, 2024
4100aad
more dubugging outputs
TerryMcGuinness-NOAA May 31, 2024
46d52bf
more dubugging outputs againZ
TerryMcGuinness-NOAA May 31, 2024
eba677e
test if EXPDIR is in base
TerryMcGuinness-NOAA May 31, 2024
e846040
added checkout of wxflow and running lit link on the runner
TerryMcGuinness-NOAA May 31, 2024
788ef8c
misspelled update in git command
TerryMcGuinness-NOAA May 31, 2024
b9cb187
needed ./ for runnign link in sorc
TerryMcGuinness-NOAA May 31, 2024
ada04dd
add checkout of gfs_utils for detect machine
TerryMcGuinness-NOAA May 31, 2024
2d01f6a
added extra ln for wxflow import in test dir
TerryMcGuinness-NOAA May 31, 2024
7ca0a7a
added the pip intall bit for wxflow from forked branch with updates f…
TerryMcGuinness-NOAA May 31, 2024
366c381
did a full init submodule in the runner
TerryMcGuinness-NOAA May 31, 2024
257c033
print config object in setup xml on running
TerryMcGuinness-NOAA May 31, 2024
3502963
revert to wxflow develop and pip install wxflow
TerryMcGuinness-NOAA May 31, 2024
205d367
mud hurling with python 3.7.16 and bash in runner
TerryMcGuinness-NOAA May 31, 2024
07f9c31
add directive to use bash in running tests in runner
TerryMcGuinness-NOAA May 31, 2024
8285280
comment out setup xml line
TerryMcGuinness-NOAA May 31, 2024
f45caca
remove extra clons and python leader to exp code
TerryMcGuinness-NOAA May 31, 2024
b457a9f
trying to submodule only wxflow and gsi_utils
TerryMcGuinness-NOAA May 31, 2024
e1b4907
added code for bash in config
TerryMcGuinness-NOAA May 31, 2024
a3ff0b6
updated pip clone from forked wxflow and removed some smack
TerryMcGuinness-NOAA May 31, 2024
5c726dd
restored setup xml
TerryMcGuinness-NOAA May 31, 2024
75f44b4
condenced the test themselfs
TerryMcGuinness-NOAA May 31, 2024
1cad3a2
just removed line for extra var to setup_expt
TerryMcGuinness-NOAA May 31, 2024
92d2208
updated HPC_ACCOUNT to ACCOUNT in bash config of ci platforms to test…
TerryMcGuinness-NOAA May 31, 2024
08b7b9f
Delete ci/scripts/tests/wxflow
TerrenceMcGuinness-NOAA May 31, 2024
e48fedd
Merge branch 'develop' into ci_unit-tests
TerryMcGuinness-NOAA Jun 3, 2024
65343f9
added addtional check to make sure the Configure object is getting po…
TerryMcGuinness-NOAA Jun 4, 2024
3c9bb58
added more details to testing Configuration updates and consolidation…
TerryMcGuinness-NOAA Jun 5, 2024
c7282dc
few shell norms
TerryMcGuinness-NOAA Jun 5, 2024
c1adce8
pynorm spaces in test codes
TerryMcGuinness-NOAA Jun 5, 2024
3e256dc
moved shell script that performs the export to the test code and adde…
TerryMcGuinness-NOAA Jun 5, 2024
55c28f4
single pynorm white space
TerryMcGuinness-NOAA Jun 5, 2024
d0bfa61
added HOMEgfs and checked for no UNKNOWN
TerryMcGuinness-NOAA Jun 5, 2024
df915d4
removed {} from script string
TerryMcGuinness-NOAA Jun 5, 2024
1387d78
removed gsi_utils from Runner
TerryMcGuinness-NOAA Jun 5, 2024
5da70e4
did not need detect machine in gsi utils
TerryMcGuinness-NOAA Jun 5, 2024
b7eeae8
udate wxflow updates to the PR
TerryMcGuinness-NOAA Jun 5, 2024
526e6f4
hardened the test to include the corner case when a user sets an envi…
TerryMcGuinness-NOAA Jun 6, 2024
494690d
update to latest update of configuration.py in wxflow (using grep for…
TerryMcGuinness-NOAA Jun 6, 2024
d30d11f
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA Jun 11, 2024
4e882f7
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA Jun 24, 2024
fc15469
added fix to rocotostat for stalled case and remove dependances to up…
TerryMcGuinness-NOAA Jun 24, 2024
892ece9
update to current gitmodules without wxflow updates
TerryMcGuinness-NOAA Jun 24, 2024
236e5f6
update to default wxflow
TerryMcGuinness-NOAA Jun 24, 2024
f553a33
needed pynorms space after :
TerryMcGuinness-NOAA Jun 24, 2024
9d329bb
still need to do the pip install of wxflow to get all the Jinja stuff…
TerryMcGuinness-NOAA Jun 24, 2024
371d532
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA Jun 25, 2024
8b527c8
Update ci/scripts/tests/test_setup.py
TerrenceMcGuinness-NOAA Jun 25, 2024
acc8d3c
Update ci/scripts/tests/test_setup.py
TerrenceMcGuinness-NOAA Jun 25, 2024
6bcc3b5
updated to python wget and renamed github action control file
TerryMcGuinness-NOAA Jun 25, 2024
8d638ee
removed other yaml action file
TerryMcGuinness-NOAA Jun 25, 2024
fd4aa91
added mkdir -p to full path to testdata
TerryMcGuinness-NOAA Jun 25, 2024
fab6843
updated python install in runner to 3.13.0
TerryMcGuinness-NOAA Jun 25, 2024
c511207
updated python install in runner to 3.12.3 for x64 Ubu
TerryMcGuinness-NOAA Jun 25, 2024
c25dda8
updated pytest to version 8.2.2
TerryMcGuinness-NOAA Jun 25, 2024
6d3a462
removed verson of pytest got get latest for pythohn 3.13.3
TerryMcGuinness-NOAA Jun 25, 2024
c52bfc2
removed verson of wxflow
TerryMcGuinness-NOAA Jun 25, 2024
dd5e58c
downgraded to python 3.11.8 because distutils was depretiated from 3.12
TerryMcGuinness-NOAA Jun 25, 2024
01bc144
took out shell wrapper that was exporting env vars
TerryMcGuinness-NOAA Jun 25, 2024
f59819d
added extra subdir for pysdo hera file system
TerryMcGuinness-NOAA Jun 25, 2024
7102f0d
investigating EXPDIR
TerryMcGuinness-NOAA Jun 25, 2024
7c14d55
investigating EXPDIR
TerryMcGuinness-NOAA Jun 25, 2024
1f599e3
still trying to debug EXPDIR key error
TerryMcGuinness-NOAA Jun 25, 2024
d2db0be
still trying to debug EXPDIR key error still
TerryMcGuinness-NOAA Jun 25, 2024
7965a9b
adding in the other tests
TerryMcGuinness-NOAA Jun 25, 2024
e18ee96
adding in the other tests crap
TerryMcGuinness-NOAA Jun 25, 2024
8b8fe34
outputing EXPDIR before running Config in setup xml
TerryMcGuinness-NOAA Jun 25, 2024
2b3a122
outputing EXPDIR before running Config in setup xml 2
TerryMcGuinness-NOAA Jun 25, 2024
f6d5dd6
tring env unclaching wxflow repo to see if it is solving the EXPDIR p…
TerryMcGuinness-NOAA Jun 25, 2024
69b226e
try test with updated wxflow (if works then get to that fix)
TerryMcGuinness-NOAA Jun 25, 2024
5300dc2
added corner case for checking expected fail for when name snapces ar…
TerryMcGuinness-NOAA Jun 26, 2024
c5b6386
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA Jun 26, 2024
f513eeb
Merge branch 'NOAA-EMC:develop' into ci_unit-tests
TerrenceMcGuinness-NOAA Jun 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .github/workflows/ci-unit_tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: CI Unit Tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Help me understand why mixing - and _ in filenames?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Help you understand? OK: CI Unit Tests, does that help?

on:
pull_request:
TerrenceMcGuinness-NOAA marked this conversation as resolved.
Show resolved Hide resolved

jobs:

ci_pytest:
runs-on: ubuntu-latest
TerrenceMcGuinness-NOAA marked this conversation as resolved.
Show resolved Hide resolved
permissions:
checks: write
pull-requests: write
TerrenceMcGuinness-NOAA marked this conversation as resolved.
Show resolved Hide resolved

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.8

- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y perl libxml-libxml-perl libxml-libxslt-perl libdatetime-perl
python -m pip install --upgrade pip
pip install pytest wxflow
# If you have a requirements.txt file, you can install all dependencies using:
# pip install -r requirements.txt

- name: Install Rocoto
run: |
git clone https://github.com/christopherwharrop/rocoto.git
cd rocoto
./INSTALL
echo "$(pwd)/bin" >> $GITHUB_PATH
TerrenceMcGuinness-NOAA marked this conversation as resolved.
Show resolved Hide resolved

- name: Run tests
run: python -m pytest ci/scripts/tests/test_rocotostat.py -v --junitxml ci/scripts/tests/test-results.xml

- name: Publish Test Results
if: always()
uses: EnricoMi/publish-unit-test-result-action@v2
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved
with:
files: ci/scripts/tests/test-results.xml
job_summary: true
comment_mode: off

Binary file added ci/scripts/tests/database.db
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should not be added to the repo. It is a binary file and we do not have git-lfs resources.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The corresponding database files (now in specific test directories) are in final states so no calls to schedulers are being made by the unit tests.

Binary file not shown.
37 changes: 37 additions & 0 deletions ci/scripts/tests/test_rocotostat.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import sys
import os

script_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.join(os.path.dirname(script_dir), 'utils'))

from rocotostat import rocoto_statcount, rocotostat_summary, CommandNotFoundError
from wxflow import which

workflow_file = os.path.join(script_dir, "workflow.xml")
database_file = os.path.join(script_dir, "database.db")

try:
rocotostat = which("rocotostat")
except CommandNotFoundError:
raise CommandNotFoundError("rocotostat not found in PATH")
rocotostat.add_default_arg(['-w', workflow_file, '-d', database_file])


def test_rocoto_statcount():

result = rocoto_statcount(rocotostat)

assert result['SUCCEEDED'] == 20
assert result['FAIL'] == 0
assert result['DEAD'] == 0
assert result['RUNNING'] == 0
assert result['SUBMITTING'] == 0
assert result['QUEUED'] == 0
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved


def test_rocoto_summary():

result = rocotostat_summary(rocotostat)

assert result['CYCLES_TOTAL'] == 1
assert result['CYCLES_DONE'] == 1
284 changes: 284 additions & 0 deletions ci/scripts/tests/workflow.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,284 @@
<?xml version="1.0"?>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to commit this file? Especially since the comment says, the workflow was automatically generated at ....

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generalized XML files (no proper names) and tested on neutral laptop.
Moved test data into distinct folders for specific test cases.

<!DOCTYPE workflow
[
<!--
PROGRAM
Main workflow manager for Global Forecast System

NOTES:
This workflow was automatically generated at 2024-05-01 14:24:16.515689
-->
<!ENTITY PSLOT "C48_ATM_0ad155f1">
<!ENTITY ROTDIR "/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not going to be static, another reason why this file does not belong in the repo.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed all proper names and are used only with databases that are "static" in that they are in final states that are being tested agains with known values.

<!ENTITY JOBS_DIR "/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto">
<!ENTITY MAXTRIES "2">
]>

<workflow realtime="F" scheduler="slurm" cyclethrottle="3" taskthrottle="25">

<log verbosity="10"><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1/logs/@Y@m@[email protected]</cyclestr></log>

<!-- Define the cycles -->
<cycledef group="gfs">202103231200 202103231200 24:00:00</cycledef>

<task name="gfsstage_ic" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/stage_ic.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfsstage_ic_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>orion</partition>
<walltime>00:15:00</walltime>
<nodes>1:ppn=1:tpp=1</nodes>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfsstage_ic.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>

<dependency>
<and>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_ctrl.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_data.tile1.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_data.tile2.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_data.tile3.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_data.tile4.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_data.tile5.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/gfs_data.tile6.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/sfc_data.tile1.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/sfc_data.tile2.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/sfc_data.tile3.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/sfc_data.tile4.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/sfc_data.tile5.nc</cyclestr></datadep>
<datadep><cyclestr>/work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/@Y@m@d@H/atmos/sfc_data.tile6.nc</cyclestr></datadep>
</and>
</dependency>

</task>

<task name="gfsfcst" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/fcst.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfsfcst_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>orion</partition>
<walltime>03:00:00</walltime>
<nodes>1:ppn=40:tpp=1</nodes>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfsfcst.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>

<dependency>
<and>
<taskdep task="gfsstage_ic"/>
</and>
</dependency>

</task>

<metatask name="gfsatmos_prod">

<var name="grp">_f000-f006 _f009-f015 _f018-f024 _f027-f033 _f036-f042 _f045-f051 _f054-f060 _f063-f069 _f072-f078 _f081-f087 _f090-f096 _f099-f105 _f108-f114 _f117-f120</var>
<var name="dep">f006 f015 f024 f033 f042 f051 f060 f069 f078 f087 f096 f105 f114 f120</var>
<var name="lst">f000_f003_f006 f009_f012_f015 f018_f021_f024 f027_f030_f033 f036_f039_f042 f045_f048_f051 f054_f057_f060 f063_f066_f069 f072_f075_f078 f081_f084_f087 f090_f093_f096 f099_f102_f105 f108_f111_f114 f117_f120</var>

<task name="gfsatmos_prod#grp#" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/atmos_products.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfsatmos_prod#grp#_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>orion</partition>
<walltime>00:15:00</walltime>
<nodes>1:ppn=24:tpp=1</nodes>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfsatmos_prod#grp#.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>
<envar><name>FHRLST</name><value>#lst#</value></envar>
<envar><name>COMPONENT</name><value>atmos</value></envar>

<dependency>
<datadep age="120"><cyclestr>&ROTDIR;/gfs.@Y@m@d/@H//model_data/atmos/master/[email protected]#dep#</cyclestr></datadep>
</dependency>

</task>

</metatask>

<task name="gfstracker" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/tracker.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfstracker_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>orion</partition>
<walltime>00:10:00</walltime>
<nodes>1:ppn=1:tpp=1</nodes>
<memory>4G</memory>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfstracker.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>

<dependency>
<metataskdep metatask="gfsatmos_prod"/>
</dependency>

</task>

<task name="gfsgenesis" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/genesis.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfsgenesis_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>orion</partition>
<walltime>00:25:00</walltime>
<nodes>1:ppn=1:tpp=1</nodes>
<memory>10G</memory>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfsgenesis.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>

<dependency>
<metataskdep metatask="gfsatmos_prod"/>
</dependency>

</task>

<task name="gfsarch" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/arch.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfsarch_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>service</partition>
<walltime>06:00:00</walltime>
<nodes>1:ppn=1:tpp=1</nodes>
<memory>4096M</memory>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfsarch.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>

<dependency>
<and>
<taskdep task="gfstracker"/>
<taskdep task="gfsgenesis"/>
<metataskdep metatask="gfsatmos_prod"/>
</and>
</dependency>

</task>

<task name="gfscleanup" cycledefs="gfs" maxtries="&MAXTRIES;">

<command>/work2/noaa/stmp/CI/ORION/2555/gfs/jobs/rocoto/cleanup.sh</command>

<jobname><cyclestr>C48_ATM_0ad155f1_gfscleanup_@H</cyclestr></jobname>
<account>nems</account>
<queue>batch</queue>
<partition>orion</partition>
<walltime>00:15:00</walltime>
<nodes>1:ppn=1:tpp=1</nodes>
<memory>4096M</memory>
<native>--export=NONE</native>

<join><cyclestr>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT/C48_ATM_0ad155f1/logs/@Y@m@d@H/gfscleanup.log</cyclestr></join>

<envar><name>RUN_ENVIR</name><value>emc</value></envar>
<envar><name>HOMEgfs</name><value>/work2/noaa/stmp/CI/ORION/2555/gfs</value></envar>
<envar><name>EXPDIR</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/EXPDIR/C48_ATM_0ad155f1</value></envar>
<envar><name>NET</name><value>gfs</value></envar>
<envar><name>CDUMP</name><value>gfs</value></envar>
<envar><name>RUN</name><value>gfs</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
<envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
<envar><name>cyc</name><value><cyclestr>@H</cyclestr></value></envar>
<envar><name>COMROOT</name><value>/work2/noaa/stmp/CI/ORION/2555/RUNTESTS/COMROOT</value></envar>
<envar><name>DATAROOT</name><value>/work/noaa/stmp/mterry/RUNDIRS/C48_ATM_0ad155f1</value></envar>

<dependency>
<and>
<taskdep task="gfsarch"/>
</and>
</dependency>

</task>

</workflow>
Loading
Loading