SFS Runs at C96mx100 #2960

NeilBarton-NOAA · 2024-09-25T19:18:25Z

Description

Adds the ability to run C96mx100 with SFS options. Included in this PR is the ability to use the interpolated/coarse grained 1 degree MOM6 ICs. Staging the ATM and OCN perturbation files without using the +03 replay ICs.

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

This has been tested in the gefs forecast-only configurations on hera and hercules.

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
I have made corresponding changes to the system documentation if necessary

changes. Lines starting

…lobal-workflow into gefs_replay_ci

Fix a bug in a variable check in the extractvars task where parenthesis in the variable name will prompt a false warning message.

Ice variables are extracted from the history files instead of the product files. This is necessary because there are certain ice variables that are needed that are only available in the history files and not the product files.

Add suggested patches to gefs_replay_ci

Shellcheck disable has been added. Also, f000 is skipped in extractvars for ocn and ice since there are no products for f000 for ocn and ice. This change will remove warning messages.

into SFS

lisa-bengtsson · 2024-10-16T19:48:54Z

parm/config/gefs/config.ufs

 export nthreads_fv3=1
 export nthreads_fv3_gfs=1
 export nthreads_ufs=1
 export nthreads_ufs_gfs=1
- export xr_cnvcld=".false." # Do not pass conv. clouds to Xu-Randall cloud fraction
+ export xr_cnvcld=".true." # Pass conv. clouds to Xu-Randall cloud fraction


@NeilBarton-NOAA we could perhaps just remove this xr_cnvcld variable from the workflow to avoid any confusion. It is default .true. in the code, and when we wanted it to be false for C96 only it made sense to keep it here. But now when we moved it back to true, and instead changed the scaling in the code - it is probably cleaner to just remove.

aerorahul · 2024-10-16T19:53:05Z

@pjpegion has proposed a system=sfs in addition to gfs|gefs variants.
@WalterKolczynski-NOAA we should ensure that it is brought in sooner rather than later, before conditionals for SFS get if-ed into the RUN=gefs code-blocks.
@JacobCarley-NOAA FYI

aerorahul · 2024-10-16T19:53:46Z

ci/cases/sfs/C96mx100_S2S.yaml

@@ -0,0 +1,20 @@
+experiment:
+ system: gefs


This is now a SFS test using gefs system.

The SFS_POST=YES is currently a proxy to system=sfs and will need to be edited.

aerorahul · 2024-10-16T19:59:52Z

sorc/build_all.sh

@@ -37,6 +37,8 @@ Usage: ${BASH_SOURCE[0]} [-a UFS_app][-c build_config][-d][-f][-h][-j n][-v][-w]
 Execute all build scripts with -v option to turn on verbose where supported
 -w:
 Use structured wave grid
+ -y:
+ Use hystrostatic version of FV3


Suggested change

Use hystrostatic version of FV3

Use hydrostatic version of FV3

aerorahul

How is this actually tested? It appears that the model has to be built with a hydrostatic option -y but that is missing from build_ufs.sh.
Also, this requires the CI system to build the ufs-weather-model in 3 configurations; unstructured waves for GFS, structured waves for GEFS (-w) and hydrostatic dynamical code for SFS (-y).
I think we need to re-consider the CI system where we are cloning 2 copies of global-workflow one each for GFS and GEFS, and now this third one for SFS.
Instead, we could refactor the current CI system to build 3 executables with a suffix of .gfs.x, .gefs.x, .sfs.x or gfs_model.x, gefs_model.x, sfs_model.x for the variants of the model options and still use the same HOMEgfs.

This will require some prep-work in the global-workflow.

aerorahul · 2024-10-16T20:00:27Z

sorc/build_all.sh

@@ -129,7 +133,7 @@ declare -A build_opts
 big_jobs=0
 build_jobs["ufs"]=8
 big_jobs=$((big_jobs+1))
-build_opts["ufs"]="${_wave_opt} ${_verbose_opt} ${_build_ufs_opt} ${_build_debug}"
+build_opts["ufs"]="${_wave_opt} ${_hydro_opt} ${_verbose_opt} ${_build_ufs_opt} ${_build_debug}"


This requires build_ufs.sh to be updated as well to make use of the -y.

@aerorahul I didn't catch an error with the merge/commit. The updated build_all.sh and other files are now uploaded

aerorahul · 2024-10-16T20:07:00Z

@WalterKolczynski-NOAA @DavidHuber-NOAA Please see this comment and provide your thoughts.

WalterKolczynski-NOAA · 2024-10-16T20:21:16Z

How is this actually tested? It appears that the model has to be built with a hydrostatic option -y but that is missing from build_ufs.sh. Also, this requires the CI system to build the ufs-weather-model in 3 configurations; unstructured waves for GFS, structured waves for GEFS (-w) and hydrostatic dynamical code for SFS (-y). I think we need to re-consider the CI system where we are cloning 2 copies of global-workflow one each for GFS and GEFS, and now this third one for SFS. Instead, we could refactor the current CI system to build 3 executables with a suffix of .gfs.x, .gefs.x, .sfs.x or gfs_model.x, gefs_model.x, sfs_model.x for the variants of the model options and still use the same HOMEgfs.

This will require some prep-work in the global-workflow.

I agree. We've already talked about doing this for the two existing variations, though it hasn't been a priority as yet. I'd like to name the executables by options rather than system, since there is nothing inherent preventing GEFS from running with an unstructured wave grid, for example. But that also may get cumbersome if we accumulate too many different build options.

I can roll this into my existing compute-build effort (indeed was already considering doing so).

NeilBarton-NOAA · 2024-10-17T18:24:00Z

The GEFS and GFS coupled forecasts CIs are successful with the latest commits. The pynorm error is unclear to me. @WalterKolczynski-NOAA any suggestions on the pynorm error?

WalterKolczynski-NOAA · 2024-10-17T18:26:47Z

The GEFS and GFS coupled forecasts CIs are successful with the latest commits. The pynorm error is unclear to me. @WalterKolczynski-NOAA any suggestions on the pynorm error?

If you click on 'details' it will take you to the pynorms log. In this case, it is just one thing:

checking ./scripts/exglobal_stage_ic.py
./scripts/exglobal_stage_ic.py:26:40: E231 missing whitespace after ','

WalterKolczynski-NOAA · 2024-10-17T18:28:35Z

ci/cases/sfs/C96mx100_S2S.yaml

This test should be disabled on wcoss2 like the other gefs tests until we change the build system to build different executables.

WalterKolczynski-NOAA · 2024-10-17T18:32:44Z

parm/config/gefs/config.efcs

@@ -39,7 +39,7 @@ if [[ "${RUN}" == "enkfgfs" ]]; then
 fi

 # Stochastic physics parameters (only for ensemble forecasts)
-export DO_SKEB="YES"
+export DO_SKEB=${DO_SKEB:-"YES"} # SKEB turned off for C96


I don't think this is going to work, because DO_SKEB is set in config.fcst first, so the default here will never be used.

DO_SKEB="NO" for C96 in config.ufs which is being propagated to the namelist. See log on hercules at /home/nbarton/GW/EXPDIR/SFS_C96mx100_S2S/LOGS_COMROOT/1994050100/fcst_mem001_seg0.log

WalterKolczynski-NOAA · 2024-10-17T19:07:09Z

parm/config/gefs/config.fcst

@@ -43,6 +43,7 @@ export FHOUT=${FHOUT_GFS}
 export FHOUT_HF=${FHOUT_HF_GFS}
 export FHOUT_OCN=${FHOUT_OCN_GFS}
 export FHOUT_ICE=${FHOUT_ICE_GFS}
+export FHZER=@FHZER@


This variable is not used.

? The variable is being propagated to the input.nml file in my test runs. see hercules at /home/nbarton/GW/EXPDIR/SFS_C96mx100_S2S/LOGS_COMROOT/1994050100/fcst_mem001_seg0.log

WalterKolczynski-NOAA · 2024-10-17T19:08:48Z

parm/config/gefs/yaml/defaults.yaml

 FHOUT_OCN_GFS: 6
 FHOUT_ICE_GFS: 6
 HPSSARCH: "NO"
 LOCALARCH: "NO"
+ SFS_POST: "NO"
+fcst:
+ reforecast: "YES"


The default should be "NO" and turned on for the tests that need it.

I changed the default to NO

WalterKolczynski-NOAA · 2024-10-17T19:10:59Z

parm/config/gefs/yaml/defaults.yaml

+ USE_OCN_ENS_PERTURB_FILES: "YES"
+ USE_ATM_ENS_PERTURB_FILES: "YES"


Do we really want to default to using these?

I changed the default to NO

sorc/build_ufs.sh

WalterKolczynski-NOAA · 2024-10-17T19:22:27Z

parm/config/gfs/config.ocn

+MOM6_INTERP_ICS=@MOM6_INTERP_ICS@
+if [[ "${MOM6_INTERP_ICS}" == "YES" ]]; then
+ export MOM6_RESTART_SETTING='n'
+ export MOM6_WARMSTART_FILE="MOM.res.nc"
+ export MOM6_INIT_FROM_Z='False'
+ export MOM6_INIT_UV='file'
+fi


Don't we need a similar block in gefs/config.ocn?

Or maybe these should be set elsewhere in the run scripts based on $MOM6_INTERP_ICS

gefs/config.ocn is currently a soft link from gfs/config.ocn. i.e., each is set. If preferred to be set someplace else, I'm happy to edit

ush/forecast_predet.sh

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

NeilBarton-NOAA and others added 30 commits July 2, 2024 15:04

initial creation of CI using replay ICs

30cf5df

change name

4b3ae39

making an S2SWA case

fe2054e

Merge remote-tracking branch 'origin/develop' into gefs_replay_ci

7b7ee9b

Merge remote-tracking branch 'origin/develop' into gefs_replay_ci

225d280

start of CI for GEFS replays # Please enter the commit message for your

b17765e

changes. Lines starting

Merge branch 'develop' into gefs_replay_ci

675cb77

removing shell check errors

e49dbcf

Merge branch 'gefs_replay_ci' of https://github.com/NeilBarton-NOAA/g…

ec4becc

…lobal-workflow into gefs_replay_ci

fixes for wave post processing

b69d653

Merge branch 'NOAA-EMC:develop' into gefs_replay_ci

084d744

Merge remote-tracking branch 'origin/develop' into gefs_replay_ci

a4bb8f3

add BREAKPOINTS variable

4ff31fd

fix shell checks

97b92ee

edit defult config.efc to be close to re-forecasting

713f41d

Fix bug in variable check

af31d2f

Fix a bug in a variable check in the extractvars task where parenthesis in the variable name will prompt a false warning message.

Add more settings to GEFS replay CI test

cec7b81

Add atmos variables requested for reforeacst

4aaca9a

Reorganize TSNOWP in paramlist

11e1957

Extract ice variables from history file instead

a005e5f

Ice variables are extracted from the history files instead of the product files. This is necessary because there are certain ice variables that are needed that are only available in the history files and not the product files.

Fix typo

2dd9a7e

Update TCDC variable to correct name

b132408

Merge pull request #7 from EricSinsky-NOAA/gefs_replay_ci_pr

3d9455b

Add suggested patches to gefs_replay_ci

Merge branch 'NOAA-EMC:develop' into gefs_replay_ci

447224c

edits to stage_ic for replay perturbations

c6b7eaa

adding icsdir to pr CI yaml

51aa2dd

Merge branch 'NOAA-EMC:develop' into gefs_replay_ci

a641591

Fix ensstat failures

4985361

updates for products

8eaef27

Add shellcheck disable and skip f000 for ocn/ice

2e7241f

Shellcheck disable has been added. Also, f000 is skipped in extractvars for ocn and ice since there are no products for f000 for ocn and ice. This change will remove warning messages.

Merge branch 'develop' into SFS

67272c3

NeilBarton-NOAA marked this pull request as ready for review October 15, 2024 18:15

NeilBarton-NOAA requested review from WalterKolczynski-NOAA and aerorahul October 15, 2024 18:17

NeilBarton-NOAA and others added 4 commits October 15, 2024 17:31

Merge branch 'develop' into SFS

ff3bf91

Merge branch 'SFS' of https://github.com/NeilBarton-NOAA/global-workflow

ceeea0b

into SFS

Merge branch 'SFS' of https://github.com/NeilBarton-NOAA/global-workflow

d08d66a

into SFS

add changes for UPP PR

d64111d

lisa-bengtsson reviewed Oct 16, 2024

View reviewed changes

lisa-bengtsson approved these changes Oct 16, 2024

View reviewed changes

aerorahul reviewed Oct 16, 2024

View reviewed changes

NeilBarton-NOAA added 3 commits October 16, 2024 16:23

merge local changes

2b97e38

updates for ci tests

035cf24

add defaults for gfs runs

8acc70a

pynorm fix

5be5a75

WalterKolczynski-NOAA requested changes Oct 17, 2024

View reviewed changes

NeilBarton-NOAA and others added 4 commits October 17, 2024 16:37

Update ush/forecast_predet.sh

e2587ae

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ush/forecast_predet.sh

cfa1543

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ush/forecast_predet.sh

35ee836

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

edits for gefs defaults

50c5b5e

NeilBarton-NOAA requested review from KateFriedman-NOAA, DavidHuber-NOAA and TerrenceMcGuinness-NOAA as code owners October 17, 2024 20:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SFS Runs at C96mx100 #2960

SFS Runs at C96mx100 #2960

NeilBarton-NOAA commented Sep 25, 2024

lisa-bengtsson Oct 16, 2024

aerorahul commented Oct 16, 2024

aerorahul Oct 16, 2024

NeilBarton-NOAA Oct 16, 2024

aerorahul Oct 16, 2024

aerorahul left a comment

aerorahul Oct 16, 2024

NeilBarton-NOAA Oct 16, 2024 •

edited

Loading

aerorahul commented Oct 16, 2024

WalterKolczynski-NOAA commented Oct 16, 2024

NeilBarton-NOAA commented Oct 17, 2024

WalterKolczynski-NOAA commented Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

NeilBarton-NOAA Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

NeilBarton-NOAA Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

NeilBarton-NOAA Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

NeilBarton-NOAA Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

WalterKolczynski-NOAA Oct 17, 2024

NeilBarton-NOAA Oct 17, 2024

	Use hystrostatic version of FV3
	Use hydrostatic version of FV3

		USE_OCN_ENS_PERTURB_FILES: "YES"
		USE_ATM_ENS_PERTURB_FILES: "YES"

SFS Runs at C96mx100 #2960

Are you sure you want to change the base?

SFS Runs at C96mx100 #2960

Conversation

NeilBarton-NOAA commented Sep 25, 2024

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Choose a reason for hiding this comment

aerorahul commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NeilBarton-NOAA Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

aerorahul commented Oct 16, 2024

WalterKolczynski-NOAA commented Oct 16, 2024

NeilBarton-NOAA commented Oct 17, 2024

WalterKolczynski-NOAA commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NeilBarton-NOAA Oct 16, 2024 •

edited

Loading