Move the variational scripts to this repository #2920

guillaumevernieres · 2024-09-13T18:12:37Z

Description

This PR "moves and refactors" the variational DA exscripts that were in the GDASapp to this repository. The ens. var. feature will be replicated/moved in a subsequent PR.

Waiting for 2 companion PR's:

ci testing: We should be able to turn on the WCDA test on Hercules

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

The latest tests were done on Hecules, earlier ones on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

Use it to define location of experimental_obs. Refs NOAA-EMC#2683

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT

ush/python/pygfs/task/marine_bmat.py

ci/cases/gfsv17/ocnanal.yaml

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT

ush/python/pygfs/task/marine_bmat.py

ush/python/pygfs/utils/marine_da_utils.py

DavidNew-NOAA

Looks good to me. No comments beyond what others have already said.

ush/python/pygfs/utils/marine_da_utils.py

ush/python/pygfs/task/marine_analysis.py

…al_obs_path * origin/develop: Update global atmos upp job to use COMIN/COMOUT (NOAA-EMC#2867) Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917) Cleanup job for GEFS (NOAA-EMC#2919) Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912) add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922) Support gefs C48 on Azure (NOAA-EMC#2881) Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)

aerorahul · 2024-09-23T18:24:48Z

ush/python/pygfs/utils/marine_da_utils.py

+ hist_date = dparser.parse(ncf.variables['time'].units, fuzzy=True) + timedelta(hours=int(ncf.variables['time'][0]))
+ ncf.close()
+ logger.info(f"*** history file date: {hist_date} expected date: {ref_date}")
+ assert hist_date == ref_date, 'Inconsistent bkg date'


raise ValueError if there is a mismatch. assert suggested use is in tests and should not be used in production code.
Use FATAL ERROR language when raising errors.

aerorahul · 2024-09-23T18:25:22Z

ush/python/pygfs/utils/marine_da_utils.py

+ f = open(yaml_name, 'w')
+ yaml.dump(bkg_list, f, sort_keys=False, default_flow_style=False)


Use save_as_yaml

aerorahul

I am still combing through this PR and commenting as find time and concrete comments. Apologies for the time, this is a big PR

KateFriedman-NOAA · 2024-09-23T18:34:06Z

sorc/link_workflow.sh

+case "${machine}" in
+ "wcoss2") EXPOBS_DIR="" ;;
+ "hera") EXPOBS_DIR="/scratch1/NCEPDEV/global/glopara/data/experimental_obs" ;;
+ "orion") EXPOBS_DIR="/work/noaa/global/glopara/data/experimental_obs" ;;
+ "hercules") EXPOBS_DIR="/work/noaa/global/glopara/data/experimental_obs" ;;
+ "jet") EXPOBS_DIR="" ;;
+ "s4") EXPOBS_DIR="" ;;
+ "gaea") EXPOBS_DIR="" ;;
+ "noaacloud") EXPOBS_DIR="" ;;
+ *)
+ echo "FATAL: Unknown target machine ${machine}, couldn't set EXPOBS_DIR"
+ exit 1
+ ;;
+esac


I have a fork branch with similar but differently done updates to handle the path to the experimental_obs folder. Please see the following diff and consider including my changes instead of the update you have in sorc/link_workflow.sh and DMPDIR in parm/config/gfs/yaml/defaults.yaml:
develop...KateFriedman-NOAA:global-workflow:feature/experimental_obs_path
I'm happy to send a PR into the branch for this PR if you are good with the updates. My updates will resolve #2683.

A pr into my branch sounds good to me @KateFriedman-NOAA , thanks.

Great, please see guillaumevernieres#8. Thanks!

DavidHuber-NOAA · 2024-09-23T19:05:58Z

ush/python/pygfs/task/marine_analysis.py

+ with open('obs_list_short.yaml', 'w') as file:
+ yaml.dump(parse_obs_list_file(self.task_config.MARINE_OBS_LIST_YAML), file, default_flow_style=False)


Prefer using the wxflow save_as_yaml method. See, e.g.

global-workflow/ush/python/pygfs/task/atm_analysis.py

Line 104 in ec63492

save_as_yaml(self.jedi.config, self.jedi.yaml)

ush/python/pygfs/task/marine_analysis.py

…nto feature/marinevar * origin/feature/experimental_obs_path: Introduce BASE_DATA variable

Introduce `BASE_DATA` and update `DMPDIR` for experimental obs

…eres/global-workflow into feature/marinevar

guillaumevernieres · 2024-09-24T20:39:28Z

Thanks for the recent reviews @aerorahul , @DavidHuber-NOAA and @KateFriedman-NOAA . The comments that I have not addressed in this PR are linked to github issues.
I retested manually on my side but the g-w ci probably needs to be triggered again.

RussTreadon-NOAA · 2024-09-26T01:14:34Z

Hera g-w CI

Install guillaumevernieres:feature/marinevar at 8002a06 on Hera in /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/pr2920.

Run g-w CI for the following configurations with the following results:

C96C48_hybatmDA

Hera(hfe02):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prgsi$ rocotostat -d prgsi.db -w prgsi.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Sep 25 2024 13:28:31    Sep 25 2024 14:20:18
202112210000        Done    Sep 25 2024 13:28:31    Sep 25 2024 22:10:22
202112210600        Done    Sep 25 2024 13:28:31    Sep 25 2024 22:50:18

C96C48_ufs_hybatmDA

Hera(hfe02):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prjedi$ rocotostat -d prjedi.db -w prjedi.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Sep 25 2024 13:28:41    Sep 25 2024 14:25:24
202402240000        Done    Sep 25 2024 13:28:41    Sep 26 2024 00:00:56

C96_atmaerosnowDA

Hera(hfe02):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/praero$ rocotostat -d praero.db -w praero.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Sep 25 2024 13:28:22    Sep 25 2024 14:25:20
202112201800        Done    Sep 25 2024 13:28:22    Sep 25 2024 20:00:28
202112210000        Done    Sep 25 2024 13:28:22    Sep 26 2024 00:25:21

C48mx500_3DVarAOWCDA

Hera(hfe02):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prwcda$ rocotostat -d prwcda.db -w prwcda.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241200        Done    Sep 25 2024 13:28:52    Sep 25 2024 14:25:22
202103241800        Done    Sep 25 2024 13:28:52    Sep 25 2024 19:50:34

All jobs successfully run to completion in each of the four tested g-w CI configurations.

RussTreadon-NOAA · 2024-09-26T10:06:29Z

@WalterKolczynski-NOAA : When can this PR be merged into develop?

Once this PR is merged into develop, PR #2875 can be updated with the post #2920 g-w develop and PR #2875 g-w CI will pass.

Alternatively, PR #2875 can be into develop before PR #2920, but we need to accept the fact that C48mx500_3DVarAOWCDA will fail until PR #2920 is merged into develop.

RussTreadon-NOAA

g-w CI successfully tested in several configurations on Hera (see results here).

Approve.

aerorahul · 2024-09-26T15:03:15Z

Since we have approval from @RussTreadon-NOAA (running tests and validating that they run to completion successfully), and issues have been opened to address reviewer comments in this PR to be addressed later, we can merge this PR.

issues have been opened to address reviewer comments to be addressed at a future time.

guillaumevernieres and others added 9 commits August 28, 2024 09:11

...

c9e41b1

var init ok, run needs tlc

53a4de8

updated checkpoint task

ba78b00

Introduce BASE_DATA variable

0dd87f0

Use it to define location of experimental_obs. Refs NOAA-EMC#2683

working finalize

4509b0c

Merge branch 'develop' into feature/marinevar

2b667f2

Merge branch 'NOAA-EMC:develop' into feature/experimental_obs_path

eb717a1

...

317e351

removed envar test yaml

ea6ae91

guillaumevernieres requested review from aerorahul, WalterKolczynski-NOAA and AndrewEichmann-NOAA September 13, 2024 18:12

github-advanced-security bot found potential problems Sep 13, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT Fixed Show fixed Hide fixed

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT Fixed Show fixed Hide fixed

guillaumevernieres added 4 commits September 13, 2024 13:30

norms, try 1

5c7eb42

Merge branch 'develop' into feature/marinevar

b4d46aa

norms, try 2

3b15dc6

norms, try 3

90ad1d3

guillaumevernieres mentioned this pull request Sep 13, 2024

Move the variational scripts to the g-w NOAA-EMC/GDASApp#1282

Merged

AndrewEichmann-NOAA reviewed Sep 13, 2024

View reviewed changes

aerorahul requested a review from DavidNew-NOAA September 14, 2024 14:44

guillaumevernieres added 2 commits September 16, 2024 17:54

addressed review comments

f1d7f8b

Merge branch 'develop' into feature/marinevar

f84ae82

DavidNew-NOAA approved these changes Sep 17, 2024

View reviewed changes

aerorahul reviewed Sep 17, 2024

View reviewed changes

ush/python/pygfs/utils/marine_da_utils.py Outdated Show resolved Hide resolved

aerorahul reviewed Sep 17, 2024

View reviewed changes

ush/python/pygfs/task/marine_analysis.py Show resolved Hide resolved

DavidNew-NOAA mentioned this pull request Sep 17, 2024

Incorporate JEDI class into marine jobs #2931

Open

KateFriedman-NOAA and others added 2 commits September 17, 2024 18:29

Merge branch 'NOAA-EMC:develop' into feature/experimental_obs_path

8981d9f

guillaumevernieres mentioned this pull request Sep 18, 2024

Use jinja template for the obs list NOAA-EMC/GDASApp#1285

Open

module tidy

9e8bbad

aerorahul reviewed Sep 23, 2024

View reviewed changes

aerorahul previously requested changes Sep 23, 2024

View reviewed changes

KateFriedman-NOAA reviewed Sep 23, 2024

View reviewed changes

DavidHuber-NOAA reviewed Sep 23, 2024

View reviewed changes

This was referenced Sep 23, 2024

Use jinja for the Marine obs list NOAA-EMC/GDASApp#1295

Open

Revisit the use of JCB in the marine_analysis module NOAA-EMC/GDASApp#1296

Open

DavidHuber-NOAA reviewed Sep 23, 2024

View reviewed changes

ush/python/pygfs/task/marine_analysis.py Show resolved Hide resolved

Merge remote-tracking branch 'origin/feature/experimental_obs_path' i…

0d4456e

…nto feature/marinevar * origin/feature/experimental_obs_path: Introduce BASE_DATA variable

KateFriedman-NOAA mentioned this pull request Sep 23, 2024

Introduce BASE_DATA and update DMPDIR for experimental obs guillaumevernieres/global-workflow#8

Merged

RussTreadon-NOAA mentioned this pull request Sep 24, 2024

Rename mtiasi to iasi NOAA-EMC/GDASApp#1292

Open

guillaumevernieres and others added 3 commits September 24, 2024 14:59

Merge pull request #8 from KateFriedman-NOAA/feature/marinevar

771efe4

Introduce `BASE_DATA` and update `DMPDIR` for experimental obs

addressed reviews

131e5eb

Merge branch 'feature/marinevar' of https://github.com/guillaumeverni…

8002a06

…eres/global-workflow into feature/marinevar

aerorahul requested a review from RussTreadon-NOAA September 26, 2024 14:56

RussTreadon-NOAA approved these changes Sep 26, 2024

View reviewed changes

aerorahul requested review from CoryMartin-NOAA, KateFriedman-NOAA, DavidHuber-NOAA, AndrewEichmann-NOAA and aerorahul September 26, 2024 15:01

Merge branch 'develop' into feature/marinevar

2b43678

aerorahul merged commit a694cf1 into NOAA-EMC:develop Sep 26, 2024
5 checks passed

RussTreadon-NOAA mentioned this pull request Sep 27, 2024

GDASApp EE2 compliance checklist for GFS v17 & v18 NOAA-EMC/GDASApp#1254

Open

10 tasks

guillaumevernieres deleted the feature/marinevar branch October 3, 2024 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the variational scripts to this repository #2920

Move the variational scripts to this repository #2920

guillaumevernieres commented Sep 13, 2024 •

edited by KateFriedman-NOAA

Loading

DavidNew-NOAA left a comment

aerorahul Sep 23, 2024

aerorahul Sep 23, 2024

aerorahul left a comment

KateFriedman-NOAA Sep 23, 2024

guillaumevernieres Sep 23, 2024

KateFriedman-NOAA Sep 23, 2024

DavidHuber-NOAA Sep 23, 2024

guillaumevernieres commented Sep 24, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA left a comment

aerorahul commented Sep 26, 2024

		f = open(yaml_name, 'w')
		yaml.dump(bkg_list, f, sort_keys=False, default_flow_style=False)

		with open('obs_list_short.yaml', 'w') as file:
		yaml.dump(parse_obs_list_file(self.task_config.MARINE_OBS_LIST_YAML), file, default_flow_style=False)

Move the variational scripts to this repository #2920

Move the variational scripts to this repository #2920

Conversation

guillaumevernieres commented Sep 13, 2024 • edited by KateFriedman-NOAA Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

aerorahul Sep 23, 2024

Choose a reason for hiding this comment

aerorahul Sep 23, 2024

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA Sep 23, 2024

Choose a reason for hiding this comment

guillaumevernieres Sep 23, 2024

Choose a reason for hiding this comment

KateFriedman-NOAA Sep 23, 2024

Choose a reason for hiding this comment

DavidHuber-NOAA Sep 23, 2024

Choose a reason for hiding this comment

guillaumevernieres commented Sep 24, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA commented Sep 26, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

aerorahul commented Sep 26, 2024

guillaumevernieres commented Sep 13, 2024 •

edited by KateFriedman-NOAA

Loading