Cesm3.0 add marbl

[mnlevy1981]

Description of changes

Updates necessary to run MOM6 with MARBL, also removes cheyenne-specific and POP-specific PE layouts

Specific notes

Requires FMS tag fi_240807, CMEPS tagged after ESCOMP/CMEPS#493, and MOM interface tag mi_240805

Fixes: #268 (support turning MARBL tracers on in MOM6)

User interface changes?: No

Testing performed (automated tests and/or manual tests): Once I'm done making changes, I'll add a comment with the testing I've done + take this out of draft mode

[mnlevy1981]

I added a new BLT1850_MARBL compset, and also an ERI test for it (in prealpha). I also updated PE layouts to increase NTASKS_OCN for MOM6%MARBL-BIO compsets.

Unrelated to MARBL, I also removed PE layouts if any of the following applied:

1. The layout was for running on cheyenne
2. The layout was for running on POP grids
3. The layout was for running on the old MOM6 tx0.66v1 grid

[mnlevy1981]

I should note that the BLT1850_MARBL compset is temporary -- once we have a chance to do some comparisons with BLT1850 and fix any issues that may arise, we want all the allactive compsets to use MARBL by default. At that point, we'll remove BLT1850_MARBL.

[jedwards4b]

If you are satisfied with the load balancing of MARBL compsets please add them to the cesm3 timing table at https://cseg.cgd.ucar.edu/timing/login/

[mnlevy1981]

Oh, and the XL pelayout needs to be retuned -- MOM6 can't get 20 SYPD with MARBL turned on, likely due to I/O issues when running with larger core counts. Reducing the number of CAM tasks might keep throughput the same but decrease overall cost?

The existing XL layout without MARBL claims 20 SYPD / 7100 pe-hrs/simyear, though I saw 17.9 SYPD / 8000 pe-hrs/simyear when I ran (during the tutorial, so computer was busier than usual). Turning MARBL on, I saw 18.3 SYPD and 11500 pe-hrs/simyear but even when the machine is cooperating I never got to the 20 SYPD in ocean-only testing.

[mnlevy1981]

At @jedwards4b's suggestion, I ran a series of PFS tests for the different pre-defined pe layout sizes:

pecount	SYPD (no MARBL)	pe-hrs / sim-year (no MARBL)	SYPD (MARBL)	pe-hrs / sim-year (MARBL)
S	2.43	5057	2.61	7075
M	9.48	5833	9.14	8740
L	11.87	6210	11.39	9708
XL	18.05	8000	15.82	13202

The XL results are interesting -- the ocean runtime is still less than ATM + max(LND+ROF, ICE), but the CPL COMM time increased quite a bit and accounts for the entire performance hit.

Component	Run Time (s) (no MARBL)	Run Time (s) (MARBL)	diff (s)
TOT	262.250	299.201	36.951
CPL	17.793	17.203	-0.590
ATM	169.303	174.386	5.083
LND	51.998	55.270	3.272
ICE	39.824	48.156	8.332
OCN	169.424	203.312	33.888
ROF	3.672	5.157	1.484
GLC	0.027	0.027	0.000
WAV	0.000	0.000	0.000
ESP	0.000	0.000	0.000
CPL COMM	45.571	83.982	38.411

Before I add any of this to the timing table, I'd like to wait and see how throughput compares (at least for the medium size layout) in the CESM3 development runs. looking at these results, I wouldn't be surprised if we need to tweak the layouts a little bit...

[jedwards4b]

Looking at the timing table for XL:

    TOT Run Time:     262.250 seconds       13.112 seconds/mday        18.05 myears/wday 
    CPL Run Time:      17.793 seconds        0.890 seconds/mday       266.07 myears/wday 
    ATM Run Time:     169.303 seconds        8.465 seconds/mday        27.96 myears/wday 
    LND Run Time:      51.998 seconds        2.600 seconds/mday        91.05 myears/wday 
    ICE Run Time:      39.824 seconds        1.991 seconds/mday       118.88 myears/wday 
    OCN Run Time:     169.424 seconds        8.471 seconds/mday        27.94 myears/wday 
    ROF Run Time:       3.672 seconds        0.184 seconds/mday      1289.11 myears/wday 
    GLC Run Time:       0.027 seconds        0.001 seconds/mday    175994.30 myears/wday 
    WAV Run Time:       0.000 seconds        0.000 seconds/mday         0.00 myears/wday 
    ESP Run Time:       0.000 seconds        0.000 seconds/mday         0.00 myears/wday 
    CPL COMM Time:     45.571 seconds        2.279 seconds/mday       103.89 myears/wday 

The ocean is waiting some 56s per coupling interval.

[jedwards4b]

@mnlevy1981 You'll need to resolve the conflict before I can merge - I can't push to your fork.