Updates baseline documentation

doc/source/users_guide/testing.rst

@@ -371,29 +371,148 @@ Interpreting test output is pretty easy, looking at an example::
 You can see that `create_test <../Tools_user/create_test.html>`_  informs the user of the case directory and of the progress and duration
 of the various test phases.
 
-===================
-Managing baselines
-===================
-.. _`Managing baselines`:
+=========
+Baselines
+=========
+.. _`Baselines`:
 
-A big part of testing is managing your baselines (sometimes called gold results). We have provided
-tools to help the user do this without having to repeat full runs of test cases with `create_test <../Tools_user/create_test.html>`_ .
+A big part of testing is managing your baselines (sometimes called gold results). We have provided tools to help the user do this without having to repeat full runs of test cases with `create_test <../Tools_user/create_test.html>`_ .
 
-bless_test_results: Takes a batch of cases of tests that have already been run and copy their
-results to a baseline area.
+-------------------
+Creating a baseline
+-------------------
+.. _`Creating a baseline`:
 
-compare_test_results: Takes a batch of cases of tests that have already been run and compare their
-results to a baseline area.
+A baseline can be generated by passing ``-g`` to `create_test <../Tools_user/create_test.html>`_. There are additional options to control generating baselines.::
 
-Take a batch of results for the jenkins user for the testid 'mytest' and copy the results to
-baselines for 'master'::
+  ./scripts/create_test -b master -g SMS.ne30_f19_g16_rx1.A
 
-  ./bless_test_results -r /home/jenkins/e3sm/scratch/jenkins/ -t mytest -b master
+--------------------
+Comparing a baseline
+--------------------
+.. _`Comparing a baseline`:
 
-Take a batch of results for the jenkins user for the testid 'mytest' and compare the results to
-baselines for 'master'::
+Comparing the output of a test to a baseline is achieved by passing ``-c`` to `create_test <../Tools_user/create_test.html>`_.::
+
+  ./scripts/create_test -b master -c SMS.ne30_f19_g16_rx1.A
+
+------------------
+Managing baselines
+------------------
+.. _`Managing baselines`:
 
-  ./compare_test_results -r /home/jenkins/e3sm/scratch/jenkins/ -t mytest -b master
+Once a baseline has been generated it can be managed using the `bless_test_results <../Tools_user/bless_test_results.html>`_ tool. The tool provides the ability to bless different features of the baseline. The currently supported features are namelist files, history files, and performance metrics. The performance metrics are separated into throughput and memory usage.
+
+The following command can be used to compare a test to a baseline and bless an update to the history file.::
+
+  ./CIME/Tools/bless_test_results -b master --hist-only SMS.ne30_f19_g16_rx1.A
+
+The `compare_test_results <../Tools_user/compare_test_results.html>_` tool can be used to quickly compare tests to baselines and report any `diffs`.::
+
+  ./CIME/Tools/compare_test_results -b master SMS.ne30_f19_g16_rx1.A
+
+---------------------
+Performance baselines
+---------------------
+.. _`Performance baselines`:
+By default performance baselines are generated by parsing the coupler log and comparing the throughput in SYPD (Simulated Years Per Day) and the memory usage high water.
+
+This can be customized by creating a python module under ``$DRIVER_ROOT/cime_config/customize``. There are four hooks that can be used to customize the generation and comparison.
+
+- perf_get_throughput
+- perf_get_memory
+- perf_compare_throughput_baseline
+- perf_compare_memory_baseline
+
+..
+  TODO need to add api docs and link
+The following pseudo code is an example of this customization.::
+
+  # $DRIVER/cime_config/customize/perf_baseline.py
+
+  def perf_get_throughput(case):
+    """
+    Parameters
+    ----------
+    case : CIME.case.case.Case
+      Current case object.
+
+    Returns
+    -------
+    str
+      Storing throughput value.
+    """
+    current = analyze_throughput(...)
+
+    return json.dumps(current)
+
+  def perf_get_memory(case):
+    """
+    Parameters
+    ----------
+    case : CIME.case.case.Case
+      Current case object.
+
+    Returns
+    -------
+    str
+      Storing memory value.
+    """
+    current = analyze_memory(case)
+
+    return json.dumps(current)
+
+  def perf_compare_throughput_baseline(case, baseline, tolerance):
+    """
+    Parameters
+    ----------
+    case : CIME.case.case.Case
+      Current case object.
+    baseline : str
+      Baseline throughput value.
+    tolerance : float
+      Allowed difference tolerance.
+
+    Returns
+    -------
+    bool
+      Whether throughput diff is below tolerance.
+    str
+      Comments about the results.
+    """
+    current = analyze_throughput(case)
+
+    baseline = json.loads(baseline)
+
+    diff, comments = generate_diff(...)
+
+    return diff, comments
+
+  def perf_compare_memory_baseline(case, baseline, tolerance):
+    """
+    Parameters
+    ----------
+    case : CIME.case.case.Case
+      Current case object.
+    baseline : str
+      Baseline memory value.
+    tolerance : float
+      Allowed difference tolerance.
+
+    Returns
+    -------
+    bool
+      Whether memory diff is below tolerance.
+    str
+      Comments about the results.
+    """
+    current = analyze_memory(case)
+
+    baseline = json.loads(baseline)
+
+    diff, comments = generate_diff(...)
+
+    return diff, comments
 
 =============
 Adding tests