Plugins/testers (#1122)

* file shouldnt be present still

* starting work

* starting work

* stash wip installer

* stash again

* WIP need consistent InputData for ROM, PP so that we can propagate them forward

* telling plugins about raven

* stash

* libraries from plugins now readable

* stash

* starting work

* stash wip installer

* stash again

* WIP need consistent InputData for ROM, PP so that we can propagate them forward

* telling plugins about raven

* stash

* libraries from plugins now readable

* stash

* temp stash

* now running tests for plugins

* example plugin connected, running tests, and using Plugin.Entity naming

* extraneous file tracked

* cleanup

* no utils import in factory

* removed numpy from utils, but plotting is hard broken now

* in-time import for variablegroups

* matplotlib lib update fixes segfault

* module-style import of utils in lib report

* pylint

* cleanup removal

* utils swapping fixes

* maybe got all of them now?

* fixed underscore escaping in existing interfaces

* install all plugins, install plugins by submodule options

* comments addressed

* extending auto docs maker

* updating CashFlow submodule ID

* PRAplugin included, also some path load order fixes

* comments

* raven only in qsub tests

* added Plugin API test to RAVEN standard testing

* cleanup

* stash

* docs and improved tests

* cleanup

* added example plugin test harness

* non default qsub type to plugin tests

doc/plugins_manual/making_new_plugin.tex

@@ -3,8 +3,8 @@ \section{Making a New RAVEN Plugin}
 Creating a new plugin is a straightforward process. It involves setting up a repository,
 establishing a basic structure, and installing in RAVEN for testing.
 
-\subsection{Setting up a Repository}
-TODO
+%\subsection{Setting up a Repository}
+%TODO
 
 \subsection{Plugin Structure}
 The following directories must be present in the main directory of the plugin in order for RAVEN to
@@ -25,4 +25,32 @@ \subsection{Installing in RAVEN}
 Use the installation script in \texttt{raven/install\_plugins.py}.
 
 This process automatically registers the plugin in the plugin directory, and informs the plugin
-about RAVEN (TODO more notes on this!)
+about RAVEN %(TODO more notes on this!)
+
+\subsection{Adding Testers}
+RAVEN automatically provides a testing harness for automated regression testing. This includes a
+variety of \emph{testers}, such as CSV checkers and XML checkers.
+
+If a plugin requires additional testers for regression testing, they can be added to the plugin and
+loaded by RAVEN's test harness at testing time.
+
+Any new testers should be added under a folder named \texttt{Testers} in the \texttt{src} directory
+of the plugin. For example, for a plugin named \texttt{examplePlugin} and a tester named
+\texttt{myNewTester}:
+\begin{lstlisting}[morekeywords={examplePlugin,myNewTester}]
+  /path/to/examplePlugin/src/Testers/myNewTester.py
+\end{lstlisting}
+Any discovered testers will be made available to the \texttt{tests} files used by the RAVEN
+regression test system; for example:
+\begin{lstlisting}[morekeywords={myNewTester}]
+[Tests]
+  [./aTestForMyPlugin]
+    type = myNewTester
+    input = my_test_input.xml
+    csv = 'TestWorkingDir/results.csv'
+  [../]
+[]
+\end{lstlisting}
+
+For more information on inheriting from and creating new testers, see the RAVEN regression system
+documentation.

framework/Models/ExternalModel.py

@@ -147,7 +147,6 @@ def localInputAndChecks(self,xmlNode):
       self.sim = utils.importFromPath(moduleToLoadString,self.messageHandler.getDesiredVerbosity(self)>1)
     ## NOTE we implicitly assume not having ModuleToLoad means you're a plugin or a known type.
     elif paramInput.parameterValues['subType'].strip() is not None:
-      print('DEBUGG known:', ExternalModel.plugins.knownTypes())
       # We assume it is a plugin. Look for the type in the plugins class list
       if paramInput.parameterValues['subType'] not in ExternalModel.plugins.knownTypes():
         self.raiseAnError(IOError,('The "subType" named "{sub}" does not belong to any ' +

plugins/ExamplePlugin/src/Testers/ExampleTester.py

@@ -0,0 +1,45 @@
+# Copyright 2017 Battelle Energy Alliance, LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+RavenFramework is a tool to test raven inputs.
+"""
+from __future__ import absolute_import
+import os
+import sys
+import platform
+
+# need to use RavenFramework as a base
+try:
+  from RavenFramework import RavenFramework
+except ModuleNotFoundError:
+  sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)),
+                               '..', '..', '..', '..',
+                               'scripts', 'TestHarness', 'testers'))
+  from RavenFramework import RavenFramework
+  sys.path.pop()
+
+class ExampleIntegration(RavenFramework):
+  """
+    Example class for integration tests in ExamplePlugin
+  """
+  def get_command(self):
+    """
+      Gets command to run this test.
+      @ In, None
+      @ Out, get_command, string, command to run
+    """
+    # same as RAVEN, but for demonstration add a little print
+    command = RavenFramework.get_command(self)
+    command += '&& echo ExamplePlugin Integration Test Complete!'
+    return command

plugins/ExamplePlugin/tests/tests

@@ -1,6 +1,6 @@
 [Tests]
  [./TestPluginExample]
-  type = 'RavenFramework'
+  type = 'ExampleIntegration'
   input = 'test_example_plugin.xml'
   csv = 'simpleTest/dumpSumOfExpOutput.csv'
   max_time = 100

rook/doc/rook.tex

@@ -84,7 +84,7 @@ \section{Configuration File}
 
 \begin{description}
 \item[test\_dir] directory where the tests are located.  Equivalent to {\tt --test-dir}
-\item[scripts\_dir] directory where the scripts are located. Equivalent to {\tt --scripts-dir}
+\item[testers\_dir] directory where the testers are located. Equivalent to {\tt --testers-dir}
 \item[add\_run\_types] add run types to the ones to be run. Equivalent to {\tt --run-types}
 \item[only\_run\_types] only run the listed types. Equivalent to {\tt --only-run-types}
 \item[add\_non\_default\_run\_types] add a run type that is not run by default. Equivalent to {\tt --add-non-default-run-types}

rook/main.py

@@ -81,8 +81,8 @@ class NoColors:
 parser.add_argument('--test-dir', dest='test_dir',
                     help='specify where the tests are located')
 
-parser.add_argument('--scripts-dir', dest='scripts_dir',
-                    help='specify where the scripts are located')
+parser.add_argument('--testers-dir', dest='testers_dirs',
+                    help='specify where the scripts are located. May be comma-separated.')
 
 parser.add_argument('--run-types', dest='add_run_types',
                     help='add run types to the ones to be run')
@@ -220,6 +220,9 @@ def get_testers_and_differs(directory):
     @ Out, (tester_dict, differ_dict), tuple of dictionaries
       returns dictionaries with all the subclasses of Tester and Differ.
   """
+  # if no testers added, that's fine
+  if not os.path.isdir(directory):
+    return {}, {}
   tester_dict = {}
   differ_dict = {}
   os.sys.path.append(directory)
@@ -319,15 +322,19 @@ def term_handler(signum, _):
   else:
     base_test_dir = [x.strip() for x in args.test_dir.split(',')]
 
-
   test_list = get_test_lists(base_test_dir)
 
   base_testers, base_differs = get_testers_and_differs(this_dir)
-  if args.scripts_dir is None:
-    scripts_dir = os.path.join(up_one_dir, "scripts", "TestHarness", "testers")
+  if not args.testers_dirs:
+    testers_dirs = [os.path.join(up_one_dir, "scripts", "TestHarness", "testers")]
   else:
-    scripts_dir = args.scripts_dir
-  testers, differs = get_testers_and_differs(scripts_dir)
+    testers_dirs = args.testers_dirs.split(',')
+  testers = {}
+  differs = {}
+  for testers_dir in testers_dirs:
+    new_testers, new_differs = get_testers_and_differs(testers_dir)
+    testers.update(new_testers)
+    differs.update(new_differs)
   testers.update(base_testers)
   differs.update(base_differs)
   Tester.add_non_default_run_type("heavy")

run_tests

@@ -122,7 +122,9 @@ if [[ $DO_RAVEN == 0 ]]; then
   echo
   echo Running $P tests ...
   # get location of ExamplePlugin test dir
-  $PYTHON_COMMAND $SCRIPT_DIR/rook/main.py --test-dir $LOCATION --add-non-default-run-types qsub "${ARGS[@]}"
+  ROOK_COMMAND="$PYTHON_COMMAND $SCRIPT_DIR/rook/main.py --test-dir $LOCATION --testers-dir $SCRIPT_DIR/scripts/TestHarness/testers,$LOCATION/../src/Testers --add-non-default-run-types qsub ${ARGS[@]}"
+  # $PYTHON_COMMAND $SCRIPT_DIR/rook/main.py --test-dir $LOCATION --add-non-default-run-types qsub "${ARGS[@]}"
+  $ROOK_COMMAND
   rc=$?
   ALL_PASS=$(($ALL_PASS + $rc))
   if [[ $rc != 0 ]]; then
@@ -147,7 +149,9 @@ if [[ $DO_PLUGINS == 0 ]]; then
     echo "********************************************************************************"
     echo
     echo Starting tests for plugin "$P" ...
-    $PYTHON_COMMAND $SCRIPT_DIR/rook/main.py --test-dir $LOCATION "${ARGS[@]}"
+    # add RAVEN testers to plugin testers
+    ROOK_COMMAND="$PYTHON_COMMAND $SCRIPT_DIR/rook/main.py --test-dir $LOCATION --testers-dir $SCRIPT_DIR/scripts/TestHarness/testers,$LOCATION/../src/Testers --add-non-default-run-types qsub ${ARGS[@]}"
+    $ROOK_COMMAND
     rc=$?
     ALL_PASS=$(($ALL_PASS + $rc))
     if [[ $rc != 0 ]]; then

scripts/install_plugins.py

@@ -43,6 +43,12 @@
                     help='show all available standard RAVEN plugins and exit')
 args = parser.parse_args()
 
+# -> manually add to install list for "all"
+# ExamplePlugin should always stay here.
+# PRAPlugin can be moved once it is in a separate repository.
+manualAddedPlugins = ['PRAplugin', 'ExamplePlugin']
+# END TEMPORARY FIXME
+
 if __name__ == '__main__':
   ### Design notes
   # "Installing" is actually just the process of registering the location of the plugin
@@ -75,12 +81,7 @@
 
   # if requested "all" install, update sources
   if args.doAll:
-    # TODO TEMPORARY FIXME for PRAplugin:
-    # -> manually add to install list for "all"
-    # This is because the plugin maintainers have not transitioned to a separate repository yet.
-    submods.append('PRAplugin')
-    submods.append('ExamplePlugin')
-    # END TEMPORARY FIXME
+    submods.extend(manualAddedPlugins)
     args.source_dir = submods
   elif not args.source_dir:
     returnCode += 1