Merge pull request #369 from nismod/develop

Release smif v1.1

.appveyor.yml

@@ -18,9 +18,8 @@ install:
   - conda update conda
   - conda config --set changeps1 false
   - conda config --set channel_priority strict
-  - conda config --add channels conda-forge
   - conda info -a
-  - "conda create -n testenv python=%PYTHON_VERSION% \
+  - "conda create -n testenv python=3.7 \
         fiona \
         flask \
         gdal \
@@ -29,13 +28,11 @@ install:
         networkx \
         numpy \
         pandas \
-        pint \
         psycopg2 \
         pyarrow \
         pytest \
         python-dateutil \
         rtree \
-        ruamel.yaml \
         shapely \
         xarray"
   - activate testenv

docs/decisions.rst

@@ -0,0 +1,159 @@
+.. _decisions:
+
+Strategies, Interventions and Decision Modules
+==============================================
+
+**smif** makes a sharp distinction between *simulating* the operation of a system, and
+*deciding* on which interventions to introduce to meet goals or constraints on the whole
+system-of-systems.
+
+The decision aspects of **smif** include a number of components.
+
+- The DecisionManager interacts with the ModelRunner and provides a list of
+  timesteps and iterations to run
+- The DecisionManager also acts as the interface to a user implemented DecisionModule,
+  which may implement a particular decision approach.
+
+A decision module might use one of three approaches:
+
+- a rule based approach (using some heuristic rules), or
+- an optimisation approach.
+
+A pre-specified approach (testing a given planning pipeline) is included in the
+core **smif** code.
+
+The Decision Manager
+--------------------
+
+A DecisionManager is initialised with a DecisionModule implementation. This is
+referenced in the strategy section of a Run configuration.
+
+The DecisionManager presents a simple decision loop interface to the model runner,
+in the form of a generator which allows the model runner to iterate over the
+collection of independent simulations required at each step.
+
+The DecisionManager collates the output of the decision algorithm and
+writes the post-decision state to the store. This allows Models
+to access a given decision state in each timestep and decision iteration id.
+
+Decision Module Implementations
+-------------------------------
+
+Users must implement a DecisionModule and pass this to the DecisionModule by
+declaring it under a ``strategy`` section of a Run configuration.
+
+The DecisionModule implementation influences the combination and ordering of
+decision iterations and model timesteps that need to be performed to complete
+the run. To do this, the DecisionModule implementation must yield a bundle
+of interventions and planning timesteps, which are then simulated,
+after which the decision module may request further simulation of different
+timesteps and/or combinations of interventions.
+
+The composition of the yielded bundle will change as a function of the implementation
+type. For example, a rule-based approach is likely to iterate over individual
+years until a threshold is met before proceeding.
+
+A DecisionModule implementation can access results of previous iterations using
+methods available on the ResultsHandle it is passed at runtime. These include
+``ResultsHandle.get_results``.  The property ``DecisionModule.available_interventions``
+returns the entire collection of interventions that are available for deployment
+in a particular iteration.
+
+Interventions
+-------------
+
+Interventions change how a simulated system operates.
+An intervention can represent a building or upgrading a physical thing
+(like a reservoir or power station), or could be something less
+tangible like imposing a congestion charging zone over a city centre.
+
+A system of interest can in principle be composed entirely of a series of interventions. For
+example, the electricity generation and transmission system is composed of a set of generation
+sites (power stations, wind farms...), transmission lines and bus bars.
+
+A simulation model has access to several methods to obtain its current *state*.
+The DataHandle.get_state and DataHandle.get_current_interventions provide
+direct access the database of interventions relevant for the current timestep.
+
+Deciding on Interventions
+-------------------------
+
+The set of all interventions $I$ includes all interventions for all models in a
+system of systems.
+As the Run proceeds,
+and interventions are chosen by the DecisionModule implementation,
+then the set of available interventions is modified.
+
+Set of pre-specified or planned interventions $P{\subset}I$
+
+Available interventions $A=P{\cap}I$
+
+Decisions at time t ${D_t}\subset{A}-{D_{t-1}}$
+
+Pre-Specified Planning
+----------------------
+
+In a pre-specified planning strategy, a pipeline of interventions is forced into
+the system-of-systems.
+
+This requires the provision of data and configuration, described step by step below
+
+- define the set of interventions
+- define the planning strategy
+- add the pre-specified strategy to the model run configuration
+
+Define interventions
+~~~~~~~~~~~~~~~~~~~~
+
+Interventions are associated with an individual model, listed in a csv file and
+added to the model configuration as described in the project configuration part
+of the documentation <project_configuration>.
+
+Note that each intervention is identified by a ``name`` entry that must be unique
+across the system of systems. To ensure this, one suggestion is to use a pre-fix
+with the initals of the sector model to which the intervention belows.
+
+An example intervention file has the headers
+
+- name
+- location
+- capacity_value
+- capacity_units
+- operational_lifetime_value
+- operational_lifetime_units
+- technical_lifetime_value
+- technical_lifetime_units
+- capital_cost_value
+- capital_cost_units
+
+and contents as follows::
+
+        nuclear_large,Oxford,1000,MW,40,years,25,years,2000,million £
+        carrington_retire,Oxford,-500,MW,0,years,0,years,0,million £
+
+Define the planning strategy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A planning strategy consists of the set of (name, build_timestep) tuples, where
+each name must belong to the set of interventions.
+
+An example from the sample project looks like this::
+
+        name,build_year
+        nuclear_large,2010
+        carrington_retire,2015
+        ac_line1,2010
+
+Add the pre-specified strategy to the model run configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The final step is to add the pre-specified planning stategy to the run
+configuration::
+
+    strategies:
+    - type: pre-specified-planning
+      description: Future energy plan
+      filename: energy_supply/strategies/plan.csv
+
+The entry should take the above format, where the filename entry refers to the
+planning strategy file composed in step two.

src/smif/cli/__init__.py

@@ -86,26 +86,122 @@
 except ImportError:
     import thread as _thread
 
-
 try:
     import win32api
+
     USE_WIN32 = True
 except ImportError:
     USE_WIN32 = False
 
-
 __author__ = "Will Usher, Tom Russell"
 __copyright__ = "Will Usher, Tom Russell"
 __license__ = "mit"
 
 
 def list_model_runs(args):
-    """List the model runs defined in the config
+    """List the model runs defined in the config, optionally indicating whether complete
+    results exist.
     """
     store = _get_store(args)
     model_run_configs = store.read_model_runs()
+
+    if args.complete:
+        print('Model runs with an asterisk (*) have complete results available\n')
+
     for run in model_run_configs:
-        print(run['name'])
+        run_name = run['name']
+
+        if args.complete:
+            expected_results = store.canonical_expected_results(run_name)
+            available_results = store.canonical_available_results(run_name)
+
+            complete = ' *' if expected_results == available_results else ''
+
+            print('{}{}'.format(run_name, complete))
+        else:
+            print(run_name)
+
+
+def list_available_results(args):
+    """List the available results for a specified model run.
+    """
+
+    store = _get_store(args)
+    expected = store.canonical_expected_results(args.model_run)
+    available = store.available_results(args.model_run)
+
+    # Print run and sos model
+    run = store.read_model_run(args.model_run)
+    print('\nmodel run: {}'.format(args.model_run))
+    print('{}- sos model: {}'.format(' ' * 2, run['sos_model']))
+
+    # List of expected sector models
+    sec_models = sorted({sec for _t, _d, sec, _out in expected})
+
+    for sec_model in sec_models:
+        print('{}- sector model: {}'.format(' ' * 4, sec_model))
+
+        # List expected outputs for this sector model
+        outputs = sorted({out for _t, _d, sec, out in expected if sec == sec_model})
+
+        for output in outputs:
+            print('{}- output: {}'.format(' ' * 6, output))
+
+            # List available decisions for this sector model and output
+            decs = sorted({d for _t, d, sec, out in available if
+                           sec == sec_model and out == output})
+
+            if len(decs) == 0:
+                print('{}- no results'.format(' ' * 8))
+
+            for dec in decs:
+                base_str = '{}- decision {}:'.format(' ' * 8, dec)
+
+                # List available time steps for this decision, sector model and output
+                ts = sorted({t for t, d, sec, out in available if
+                             d == dec and sec == sec_model and out == output})
+                assert (len(
+                    ts) > 0), "If a decision is available, so is at least one time step"
+
+                res_str = ', '.join([str(t) for t in ts])
+                print('{} {}'.format(base_str, res_str))
+
+
+def list_missing_results(args):
+    """List the missing results for a specified model run.
+    """
+
+    store = _get_store(args)
+    expected = store.canonical_expected_results(args.model_run)
+    missing = store.canonical_missing_results(args.model_run)
+
+    # Print run and sos model
+    run = store.read_model_run(args.model_run)
+    print('\nmodel run: {}'.format(args.model_run))
+    print('{}- sos model: {}'.format(' ' * 2, run['sos_model']))
+
+    # List of expected sector models
+    sec_models = sorted({sec for _t, _d, sec, _out in expected})
+
+    for sec_model in sec_models:
+        print('{}- sector model: {}'.format(' ' * 4, sec_model))
+
+        # List expected outputs for this sector model
+        outputs = sorted({out for _t, _d, sec, out in expected if sec == sec_model})
+
+        for output in outputs:
+            print('{}- output: {}'.format(' ' * 6, output))
+
+            # List missing time steps for this sector model and output
+            ts = sorted({t for t, d, sec, out in missing if
+                         sec == sec_model and out == output})
+
+            if len(ts) == 0:
+                print('{}- no missing results'.format(' ' * 8))
+            else:
+                base_str = '{}- results missing for:'.format(' ' * 8)
+                res_str = ', '.join([str(t) for t in ts])
+                print('{} {}'.format(base_str, res_str))
 
 
 def run_model_runs(args):
@@ -243,6 +339,26 @@ def parse_arguments():
     parser_list = subparsers.add_parser(
         'list', help='List available model runs', parents=[parent_parser])
     parser_list.set_defaults(func=list_model_runs)
+    parser_list.add_argument('-c', '--complete',
+                             help="Show which model runs have complete results",
+                             action='store_true')
+
+    # RESULTS
+    parser_available_results = subparsers.add_parser(
+        'available_results', help='List available results', parents=[parent_parser])
+    parser_available_results.set_defaults(func=list_available_results)
+    parser_available_results.add_argument(
+        'model_run',
+        help="Name of the model run to list available results"
+    )
+
+    parser_missing_results = subparsers.add_parser(
+        'missing_results', help='List missing results', parents=[parent_parser])
+    parser_missing_results.set_defaults(func=list_missing_results)
+    parser_missing_results.add_argument(
+        'model_run',
+        help="Name of the model run to list missing results"
+    )
 
     # APP
     parser_app = subparsers.add_parser(

src/smif/data_layer/__init__.py

@@ -6,8 +6,9 @@
 #         from smif.data_layer import DataHandle`
 from smif.data_layer.data_array import DataArray
 from smif.data_layer.data_handle import DataHandle
+from smif.data_layer.results import Results
 from smif.data_layer.store import Store
 
 # Define what should be imported as * ::
 #         from smif.data_layer import *
-__all__ = ['DataArray', 'DataHandle', 'Store']
+__all__ = ['DataArray', 'DataHandle', 'Results', 'Store']