Possible Cylc Futures

This wiki outlines some of the topics which were discussed in June 2018 in Exeter.

Cylc currently scales comfortably to many thousands of tasks. The suite configuration, however, does not scale so gracefully. There is a demand for a modular suite configuration but it is difficult to satisfy all of the drivers of modularity with current approaches (see the aborted PR https://github.com/cylc/cylc/pull/2311).

These are descriptions of futuristic approaches I could see the cylc scheduler taking to meet the rising demands of suite scalability both from the perspective of performance but also [mainly] usability.

System Architecture

Separate the current code base into a kernel - shell model.

Motivation

The cylc kernel is currently pretty heavyweight with configuration logic mixed in with runtime logic (e.g. cylc.config). Separating the two paves the path to a cleaner leaner kernel which scales better and is able to be configured and run in different ways.

Kernel Structure

The Cylc kernel is currently monolithic, we could design it as a pool of micro-services but that's probably a bad difficult idea (cite GNU Hurd). Perhaps best to use a hybrid architecture where the scheduler is monolithic but with some functionality factored out into dynamically loaded modules enabling the kernel to be run in different configurations (see "Frames").

Kernel Shell Divide

Shell:

• Suite configuration
• Suite reload states
• Suite control commands
• Log retrieval
• ...

Kernel:

• Scheduling algorithm

• Message broker

• Modules dynamically loaded based on suite configuration:

  • Job submission
  • File creation (i.e. job output files, directory structure, etc)
  • Cycling?
  • ...

The shell should create the suite configuration object which is then loaded by the kernel. This way the configuration code is left behind in the shell process making the kernel much leaner.

Data Objects

The kernel would be stateless and work with two data objects, configuration and suite state. These objects would be python interfaces mapping onto the data storage model (i.e. database for the suite state object, Python object store for the suite configuration?).

This would allow us to easily change the data storage method, for example to support a different database solution a single plugin can be written.

If all modifications to suite state were written to the database then all would be transactioned which would allow us to easily roll back the suite state on demand.

Hierarchical Suites

An Advanced (though fairly simple) model for providing

Motivation

Cylc suites are Monolithic beasts. Cylc/Jinj2 imports/includes allow us to spread out over multiple files but fail to provide a modular architecture.

As suites continue to become more complicated the ability to divide them into components and develop each in some form of isolation becomes desirable.

File Structure

In this model a Cylc suite becomes a python module, tasks become executables. Sub suites are python modules. Nesting can go as many levels deep as desired.

mysuite/
   __init__.py
   foo
   bar
   baz/
      __init__.py
      pub

Sub Suites

Sub suites (e.g. baz in the example above) can be composed into the parent suite in the same manner in which tasks are:

foo >> bar >> baz

This enables us to run sub-workflows without requiring prior knowledge of their structure. In this case baz represents a single task, it could be an independently cycling workflow consisting of many tasks.

For deeper integration (e.g. writing a dependency to or from tasks in the sub suite) components of the sub suite can be imported:

import baz

bar >> baz.pub

As sub suites are python modules parts of the implementation can be imported from the sub suite into the parent suite. The parent can inspect the child but not the other way around.

Frames

Each suite is written in a "frame", this determines the way in which it is run and the way in which it may compose sub suites or be composed by a parent suite.

Cylc currently offers one "frame", the CLI frame in which all tasks are bash commands. Possible frames:

• CLI Frame (necessary)
• Python Frame (sensible)
• Bash Frame (possible)

Sub Suite Execution/Composition

The simplest way to handle sub suites is to compose them into the parent suite. This is probably the main use case. We could also permit running a sub suite as an independent [standalone] suite if scalability constraints require it.

Particularly for workflows written using the Python frame it may also be desirable to submit a sub suite as a job. This is especially useful for regular(ish) shaped python job arrays. To do this the kernel would need to be run in a different configuration:

• The job submission module need not be loaded.
• A different file creation module would be required.

User Presentation

Sub suites represent an additional level of hierarchy in the gui representations which could be "un-grouped" in the manner families currently are or graphed independently.

Another possibility would be to simplify the task status icons so that we could use colour to represent sub-suites. For motivation see the aborted PR https://github.com/cylc/cylc/pull/1932, which placed graph visualisation settings into "indicators" in the task status dot/graph-node/tree-icon.

Config API And Python Config API

See https://github.com/cylc/cylc/issues/1962

Motivation

The Python API should:

• Make writing programmatic suites easier.
• Configure cylc logic with Python objects rather than with strings (e.g. foo >> bar rather than 'foo => bar').
• Provide "on the fly" validation meaning that (for the first time since Jinja2) we can provide sensible error messages with line numbers.

Approaches

Python API could either:

1. Open up the OrderedDictWithDefaults datastructure.

   import cylc.config
   
   cfg = cylc.config.Config()
   
   cfg['scheduling']['dependencies']['graph'] = 'hello_world'
   cfg['runtime']['hello_world']['script'] = 'echo "Hello World!"'

   This option allows us to replace Jinja2 with Python but otherwise offers what we currently have.

2. Provide functions for constructing a configuration.

   from cylc import Task, Graph
   
   hello_world = Task('hello_world')  # would run the task mysuite/hello_world
   Graph(hello_world)

   This offers a much more elegant solution but requires an interface which can build a cylc configuration object.

Basics

Assuming option 2 (above), some possible syntax:

# Tasks
foo = Task('foo')
bar = Task('bar',
   env={'BAR': 'pub'}
)

# Dependencies

foo >> bar

# States

foo.state['fail'] >> bar

# Parameters
num = Param('num', range(10))

baz = TaskArray('baz', num)
pub = TaskArray('pub', num)

baz >> pub[0:5]

# Useful Parameters
num = Param('num', {
   1: {'foo': 'a', 'bar': 'b', 'baz': 'c'}
   2: {'foo': 'd', 'bar': 'e', 'baz': 'f'}
})

bol = TaskArray('bol', num
   env={'variable': num['foo']}
)

foo >> bol

# Cycling (somewhat tricker, a simple option)

graph = Graph(
   foo >> bar,
   bar >> baz
)
graph.cycle('P1D')

# Inter-cycle offsetd

graph = Graph(
   foo >> bar,
   bar('-PT1H') >> baz
)

Event Driven Scheduler

Presently the scheduler is self-assembling based on a "pool" of self-spawning tasks.

Spawn-on-demand requires the suite to have knowledge of the pattern of its graph.

This would allow us to spin up task proxies only when required cleaning up the distinction between tasks and jobs (TaskDefs become Tasks (nodes in the pattern), TaskProxies become Jobs (nodes in the Graph)).

Graphs

At present a suite has a single graph with cycling muddled in with dependencies.

Instead we could define graphs as standalone entities. To cycle a graph we provide a "driver" which repeatably "fires" off (inserts) the graph with the cycle as a parameter.

The "driver" could be cycling, equally it could be a wall-clock trigger or an external event. This opens up the opportunity for suites to respond to irregular events which do not fit into any current cycling model e.g. certain observations type workflows.

The cycling "driver" kicks off graphs with a cycle as a parameter, equally we should be able to "kick" off graphs using cylc parameters enabling subgraphs to be elegantly composed into parent workflows, for instance an ensemble workflow could be defined ensemble(animal), then used in the parent graph Foo >> ensemble(mammals).

This also solves the first and last cycle problems. No longer would we need to write dependencies like install_cold[^] => foo. For instance, the following is an example of a type of suite we see fairly often for data processing where cycles are independent:

[scheduling]
    max active cycle points = 100
    initial cycle point = 2000
    final cycle point = 2100
    [[dependencies]]
        [[[R1/^]]]
            graph = """
                make_something => install_cold<platform>
            """
        [[[P1D]]]
            graph = """
                install_cold<platform=a>[^] => foo => bar => baz
                install_cold<platform=b>[^] => qux
                bar & qux => pub
            """
        [[[R1/$]]]
            graph = """
{% for cycle in ... %}
                qux[{{cycle}}] & bar[{{cycle}}] => post_process => tidy
{% endfor %}
            """

And here is how it could be thought of as three distinct graphs:

    startup = cylc.graph(
        make_something >> install_cold(platform)
    )

    workflow = cylc.graph(
        foo >> bar >> baz,
        bar & qux >> pub
    )

    wrap_up = cylc.graph(
        post_process >> tidy
    )

    cylc.run(
        start_up >> workflow.every('P1D') >> wrap_up
    )

Events

With an event driven system we shouldn't need to iterate over the task pool to evaluate prerequisites etc. The graph could be something like a decision tree which would transform graphing events from an iteration problem as present (we must loop through the task pool) to a lookup / graph traversal problem (when an event occurs locate the associated node in the graph then follow the tree to find the required actions).