Skip to content
oleweidner edited this page Mar 25, 2013 · 45 revisions

.

ATTENTION: bliss is now saga-python. Please check out the new website for the latest releases and documentation!

You are interested in developing a new middleware plug-in for Bliss? That's awesome! Community involvement is critical for the success of project like Bliss. To make your life a little easier, we'll guide you through the whole process with an example.

There are three high-level steps involved in the process: (1) you fork the Bliss repository on Github, (2) you implement your changes / additions to the codebase, and (3) you send us a pull request if you think your changes should be merged back into the main Bliss repository.

Step (1) and (3) are explained in detail on the Github help pages linked above. In this document, we focus on the code development aspects by walking you through the steps necessary to develop a dummy job adaptor.

Create the Plug-In Directory

After you have cloned the latest version of Bliss, change to the bliss/plugins directory. Create a new directory for your plug-in - for this example, we call it nulljob. Next, copy the contents of the job TEMPLATE directory into your newly created nulljob directory:

cd bliss/plugins
mkdir nulljob
cp -r TEMPLATES/job/*.py nulljob/

The files you have just copied provide a fully "functional" dummy plugin. Have a look at the job.py file if you want - it should be sufficiently documented. There are a few important lines (you don't have to change anything for now - just have a look):

  • class NullJobPlugin(JobPluginInterface): - This class implements the SAGA job package interface.
  • _name = 'saga.plugin.job.null' - The unique identifier of the plugin
  • _schemas = ['null'] - The URL schemes the plugging supports (here: null://)
  • _apis = ['saga.job'] - APIs supported by this plug-in (here: the job api)

Register Plug-In with the Runtime

Currently, Bliss' plug-in mechanism is somewhat static: every single plug-in needs to be registered statically with the runtime, otherwise Bliss won't consider it during runtime. At some point. we plan to substitute this somewhat inflexible mechanism with some sort of a configuration file; however, this is not in place yet.

In order to register your plug-in with the runtime, open the file bliss/plugins/registry.py and add the following lines:

from bliss.plugins.nulljob import NullJobPlugin
_registry.append({"class"   : NullJobPlugin,
                  "apis"    : NullJobPlugin.supported_apis(),
                  "name"    : NullJobPlugin.plugin_name(),
                  "schemas" : NullJobPlugin.supported_schemas()})

Register Plug-In with the Installer

Now that everything is in place, the last step is to register your new plug-in with the Bliss installer. This step ensures that (a) your plug-in will get pulled into the Bliss source releases and (b) your plug-in gets properly installed by pip and easy_install.

First, open the file MANIFEST.in (this file controls the stuff that goes into the source releases) and add the following line:

include bliss/plugins/nulljob/*.py

Next, open the file setup.py (this is the file that controls what gets installed by easy_install/pip), locate the 'packages': [] list, and add the following entry:

bliss.plugins.nulljob,

Give it a Try

Your basic plug-in skeleton should now be in place. It's time to check that it is actually working. The easiest way to achieve this is to write a simple Bliss test script that triggers your nulljob plug-in.

Create a new file, name it /tmp/nulljobtest.py and cut & paste the code snippet below. Make sure that you don't put the file in the Bliss root directory, otherwise it will pick up the local Bliss, instead the one installed by easy_install. While this would still work, it won't tell you if easy_install installs your plug-in properly :

#!/usr/bin/env python
import bliss.saga as saga

def main():
    try:
        js = saga.job.Service("null://localhost")
        print js._get_runtime_info()
    except saga.Exception, ex:
        print "Oh, snap! An error occured: %s" % (str(ex))

if __name__ == "__main__":
    main()

Next, create a clean virtualenv, install your local Bliss version and run the test script. This will ensure that (a) your plug-in gets installed properly and (b) that your plug-in gets triggered for null://-type URLs. Run the following commands in the Bliss root directory:

virtualenv --no-site-packages /tmp/Bliss
source /tmp/Bliss/bin/activate
easy_install .
SAGA_VERBOSE=4 python /tmp/nulljobtest.py

The last few lines of the output should look somewhat similar to this:

02/29/2012 01:23:48 PM - NullJobPlugin(0x106c059e0) - INFO - Registered new service object 
02/29/2012 01:23:48 PM - Service(0x1067a3c50) - INFO - Bound to plugin 
02/29/2012 01:23:48 PM - Runtime(0x106c05710) - INFO - Found an existing plugin instance for url scheme null://: 
Hi there. I am the Null Plugin and I have nothing to say!
02/29/2012 01:23:48 PM - NullJobPlugin(0x106c059e0) - INFO - Unegistered service object 

If you instead get an exception like this:

Exception: Couldn't find a plugin for URL scheme 'null://' and API type 'saga.job'

your plug-in wasn't registered properly with the Bliss runtime. If that's the case, carefully review the steps above and make sure that you didn't miss anything.

Implement Your Own Plug-In

Now that you have a working skeleton in place, it is time to venture out and implement your own plug-in. You can use the skeleton that you have created in the previous steps, but make sure that you go through each step again and replace nulljob with whatever name you want to use for your plug-in.

It's hard to give you any further guidelines on how to implement the specific details of your plug-in since this is somewhat dependent on the middleware / service you want to interface with. However, here are a few guidelines that we consider best practice:

  • Think Before you Start Hacking: A simple, elegant and thought-through solution is always preferred over a messy hodgepodge of code. If you send us a pull-request for the latter, you will get ridiculed ;-).
  • Avoid Exotic Dependencies: The general rule is: if it's not in PyPi, don't use it! Remember, the whole idea behind Bliss that you can install it anywhere with a simple easy_install Bliss.
  • Never Hard-Code Variable Parameters: That should be pretty self explanatory. This applies especially for passwords (!) and paths to command-line tools you might want to invoke from your plug-in.
  • Check Out Existing Plug-Ins: For example, have a look at the local or pbs+ssh plug-ins. Maybe you can find some inspiration?
  • Report Bugs: If you think there's a problem with Bliss (and we know that there have to be tons of bugs in the code), please don't hesitate to file an issue. Same applies to feature requests!

Logging

One thing you should never ever do, is to use print statements in your plug-in that write directly to stdout or stderr. We consider that a capital sin and we will reject your plug-in if it doesn't adhere to this. That being said, Bliss comes with its own logging mechanism that you can use for debugging purposes.

The Bliss logging mechanism is controlled by the environment variable SAGA_VERBOSE. The log-levels are as follows:

  • 1 - ERROR
  • 2 - WARNING
  • 3 - INFO
  • 4 - DEBUG

You can write messages to these four log-level from within your plug-in using the methods below. The logging engine will automatically add the date, plugin-name and plugin instance to the log message.

  • self.log_error("Write log message to ERROR channel")
  • self.log_warning("Write log message to WARNING channel")
  • self.log_info("Write log message to INFO channel")
  • self.log_debug("Write log message to DEBUG channel")

Exceptions

The SAGA standard defines the exception types listed below. Please refer to the SAGA specification for more details on exceptions. If you are not sure which of the exception types to raise, use NoSuccess - that sort of always works.

  • AlreadyExists = bliss.saga.Error.AlreadyExists
  • AuthenticationFailed = bliss.saga.Error.AuthenticationFailed
  • AuthorizationFailed = bliss.saga.Error.AuthorizationFailed
  • BadParameter = bliss.saga.Error.BadParameter
  • DoesNotExist = bliss.saga.Error.DoesNotExist
  • IncorrectState = bliss.saga.Error.IncorrectState
  • IncorrectURL = bliss.saga.Error.IncorrectURL
  • NoSuccess = bliss.saga.Error.NoSuccess
  • NotImplemented = bliss.saga.Error.NotImplemented
  • PermissionDenied = bliss.saga.Error.PermissionDenied
  • Timeout = bliss.saga.Error.Timeout

It is common practice in Bliss to write an ERROR log-message for the actual exception gets raised. There's a convenience method for that - you should always use it:

self.log_error_and_raise(bliss.saga.Error.NoSuccess, "Oh noes :(")