kubernetes integration

[wheaton5]

The sanger is moving to some cloud service called openstack and using a kubernetes cluster https://kubernetes.io/ for job submission. It isn't fully set up yet so I can't test it, but they are initially creating support with nextflow pipeline management system. It is somewhat similar to martian, but I think martian still has some key advantages (not least of which is that nextflow uses groovy and I can't even run it from a head node here due to memory limits...). I don't know much about kubernetes, but how hard do you all think it would be to get working with that? Probably not terrible but I can't really test anything at the moment so blech.

[adam-azarchs]

At a very basic level, this is easy. It looks like there are "manila" plugins offering shared nfs filesystems, which are a prerequisite for martian, and "sahara" looks like it supports shell jobs, which could be used to integrate with martian. It might be nessessary to create a wrapper script to transform the job template which martian gives to the job manager binary's standard in into whatever command is required to make the job actually run.

This is definitely something we'll look into eventually, although I cannot promise a schedule. I don't have easy access to an openstack cluster to test on. If you would like to experiment with it on your own, I'd be happy to offer advice if you run into trouble.