SAGA Tutorial Part 5: A More Complex Example: Mandelbrot
In this example, we split up the calculation of a Mandelbrot set into several tiles, submit a job for each tile using the SAGA Job API, retrieve the tiles using the SAGA File API and stitch together the final image from the individual tiles. This example shows how SAGA can be used to create more complex application workflows that involve multiple aspects of the API.
In order for this example to work, we need to install an additional Python module, the Python Image Library (PIL). This is done via pip:
pip install PIL
Next, we need to download the Mandelbrot fractal generator itself. It is really just a very simple python script that, if invoked on the command line, outputs a full or part of a Mandelbrot fractal as a PNG image. Download the script into your $HOME directory:
curl --insecure -Os https://raw.github.com/saga-project/bliss/master/examples/advanced/mandelbrot/mandelbrot.py
You can give mandelbrot.py a test-drive locally by calculating a single-tiled 1024x1024 Mandelbrot fractal:
python mandelbrot.py 1024 1024 0 1024 0 1024 frac.gif
In your $HOME directory, open a new file saga_mandelbrot.py with your favorite editor (e.g., vim) and paste the following content:
import sys, time, os
import bliss.saga as saga
from PIL import Image
# the dimension (in pixel) of the whole fractal
imgx = 2048
imgy = 2048
# the number of tiles in X and Y direction
tilesx = 2
tilesy = 2
if __name__ == "__main__":
try:
# list that holds the jobs
jobs = []
# create a working directory in /scratch
dirname = 'sftp://localhost/%s/mbrot/' % '/tmp'
workdir = saga.filesystem.Directory(dirname, saga.filesystem.Create)
# copy the executable into our working directory
mbexe = saga.filesystem.File('sftp://localhost/%s/mandelbrot.py' % os.getcwd())
mbexe.copy(workdir.get_url())
# the saga job services connects to and provides a handle
# to a remote machine. In this case, it's your machine.
# fork can be replaced with ssh here:
jobservice = saga.job.Service('fork://localhost')
for x in range(0, tilesx):
for y in range(0, tilesy):
# describe a single Mandelbrot job. we're using the
# directory created above as the job's working directory
outputfile = 'tile_x%s_y%s.gif' % (x,y)
jd = saga.job.Description()
jd.queue = "development"
jd.wall_time_limit = 10
jd.total_cpu_count = 1
jd.working_directory = workdir.get_url().path
jd.executable = 'python'
jd.arguments = ['mandelbrot.py', imgx, imgy,
(imgx/tilesx*x), (imgx/tilesx*(x+1)),
(imgy/tilesy*y), (imgy/tilesy*(y+1)),
outputfile]
# create the job from the description
# above, launch it and add it to the list of jobs
job = jobservice.create_job(jd)
job.run()
jobs.append(job)
print ' * Submitted %s. Output will be written to: %s' % (job.jobid, outputfile)
# wait for all jobs to finish
while len(jobs) > 0:
for job in jobs:
jobstate = job.get_state()
print ' * Job %s status: %s' % (job.jobid, jobstate)
if jobstate is saga.job.Job.Done:
jobs.remove(job)
time.sleep(5)
# copy image tiles back to our 'local' directory
for image in workdir.list('*.gif'):
print ' * Copying %s/%s back to %s' % (workdir.get_url(), image, os.getcwd())
workdir.copy(image, 'sftp://localhost/%s/' % os.getcwd())
# stitch together the final image
fullimage = Image.new('RGB',(imgx, imgy),(255,255,255))
print ' * Stitching together the whole fractal: mandelbrot_full.gif'
for x in range(0, tilesx):
for y in range(0, tilesy):
partimage = Image.open('tile_x%s_y%s.gif' % (x, y))
fullimage.paste(partimage, (imgx/tilesx*x, imgy/tilesy*y, imgx/tilesx*(x+1), imgy/tilesy*(y+1)) )
fullimage.save("mandelbrot_full.gif", "GIF")
sys.exit(0)
except saga.Exception, ex:
print 'Problem during execution: %s' % ex
sys.exit(-1)
except KeyboardInterrupt:
# ctrl-c caught: try to cancel our jobs before we exit
# the program, otherwise we'll end up with lingering jobs.
for job in jobs:
job.cancel()
sys.exit(-1)Save the file and execute it via the python interpreter (once again: make sure your virtualenv is activated and that your $HOME/.profile is prepared as described in section 2):
python saga_mandelbrot.py
The output should look something like this:
* Submitted [ssh://localhost]-[652593]. Output will be written to: tile_x0_y0.gif
* Submitted [ssh://localhost]-[652594]. Output will be written to: tile_x0_y1.gif
* Submitted [ssh://localhost]-[652595]. Output will be written to: tile_x1_y0.gif
* Submitted [ssh://localhost]-[652596]. Output will be written to: tile_x1_y1.gif
* Job [ssh://localhost]-[652593] status: saga.job.Job.Pending
* Job [ssh://localhost]-[652594] status: saga.job.Job.Pending
* Job [ssh://localhost]-[652595] status: saga.job.Job.Pending
* Job [ssh://localhost]-[652596] status: saga.job.Job.Pending
* Job [ssh://localhost]-[652593] status: saga.job.Job.Running
* Job [ssh://localhost]-[652594] status: saga.job.Job.Running
* Job [ssh://localhost]-[652595] status: saga.job.Job.Running
* Job [ssh://localhost]-[652596] status: saga.job.Job.Running
...
* Job [ssh://localhost]-[652593] status: saga.job.Job.Done
* Job [ssh://localhost]-[652594] status: saga.job.Job.Done
* Job [ssh://localhost]-[652595] status: saga.job.Job.Done
* Job [ssh://localhost]-[652596] status: saga.job.Job.Done
* Copying sftp://localhost//scratch/0000/train115/mbrot//tile_x0_y0.png back to /home1/0000/train115
* Copying sftp://localhost//scratch/0000/train115/mbrot//tile_x0_y1.png back to /home1/0000/train115
* Copying sftp://localhost//scratch/0000/train115/mbrot//tile_x1_y1.png back to /home1/0000/train115
* Copying sftp://localhost//scratch/0000/train115/mbrot//tile_x1_y0.png back to /home1/0000/train115
* Stitching together the whole fractal: mandelbrot_full.gif
Back: [Tutorial Home](SAGA Tutorial)