Proposal #6: persist consumers' log and status

Current status

At the moment, completed consumers info are only stored in memory. There are various drawbacks in this behaviour, such as:

• info are lost on geobatch restart
• memory is wasted on unused resources
• only a (low) finite number of consumer info instances are retained at any time.

To reduce this problem, some solutions are already in place (such as having a definite number of successfully completed consumers, keeping the most recent ones and discarding the older ones), but the failed instances are kept in memory until the administrator reviews and purges them; when the failed instances fill a configurable number of maximum retained instances, new instances will not be allocated and the flow will not accept new inputs.

Proposal

GeoBatch already handles a temporary work directory for each consumer instance. In there, there's a working subdir for every action in the flow, where the action running for that specific consumer instance may store it own temporary data. (When are removed these dirs?)

The idea is to store inside the temp dir of the consumer instance some files, namely:

• a file denoting the current status of the related consumer (running, completed, failed)
• the full log file (now only in the geobatch full log)
• the event log file (the info now readable in the geobatch UI)

In this way, GeoBatch can remove from memory any information about completed consumers (both failed and successful). The GUI will show only the currently running consumers, and a notice if there are any unchecked failures (GB could scan all existing consumer dirs at boot time). Then, in a different page, the admin can look the completed consumers list, examine their logs, mark failed runs as checked (in order not to have warning about them, leaving the logs where they are), remove the dir from the filesystem.