CV2-3551 add local queue consumption and re-work a ton of the startup flow to accommodate

[DGaffney]

Alright, so a few things happening here as per conversations last week:

1. Break queue class into a parent queue class and two subclasses, worker and processor,
2. Worker is responsible for its own concept of an input/output queue, and nobody else needs to think about that idea,
3. Processor sits on any output queues matching a given input queue string, and fires callbacks,
4. and rework the makefile / starter shell file to run both workers simultaneously on start.

This should allow us to run everything end-to-end locally at this point. We should be able to just set MODEL_NAME= mean_tokens__Model and then start up the environment, and be able to operate full round trips

[skyemeedan]

not in this PR, but we should probably call this in parallel

[DGaffney]

Cosign

[skyemeedan]

probably want some (tuneable?) time interval in here?

[skyemeedan]

what happens when the worker dies?

[DGaffney]

This all happens in the context of sqs messages, so the message eventually gets tossed back into available messages and some other worker will pick it up - should be fairly foolproof as it just ties into existing foolproof frameworks...

[skyemeedan]

I don't think there will be dataloss because queues, but if the worker dies, we will stop pulling things off the queue while other parts of the system are still happily putting things in queues ... but will AWS know that the service needs to be restarted?

[DGaffney]

In our discussions so far, my understanding is that one of the big reasons to use SQS directly would be to make our system scale according to queue depth. As the queue grows, we would increase the number of workers to address. A hung machine would still potentially need to be taken out back individually, but it shouldn't arrest all work entirely.