Fix when resque failure backend is already multiple #803
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goal
We had an outage today after recently introducing Bugsnag into our codebase because our Resque failure backend was already
Resque::Failure::Multiple
, which this code looks like it handles, except for a small mistake with the operator:The backend is likely to be the
Resque::Failure::Multiple
class, not a sub-class.This meant that when the bugsnag instrumentation code ran and we ended up with:
So a failure was reported to Redis, then Bugsnag, then Redis, then Bugsnag, then Redis, then Bugsnag, and so on, until we got a "stack overflow" error.
For us, we had a worker which needed cleanup during resque boot, which involves reporting a failure, so none of our resque workers would boot or process work, resulting in an outage of all background queue processing.
Testing
The tests here are pretty literal and testing the implementation more than the outcome. It might be possible to refactor them to test the Resque side of things a little more, but that's a little more than I'd like to chew off in this PR.
We are currently using
BUGSNAG_DISABLE_AUTOCONFIGURE
to work around, and would love to get this change merged and released quickly so we can use Bugsnag error reporting for Resque 🙏