Making request while in :disconnected state.

[lukebakken]

Hi everyone -

While diagnosing this error we found that this sequence of events would lead to a :disconnected error:

• Start an application using a large HTTP2 pool (we used a count of 64).
• Start making requests using that pool, but don't make enough requests to use all connections.
• After a period of time, the server sends GOAWAY due to inactivity, the state machine enters the :disconnected state which sets a 500ms timer to re-connect.
• Before the timer fires, another request is attempted and we hit this code

Once we figured out this sequence of events, we opened this issue to fix the FunctionClauseError. We are catching that exception and re-trying as a workaround. We've also lowered the HTTP2 pool size to reduce the chances of hitting that case. But, anyone could hit that case, so I'm wondering what should be done...

• Document it and other exceptional cases as potential places to re-try. "Hey, you should be ready to re-try your request after a delay if you hit :disconnected"
• Re-try it in the library. Something similar was rejected here - https://github.com/keathley/finch/pull/97

Thoughts? Thanks again for the hard work put into this library.

[keathley]

I'm not sure that we can safely retry the request for the user here. Well...we could. But we'd add a lot of complexity to do it. We'd need to have a heuristic that the user can tune for the number of retries that they can attempt, the back-off ratios, etc. So, if we're talking about retries, I'd like to leave that in the hands of the user.

I wonder if it would be better to solve this by not using a disconnected connection in the first place. That would mean updating the pooling logic to avoid connections that are in a disconnected state. This would definitely increase the amount of state to manage, but it would give us better control over the pool and would allow us to use smarter load balancing across connections in the future. This is the solution that I'm currently leaning towards.

[djbberko]

I know this is an old thread, but I've been running into this issue lately. I've also tried lowering the pool_size, but still getting regular :disconnected state. Is the solution to just retry?