Currently, each file upload is a REST API call, which would always hit any reasonable REST API rate limit with many files workflows. The idea exposed in reanahub/reana-client#406 about uploading the whole analysis workspace to REANA seems like the long term solution for this issue:

Uploading too many files at the same might be problematic for the server. A good strategy to alleviate this can be packaging the whole directory in a tarball and upload it.

So to address this issue we have to:

• In any case, if something like this happens we need to provide a better error message
• Address upload: upload full directory when files are not specified reana-client#406

We could amend the reana-client to respect the rate limits a bit better on the client side:

• when uploading directories with many files, do some sleep() between them because we know this can cause troubles;
• even better, when seeing the rate limiter message in the response for any REST API call, do not give up but rather retry after N seconds.

Currently, none of the Invenio App rate-limiting configurations is set (see here). This causes the default rate-limiting values to be loaded causing the problem described in this issue.

We could either use Invenio-App's capability of rate-limiting or looking into Traefik's rate-limiting.

To avoid the problem at hand we could have, apart from the REANA-Client fix, no rate-limiting by default in the server side, setting it only on demand if provided as Helm config.