Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
materialize-postgres: limit the absolute number of documents in a que…
…ued batch There is a crude limit on batch size based on queued document sizes already, but it seems that very small documents (particularly small load keys) have a high ratio of "overhead" to their actual data size, and that can result in excessive memory usage if there are lot of these documents to process. This adds an absolute limit on the number of queries that can be queued at once, to prevent tiny documents from blowing up the connector's memory. A limit on number alone is not enough since documents can be huge, and so it seems that a limit on queued bytes is not enough since documents can be tiny.
- Loading branch information