Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io.ReadAll consumes considerable amount of memory when searching 10K hits #1623

Open
dilshandcs opened this issue Jun 16, 2022 · 0 comments
Open

Comments

@dilshandcs
Copy link

dilshandcs commented Jun 16, 2022

Please use the following questions as a guideline to help me answer
your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[X ] elastic.v7 (for Elasticsearch 7.x)
[ ] elastic.v6 (for Elasticsearch 6.x)
[ ] elastic.v5 (for Elasticsearch 5.x)
[ ] elastic.v3 (for Elasticsearch 2.x)
[ ] elastic.v2 (for Elasticsearch 1.x)
olivere/elastic/v7 --> v7.0.21

Please describe the expected behavior

I'm doing a performance test of an endpoint which is trying to get 10000 hits from ES. The profiling give me a high memory usage. But I am expecting a reasonable memory consumption

Please describe the actual behavior

pprof tool shows me that io.ReadAll is using considerable amount of memory when searching 10K hits from ES. This bound our application to a certain amount of concurrent requests due to the memory overhead. See the following part from the memory profile

image

I used the io.Copy instead of io.ReadAll and tried the same amount of concurrent requests. This improves the situation significantly. see the profile below

image

See the chages I made to response.go file below
image

According to above example, io.ReadAll has an allocated memory space of around 145MB while other has only around 57MB. I think this will significantly improve the memory usage of any application, and this is an issue we are facing as well. Shall I create a PR with this change?

Any steps to reproduce the behavior?

Difference of memory usage between io.Copy vs io.ReadAll can be compared when searching for 10K hits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant