-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory Leak in case of "IncompleteMessage: connection closed before message completed" #3610
Comments
In case of
This magically made memory graph horizontal, no leakage. No clue why this is happening. |
@satyamagarwal249 thank you for the write-up. Can you help to post the minimal code for the client too, so that we can easily reproduce this? |
Also, from your last comment, it seems like it has something to do with the io you're using. Is it possible that the implementation of io in your code is causing the leakage? |
Hi, We are using hyper v0.14.26 and notice memory leak with a similar pattern in our services. Just want to do a +1 and check if there are any new updates related to this issue |
Version
hyper 0.14.28
tokio 1.35.0
tokio-openssl 0.6.3
Description
Observing memory increase for "Incomplete Message error".
testing secario:
server.go:
configs:
- http1 only
- disable pooling for simplicity: max_idle per pool=0
driving_client.rs
Client is always getting connection reset by server. This is leading to two types of error response in hyper:
(A)
hyper::Error(BodyWrite, Os { code: 104, kind: ConnectionReset, message: \"Connection reset by peer\" }))
(B)
hyper::Error(IncompleteMessage)
where (A) does not seems to cause memory leakage, while (B) seems to.
I observed that:
When I run my tests (15mins):
In high bandwidth (around 5MBps),
In low bandwidth scenario (100KBps),
see memory growth of case 1 vs case2:
hyper::Error(Connect, SSL(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: \"SSL routines\", function: \"tls_post_process_server_certificate\", reason: \"certificate verify failed\", file: \"../ssl/statem/statem_clnt.c\", line: 1883 }]))) }))
and it kept retrying. memory was constant: no leakage. It seems such leakage only when some data flushing happens.Note:
malloc_trim calls seems to clean the leakage to great extent.
This was I tried to reproduce locally on my laptop where for just 1MB request, in few minutes 100MB of leakage was observed (where graph is continuously increasing). I have seen similar scenario in production, where it lead to around 6GB of memory leakage, which is what the main concern is.
See in above graph, where the memory shooted to 6GB for our app (ideally our app consumes 1 to 1.4GB max at a time). On debugging, I found we received 157310 instances of 503 errors from S3 (rate-limit by them). our each request was around 28KB and when I plotted timeseries graph of error blocks size accumulated, the growth and curvature of graph was completely matching.
Note that sudden dip of 1GB in memory at 15:54:30 is due to
malloc_trim
call. Above run was on EC2, wheremalloc_trim
cleared only 1GB out of leaked 5GB, while on local runs on ubunut laptopmalloc_trim
seems to clear all leakage due to error blocks.Overall observation is: when there is connection reset by server while some data is already flushed by hyper client, there is possibilty of memory leakage based on some racing condition of when/how hyper gets to know about conncetion closure.
like Error A, was received by hyper during
poll_write
, which makeAsyncWrite
call to n/w: tokio_openssl::SslStream:poll_write.Where os informed about connection reset. This case does not seems to be causing memory/resource leak.
logs
poll_read
, which makesAsyncRead
call to n/w: tokio_openssl::SslStream:poll_read. Where in 0 bytes response indicates that connection closed while some message was expected. This case seems to be causing memory/resouce leak. It also errored on shutdown as:error shutting down IO: Transport endpoint is not connected (os error 107)
logs
the buffered data on closed connection is causing leakage some where in some scenarios (
incomplete message
is one of them). I am still in process of figuring out how it cleans on closed connection.As the errored blocks seems to get accumulated in memory and is unbounded, this issue can cause serious troubles. As an Application will keep consuming unbounded memory (in error scenarios) may suddenly crash.
The text was updated successfully, but these errors were encountered: