WARN [UPLOAD] Upload failed: *trace.ConnectionProblemError uploader failed to reserve upload part

[aqiank]

Hi, one of my Teleport nodes suddenly keeps failing to upload sessions after I upgrade all my instances to 16.0.4. I don't believe there was any ongoing session for the particular node at the time of upgrade and there wasn't any changes on the node that I'm aware of. What could be the issue? Thanks!

2024-07-06T04:00:00Z WARN [UPLOAD]    Upload failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError uploader failed to reserve upload part
Stack Trace:
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:537 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
        github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:464 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func2
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: uploader failed to reserve upload part] session:1a194b76-2b4e-45a9-a2cb-7569feb9792f filesessions/fileasync.go:465
2024-07-06T04:00:00Z WARN [UPLOAD]    Increasing session upload backoff due to error, will retry after 1m42.698478511s. filesessions/fileasync.go:228

[zmb3]

How's disk space look on this node?

[aqiank]

It has 58G free space (5% out of 1.2T).