-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ListObjectVersionsPagesWithContext can get into an infinite loop when using URL encoding #4672
Comments
Hi @clumio-nick, Thanks for letting us know. I have ran your code and confirm that this is an issue. If a user passes the key in this encoded form then S3 will return the first page again (does not continue pagination). To give you a little background: all the SDKs are code generated from the various AWS service models, this should have been specified correctly in the service's model. Since upstreaming this to the service might take a very long time, I think we might need to handwrite the paginator which will also take a while. In the meantime my only suggestion to you is not to use the Sorry for the inconvenience. All the best, |
Hi @clumio-nick , I apologize for not getting to this sooner. If this is still an issue with the Go SDK v2, please open a new issue there and I will take a look. Thanks again, |
Comments on closed issues are hard for our team to see. |
Describe the bug
When EncodingType is set to s3.EncodingTypeUrl, ListObjectVersionsPagesWithContext can call "fn" repeatedly with the same keys. It never reaches the end of the listing, and only stops if "fn" returns false.
This happens if there is an object key containing a backslash ("\").
Expected Behavior
I expect each call to "fn" to have different keys, and I expect the listing to eventually end even if "fn" always returns true.
Current Behavior
"fn" is called repeatedly with the same keys.
Reproduction Steps
Run this program as "go run main.go -bucket <your-s3-bucket> -prefix <prefix-with-no-objects>.
It will create two objects under the prefix and attempt to list them with ListObjectVersionsPagesWithContext.
However, it gets into an infinite loop returning the first object again and again.
Note: this program does not clean up the objects it created, you have to delete them manually.
Sample output:
The program
Possible Solution
It's possible that ListObjectVersionsPagesWithContext doesn't URL-decode NextKeyMarker from the ListObjectVersions response before using it as the KeyMarker for the next call to ListObjectVersions.
The URL-encoding of "\" is "%5C", and "%" comes before "\" lexicographically.
Additional Information/Context
No response
SDK version used
github.com/aws/aws-sdk-go v1.44.83
Environment details (Version of Go (
go version
)? OS name and version, etc.)go version go1.19.4 darwin/amd64, macOS Ventura 13.1
The text was updated successfully, but these errors were encountered: