Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The OID of the put object can be saved into index file immediatly in upload-bin #3655

Open
AliceInHunterland opened this issue Nov 2, 2024 · 3 comments
Labels
cli Command line interface enhancement Improving existing functionality I2 Regular impact S3 Minimally significant U3 Regular
Milestone

Comments

@AliceInHunterland
Copy link
Contributor

Is your feature request related to a problem? Please describe.

I'm always frustrated when it appears that OID can be calculated in the uploadObj function. So we do not need to perform lots of searches for OIDs after the main batch of blocks is done.

Describe the solution you'd like

Return OID from uploadObj and save it into the index file right away. Ref. a0a0358

Describe alternatives you've considered

Leave as is.

@AliceInHunterland AliceInHunterland added I2 Regular impact U3 Regular enhancement Improving existing functionality cli Command line interface S3 Minimally significant labels Nov 2, 2024
@AnnaShaleva
Copy link
Member

Consider #3650 (comment) during implementation.

@AliceInHunterland
Copy link
Contributor Author

Consider #3650 (comment) during implementation.

And uploadIndexFiles can't be removed anyway, we must consider cases when you don't have all OIDs in-place.

In a0a0358 uploadIndexFiles is not removed but moved to the first place.

  1. Check and upload index files using uploadIndexFiles, and return a partially filled buffer (if oldestMissingBlockIndex\ indexFileSize!=0).
  2. Upload blocks from oldestMissingBlockIndex and when the buffer is full - upload the index file.

What do you mean by cases when you don't have all OIDs in-place.?

@AnnaShaleva
Copy link
Member

is not removed but moved to the first place.

OK. It needs some care.

What do you mean by cases when you don't have all OIDs in-place.?

E.g. uploading index file without block object uploads. Or with partially uploaded blocks. I'm not saying that your commit doesn't contain this code, but such edge cases must be handled with care and we must ensure that script works properly for such edge cases.

@AnnaShaleva AnnaShaleva added this to the v0.108.0 milestone Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli Command line interface enhancement Improving existing functionality I2 Regular impact S3 Minimally significant U3 Regular
Projects
None yet
Development

No branches or pull requests

2 participants