rkyv support?

[ahayworth]

Hello!

I've started implementing support for rkyv, a zero-copy deserialization framework - my use-cases entail mmaping data in from disk, and rkyv allows me to do that that quite efficiently. My initial work is here.

Would you be interested in a PR that adds support for rkyv more generally? If so, I'd be happy to clean up my initial work and contribute it here for everyone to use. I've found it surprisingly easy to do so far (I think it speaks to the good design choices made here) and would be pleased to do so if it's a feature you want.

(If you do want this feature, I'd be interested in your thoughts as to the general direction I've taken with the initial bit of work; if that's not the general design pattern you'd like to see then I can adjust.)

[kampersanda]

Hi @ahayworth,

Thank you for this issue and for letting me know about rkyv! I wasn't aware of this great library before.

Yes, I'm interested in the mmap implementation. It would open up new use cases for sucds. However, since sucds is still under development, I'd like to avoid duplicating code as much as possible.

Is it possible to implement code that reuses existing code?

[ahayworth]

👋 The Christmas holidays took me away from GitHub a little longer than expected! 🎄

I'd like to avoid duplicating code as much as possible.
Is it possible to implement code that reuses existing code?

The short answer is - yes, I believe it's possible; I'll experiment a little bit and see if I can come up with something less duplicative. The design of rkyv does make that a little difficult, as noted by the authors of epserde (a "basically zero-copy" alternative).

[somethingelseentirely]

Since succinct datastructures generally lack pointers and are often static/write-once, it seems like there isn't that much of a need for using a serialization framework?

An API to safely cast the datastructures from/to bytes::Bytes or &[u8] should suffice and might be agnostic enough that the other serialization libraries could also be supported through shims.

[ahayworth]

An API to safely cast the datastructures from/to bytes::Bytes or &[u8] should suffice and might be agnostic enough that the other serialization libraries could also be supported through shims.

Generally speaking, yes I would agree.

(Also: I have not had any time whatsoever to work on this, unfortunately. The delay this time is partially due to a new baby! The delay is also due to work responsibilities shifting away from the initial reasons I wrote out the rkyv support in the first place. If anyone wants to carry on with this work, please feel free to use my branch as a starting point if it suits you; I'm not going to be able to get to it anytime soon. ❤️ )