You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
First off, thanks for putting together and supporting such a useful utility, @toluaina!
I saw a comment you'd made on Stack Overflow about the possibility of extending pgsync to work with Weaviate. I'm considering doing this because, IMO, pgsync is the simplest option in its class in terms of implementation and predictability. Frankly, the async and "eventually" consistent approaches in some of the other out-of-the-box tools make me very uneasy as they seem very difficult to reason about and introduce all sorts of data race conditions.
So, the reason I'm starting this discussion is because I'd be curious to know just how much of an undertaking you think this might be? Hours? Days? Weeks? It seems like it should be pretty straightforward (the MVP would be to dispatch to either Elastic/Open Search or Weaviate using an env var; implement a base class which mimics the ES/OS one already in place; create a Weaviate subclass) but, of course, the devil is always in the details. Also, is it something you'd consider pulling in upstream?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
First off, thanks for putting together and supporting such a useful utility, @toluaina!
I saw a comment you'd made on Stack Overflow about the possibility of extending pgsync to work with Weaviate. I'm considering doing this because, IMO, pgsync is the simplest option in its class in terms of implementation and predictability. Frankly, the async and "eventually" consistent approaches in some of the other out-of-the-box tools make me very uneasy as they seem very difficult to reason about and introduce all sorts of data race conditions.
So, the reason I'm starting this discussion is because I'd be curious to know just how much of an undertaking you think this might be? Hours? Days? Weeks? It seems like it should be pretty straightforward (the MVP would be to dispatch to either Elastic/Open Search or Weaviate using an env var; implement a base class which mimics the ES/OS one already in place; create a Weaviate subclass) but, of course, the devil is always in the details. Also, is it something you'd consider pulling in upstream?
Beta Was this translation helpful? Give feedback.
All reactions