RDF-star for git-like versioning of a repository?

[fkleedorfer]

I am looking for a good solution for versioning an RDF repository, ideally supporting git-like commit histories, branches/forks, merging, tagging, and materializing versions at a given commit. It has been proposed to use RDF-star for annotating triples in such a way as to allow versioning of a dataset.

Part of the solution would be to use RDF-star for a given repository and instead of adding/deleting triples: For any modification, one would add RDF-star triples which each quote a triple, state whether it is added or removed, and refer to an IRI representing the 'commit'. The commit IRI can in turn be linked to some commit metadata. (Let's call such RDF-star triples 'versioned triples')

In contemplating such a system, I hit a first roadblock at the location of the logic for it: With INSERT/DELETE WHERE, the client can manipulate triples without ever learning what difference is caused to the dataset in terms of triples added/removed. Thus, it is not possible that the client generates these versioned triples. Unless we want to limit the kinds of queries a client can make, the versioning system would have to be implemented by the repository.

Thus, if I took a stab at this: what would be the appropriate point in the RDF4J Repository stack to introduce this functionality? It would have to be at a point where deletions and additions have been determined but not yet written to the store. At that point, the triples would have to be converted into versioned triples and written to the store.

In addition to this rather specific question, any comments, pointers and ideas regarding versioning are appreciated!

[hmottestad]

We have what's called a NotifyingSail which is an interface that our storage layers implement to push notifications of which statements are added and which are removed. We use it in the ShaclSail and in the reasoners.

Here is an example: https://github.com/eclipse/rdf4j/blob/58bcc2f0e842a213450d59535c74b52137a51507/core/sail/inferencer/src/main/java/org/eclipse/rdf4j/sail/inferencer/fc/AbstractForwardChainingInferencerConnection.java

[fkleedorfer]

Ok good starting point, thanks!

Now whether a NotifyingSail/SailChangedListener or just a Sail is the right thing to use here depends on the interaction pattern. Inference never deletes triples, so if we did versioning using inference, we'd be adding it on top of a repository - I had not thought of it like that, but it does sound appealing. I am not sure it is possible, though.

[kenwenzel]

Our data platform enilink also uses a comparable approach to implement a change tracker:
https://github.com/enilink/enilink/blob/master/bundles/workbench/net.enilink.platform.workbench/src/main/java/net/enilink/platform/workbench/components/ChangeTracker.java

[fkleedorfer]

Tracking changes is one aspect of it. The code you provided has a lot of enilink dependencies so it's not so evident for me to see what it does, but it does give an idea

[kenwenzel]

The change tracker listens for changes on a statement level and groups them into change descriptions consisting of an agent, the transaction time as well the added/removed statements. The added and removed statements are described using reification but RDF-star could also be leveraged here.

[fkleedorfer]

After giving it some thought, this sounds very useful for my situation. Would you suggest using the enilink implementation directly or did you insinuate porting the functionality to RDF4J?
EDIT:
Just looked into enilink and found the RDF4J-based KOMMA framework, which the ChangeTracker is using, so it's not something I am going to be able to just drop into my RDF4J-spring application. So we would be talking about replicating that functionality in RDF4J itself, I guess. Would you be interested in sharing your experience with the approach so we can build something similar for RDF4J?

[fkleedorfer]

Here are a few thoughts on this, let me know what you think:

1. A change tracker that writes into the RDF store bloats the database and makes writes slower. Any ideas for mitigation?
2. A change tracker that writes to the file system (or some other sink) makes accessing the 'commits' harder for an application that is already designed to interact with RDF4J. However, application access to the commits is almost the whole point (show a change history, maybe materialize an older version).
3. All approaches to RDF diffing have to handle blank nodes somehow. How is it done in the enilink ChangeTracker?
4. If I am not interested in the change history of the whole quad store, but only certain objects (RDF molecules, if you will), is there an efficient way to extract the changes pertaining to that object from the commit history?
5. Follow-up question: Is there a data design approach optimized for tracking changes? Using a named graph for each 'object' we want to keep track of comes to mind.
6. You need to funnel transaction data (a user id, maybe some session id) to the change tracker. If everything is done at a higher level (KOMMA) then there may be ways to do it, but how could one do it with plain RDF4J?
7. How do you aggregate changes into one 'commit'? I see there is a time limit - have you considered using higher-level aggregation of modifications (such as an application-level concept of a 'commit' or a user session?
8. Knowing what you know from enilink/komma, at what level would you suggest to implement the change tracker? Is it the listener on the NotifyingSail?

[kenwenzel]

1. This could be true. But having the changes as RDF allows us to directly use the RDF4J-APIs and SPARQL without any custom code to create and query the change descriptions.
2. If this is deserved then RDF-HDT could be a good solution. Although formats like parquet may also work very well, they would require a well-designed format to manage RDF data.
3. Changes are only tracked on a statement level. There is no comparison between resources.
4. You can simply use SPARQL to query for change descriptions that contain reified statements regarding a certain subject. See also https://github.com/komma/komma/blob/master/bundles/vocab/net.enilink.vocab.rdfs/src/main/java/net/enilink/vocab/rdf/Statement.java
5. enilink currently uses an "audit" named graph for each named graph that is changed (enilink calls named graphs models and therefore has an audit model for each model that uses change tracking). We just prefix the model's URI with enilink:audit:.
6. A specific SAIL implementation could be parameterized with a Context object that could (dynamically) provide such information via getters.
7. The best approaches would be to either create a change for each transaction or to introduce some kind of interaction or session as you suggested. The change tracker in enilink is currently more or less a quick and dirty PoC.
8. In KOMMA it is currently just a listener that is injected into KOMMA's IDataManager (which is a wrapper for an RDF4J repository). In contrast to RDF4J itself its implementation also allows to track changes for remote repositories (although this could be slow as every statement needs to be verified in the worst case). For improving this situation a bit the change support has three modes of operation: https://github.com/komma/komma/blob/ec9ee4eab7d1b2742c051ac2d686b1dd909cf060/bundles/core/net.enilink.komma.dm/src/main/java/net/enilink/komma/dm/change/IDataChangeSupport.java#L15
   The default mode is VERIFY_NONE but this can't be used for undo/redo scenarios.