Use Case: Provider/Subscriber Based Database Partnership

[boxfoot]

This is a use case as part of the Use Cases meta-issue (#44). It expands on @klambacher 's "Integrators" use case:

Integrators want to combine information into existing systems or datasets. They are committed to the system they use, and may have a substantial existing data set that is outside of the collection policy of the target dataset they want to integrate. They care deeply about ongoing maintenance and updates of the information and often move large quantities of information frequently. Over time, this can sometimes become a bi-directional sharing relationship if the initial integration is successful. Efficient ways to determine changes and keep data in sync between systems is what matters, and search features are generally irrelevant beyond the ability to query changes and do sanity checks. Efficient movement of data in bulk is important and pull-and-display APIs often require too many requests to collect information. "I'd like to integrate your information into our existing database, on this schedule"

Here's what folks are trying to achieve:

1. SUB is an organization that uses Human Services data. They maintain their own database, which they need to use for various operational purposes. PRO is another organization that maintains similar data. Rather than curate an entire database on their own, SUB wants to "subscribe" to the database from PRO as the core of their database.
2. The partnership agreements look something like this:
   • PRO and SUB agree to a scope covered by the subscription (e.g. resources in these taxonomy categories covering this geography)
   • SUB will mirror the relevant part of PRO's database into their own database
   • Whenever PRO updates, adds, or deletes relevant resources, they will notify SUB so that SUB can pull the changes into their copy.
   • Whenever SUB updates or adds a resource, they will submit the change to PRO as a suggested change. PRO will review the changes and either accept them or reject them. Ideally, PRO will notify SUB about the outcome of these updates, especially rejections (accepted edits will come through as the core data set regardless). SUB can submit these changes either as specific changes ("change phone number to X") or as flags ("phone number was disconnected, needs investigation")

[NeilMcKLogic]

@boxfoot this is indeed a very relevant topic and timely too. @kinlane had floated an earlier topic about Webhooks in HSDA spec which could be one component of this use case wherein SUB notifies PRO of a change (and vice-versa).

Have you had a chance to look at the currently proposed HSDA spec to see how well or not it will handle this scenario? I'm guessing not and we may want to tread our own path and bring it back to the spec group.

[tuckerbuchy]

👍 We have a similar need where we are importing a large number of records from a local 211 provider, and wish to remain in up to data through "nightly" data dumps and syncing.

[kinlane]

Adding this to list under the orchestration (evented, webhooks, notifications). Which will work with the meta data system to deliver all the functionality described above.