All or nothing data transactions

[nickpalladino]

We have a need to create and update data in an all or nothing manner, no partial updates in the case of network or system errors. It seems like a lot of BrAPI applications writing data would want a similar capability. Below are some cases where this has come up for us.

Batching Data

Due to server processing times and maximum message sizes requests can fail/timeout for calls with a lot of records, for example POST /germplasm. Up until this point data has been split into smaller batches on the client side and sent. However this naive approach can result in partially written data in the event of network or system errors. We allow uploading germplasm records from a spreadsheet file to a brapi service and with this approach we cannot guarantee all data in the file is created.

Aggregate Datasets

Another area where this comes up is with aggregate datasets. For example we allow creating experiments in a brapi service from spreadsheets. These experiments depend on a number of different BrAPI objects being created as a set: trial, study, observationunits, etc.

There are a lot of different possible approaches to dealing with this type of issue in a REST api. Examples include structuring endpoints so that single calls can be made, different types of transaction api schemes, compensating transactions, etc.

Has there been any work or discussion on this topic in the broader BrAPI community?

[nickpalladino]

Notes from discussion with Pete and Breeding Insight developers:

Example:
50K germplasm from DeltaBreed -> BrAPI Service
split into 5 x 10K germplasm chunks

1. POST /transactions (get a txid)
2. Make normal BrAPI data calls with txid to indicate they're part of a single tx
3. [failure case] POST /transaction/rollback/txid
4. [success case] POST /transaction/commit/txid

How Peter would implement in BJTS:

• Additional column (status, time input, txid?) for pending/uncommitted data
  • Would this mean additional field(s) on each BrAPI object?
  • Could this help with data purgatory/QC?
  • Status, timestamp would be useful for logical/soft deletes.
• Timed process within server (on startup, every 6 hours for example) rollback everything after 6 hours automatically
• If POST /transaction/rollback/txid is called, server would delete everything with this txid.
• If POST /transaction/commit/txid is called, server would update everything with this txid (change status).

Concerns:

• Implementation complexity on server side