-
Notifications
You must be signed in to change notification settings - Fork 14
SyncAndRestoreAPI
- CommCare 2.12 - Added the concept of "extension" indices affecting syncing
- CommCare 2.35 - Added new optional parameters
device_id,user_idto the sync request to facilitate edge cases - CommCare 2.38 - OpenRosa Request header should now default to 2.1 to enable the receipt of multiple indexed fixtures.
There are two modes with which a remote device will interact with a server's Case database, restore mode, and sync mode. restore mode presumes that the server will only need to push back a full set of a user's previous case submissions (if a phone is lost, and needs to be replaced, for instance). sync mode presumes that the server is potentially pushing down cases to users which they had not seen before, reassigned from other users, or created from an external system. Essentially, restore mode is identical to sync mode, merely only performing the base case synchronization from scratch.
Case data is submitted from devices to the server via JavaRosa Form HTTP submissions of XForms containing Case XML transactions. Servers are expected to parse these blocks as outlined in the specification and maintain a repository of the cases which are maintained there.
When a user syncs with the server, uses their CommCare application, or logs into the sever to use an online CommCare interaction, there exists a set of cases that should be "available" to that user. This (roughly) includes cases that are owned by that user, or cases that are derivative to cases that the user owns. It will not include all cases (even if, for instance, the user was connecting on a web browser to a server storing all cases). This set of cases is referred to as the user's scope.
When syncing the server and phone both determine the current user's scope. They then identify what is currently on each device, what cases should be in scope and out of scope, and what changes need to be made to reconcile the scope so that a user on any device "sees" the same set of data.
This document outlines both how the user's current scope is determined and how a remote device and the central server should negotiate to reconcile them and receive updates for all cases in that scope.
The first aspect of scope is "ownership". All cases have the concept of an "owner", which is either an individual user or a group of users.
Case ownership can be determined in two ways. Direct ownership or group ownership. In direct ownership, the owner of a case is specified by setting the owner_id of the case to be the id of a registered user. In group ownership, the server defines a group which has an id, and is associated with a set of user id's. If the owner_id is set to that of a group, each user id associated with a group id is considered an owner of the case.
The groups an individual user is a part of should be defined on the server, and sent to the phone during any restore actions as defined below. The groups are sent down as a fixture associated with an individual's user_id.
<fixture id="user-groups" user_id="ID_OF_RESTORING_USER">
<groups>
<group id="ID_OF_GROUP">
<name>NAME_OF_GROUP</name>
</group>
<group id="ID_OF_GROUP">
<name>NAME_OF_GROUP</name>
</group>
...
</groups>
</fixture>Case indices are the second component of identifying a user's scope. A case index can work in two directions.
If a case has child index from a case C to a case P this outlines that case C relies on the existence of case P to function correctly. This means that as long as case C is in the user's scope that case P will be in scope well, regardless of whether case P is owned by the user, and even if case P is closed.
If, conversely, a case has an extension index to another case, this signifies that the extension case E is only meaningful in the context of the host case H, and that the absence of the host case should also result in the absence of the extension case. If case H is not relevant in the user's scope, E will be removed. If H is in scope and E is open, E will be included as well.
Although a case can have multiple named indices to another case, it can only have one relationship with that case in order for a reasonably deterministic sync to take place. The child index relationship takes precedence to the extension relationship, so if case C1 has a child index to case C2 [IC12] and has an extension index to that case [IE12] the retention table for those cases is
| C1 Alive | C1 Dead | |
| C2 Alive | retained(C1, C2) | retained(C1, C2) |
| C2 Dead | retained(C1) released(C2) | released (C1, C2) |
Both the server and remote devices should have a consistent set of cases which are relevant to the user. In order to determine what cases to send over or manipulate, first the set of relevant cases for a user is defined, then it is identified which need to be communicated.
The first time that a device requests a sync from the server, the server identifies the full set of data to be transmitted to the phone using the following process. After any manipulations of case data, clients should ensure that the scope of data seen by the user is consistent with the outcome of this process.
The set of all owned cases for a user cuo is defined as all cases, without an extension index, whose owner is the user or a group that the user is a member of. The full universe of cases which are in the user's scope, cu, is recursively defined as follows:
- A case is in cu if it is in cuo. "All cases that belong to a user are in its scope."
- A case is in cu if a case that indexes it with a child relationship is in cu. "Child cases pull their parents into scope."
- A case is in cu if it indexes a case in cu with an extension relationship. "Cases pull their extension cases into scope."
(Practically you start with the first step and then alternate between the second two until neither of them changes the set.)
This forms a set of directed graphs with two kinds of special nodes. Root nodes, which are defined as nodes which themselves maintain no outgoing indices, and Leaf nodes, which are nodes which have no incoming indices. The algorithm itself assumes that the generated graph(s) will be acyclical.
Once cu is established, it is pared down to contain only live cases. live cases are determined from the following process. Note that a case can only bear one mark at a time and the result of a pass should only be able to set the mark to one value.
- Initialize the mark for each case:
- Closed are marked dead
- Open cases with an extension index are marked abandoned
- Open cases without an extension index are marked live
- Traverse all cases starting at the leaves, moving breadth first. For every case marked live each case it indexes with a child relationship is marked live. "Children pull in their parents."
- Traverse all cases in reverse starting at the roots, depth first. For every case marked live each case that indexes it with an extension relationship is marked as live if currently marked abandoned. "Cases pull in their open extension cases."
cul is the set of all cases (in cu) marked live at the end of this process.
The initial sync to the device consists of the <registration/> for the user who is requesting the data restore, <case/> transactions for all cases in cul, the <fixture/> for the groups the user is in if relevant, and a <sync/> payload as defined below, containing ut0, the starting sync token.
If the phone will continue synchronizing with the server, each form that it sends will include the HTTP header
| Header | Values |
|---|---|
| X-CommCareHQ-LastSyncToken | utN - the last sync token received when this form was completed. |
When the next synchronization action occurs, the user will submit utN (the last sync token received from a successful sync). This signals that the server should only send down transactions clN+1 defined by
[
clN (The set of cases that the user should have after the last sync, N)
+
cld_N (The set of cases that were changed only by the user in transactions tagged by utN)
]
+
clN+1 (The set of cases which were modified by the server and need to be updated on the phone)
->
cul (The current set of cases consider to be authoritative)
As well as any modifications to the user registration, or the group fixtures.
If the phone fails to process a sync action (WLOG referred to as utN+1), often due to a failed download of the restore payload, when it attempts to sync in the future, it will include the last successful token (utN), and will continue to use that token in submissions up until another sync action. A CommCare server must be prepared to generate a new payload (clN+1_2) and a new token (utN+1_2) for a previously existing token (utN) until it identifies a submission or sync action with the new token submitted. If the existing token (utN) is submitted again, it can be presumed that any new token (utN+1_n) can be discarded, along with any state associated with it.
The payload to be returned by the server to tag the current sync with a sync token should be the following, occurring at the top of the envelope before any case, fixture, or user registration transactions.
<sync xmlns="http://commcarehq.org/sync">
<restore_id>SYNC_TOKEN_HERE</restore_id>
</sync>Special care should be taken when processing case attachments as outlined on the appropriate API page
A device additionally requests a confirmation from a server about the what the current set cul should be when it is requesting a sync. It does so by producing a digest of the current live cases on the phone and submitting it to be verified on the server.
The digest is computed by first establishing a consistent set cul on the device by purging any cases from the phone which are not live.
The set of cases on the device should now consist of the join of the two sets
clN (Case set after the last sync) cld_N (Case transactions from the device since the last sync)
The devices will then iterate through the cases to generate the digest
digest = 0x00
for all case c:
digest = digest ⊕MD5(c.caseid)
The server can then recreate the set (clN, cld_N) and the associated digest and verify that the state is the same before producing and sending the next set of transactions and state tokens.
The digest should be submitted with the "CommCare State Hash" prefix:
ccsh:[digest]
If the state is not identical, the server should respond with an HTTP response code 412 for Precondition Failed. The server should not send down any case blocks as part of this transaction. At this point it is the phone's responsibility to reconcile the case lists, typically by submitting any unsent case transactions, submitting logs, purging its case database, and starting from scratch with the server's state.
Compatible servers should provide a URL which can be be queried with a GET request. The request should be capable of accepting any of the following URL parameters
| Parameter | Values | Optional? |
|---|---|---|
since |
utN - the last sync token received | Yes - Defaults to sync from scratch |
state |
stN - the state token for the phone's current state | Yes - Defaults to not checking the state |
version |
Case XML Spec version (1.0, 2.0, etc) | Yes - Defaults to 1.0 |
device_id |
unique uuid id for the "device" (IMEI, MAC) requesting a restore | yes |
user_id |
The ID of the restoring user. Helpful for resolving ambiguity | yes |
The server should respond with an XML payload as specified in the OpenRosaRequest API with the appropriate transactions for the sync action.
See the OpenRosaRequest page for a list of changes and standard versions.