modeldb differences with cvode version 2 and 3

[nrnhines]

69 nrn-modeldb-ci models exhibit gout differences between current master and #1960 which replaces the built-in sundials version 2 code with the external/sundials version 3 repository. Most of the model differences are minor in the sense that result visualization with diffgout looks the same unless zooming by a large factor. However 10 models are different to a degree that requires some thought about the origin of the differences. The differences may involve different time steps, may reveal great sensitivity to the timing of action potentials, and may suggest more testing with tighter tolerances to verify that cvode2 and cvode3 converge. Discussion about when or if #1960 should be merged to master is solicited. The models selected for more thorough observation are:

93325/nfrost
187474/reducedhaymodel
6763/canavier1999
82849/pcell
19696/sc-pp
206267/GrC_FHF_ModelDB
116094/DendroDendriticInhibition
2487/mit4
32992/synchro-ca1
93326/ngetting

Subsequent comments detail the differences ovserved with these models.

[nrnhines]

$ diffgout master/93325/nfrost/gout cv3/93325/nfrost/gout
master/93325/nfrost/gout color 2
cv3/93325/nfrost/gout color 1
sim 1 graph 1 line 1 has 11261/11816 points
sim 1 graph 1 line 1 differs by 250

and expanding the view of the last few DSI cell voltage spikes starting at 85000ms

This is a 5 cell 21 NetCon model with cvode.atol() equal to 1e-3. Given that NetCon spike triggering by default has a resolution of dt at the time the trigger voltage is passed, it makes sense, before decreasing cvode tolerance, to see if second-order threshold detection has an effect on relative accuracy of the spikes. With cvode.condition_order(2) the last spike occurs at 87847.6 ms for cv3 and 88101.2 for master. Then if we also decrease tolerance to cvode.atol(1e-4),
the last spike occurs at 89777.3 and 89793.1 respectively. With a tolerance of 1e-5, the times are 89674.9 and 89660.9

[nrnhines]

$ diffgout master/187474/reducedhaymodel/gout cv3/187474/reducedhaymodel/gout
sim 1 graph 0 line 0 has 847/1924 points
sim 1 graph 0 line 0 differs by 79.9035

Why are there so many fewer points with cvode3?

Why is the line so stepwise when zooming in on the spike?

Why do the cvode versions differ in the details of a step?

In this model, cvode.atol() is 5e-5

The answer to the last two questions is that the resolution of the gout file data is 6 digits. E.g. there are 6 identical times at 10204.1 . The format of gout files needs to be changed to handle detailed time steps at large times. Note that the cvode3 trajectory in the simulator graph looks like

[nrnhines]

$ diffgout master/6763/canavier1999/gout cv3/6763/canavier1999/gout

The disturbing plots here are graphs 3 and 4

Turns out those are spaceplots and so are snapshots of the v and nai at the end of a run at 1e5 ms after many nmda synaptic events. For example the voltage trajectory at two locations is

[nrnhines]

$ diffgout master/82849/pcell/gout cv3/82849/pcell/gout
sim 1 graph 1 line 2 has 638/988 points
sim 1 graph 1 line 2 differs by 93.1144

In graph 1, the second spike location difference (358.95 and 378.186) needs explanation.

For the line with two spikes, the IClamp amplitude is 0.055
The second spike location as cvode.atol is decreased is

atol   master (steps)    cv3  (steps)
1e-3    358.95  (989)    378.186 (639)
1e-4    358.864 (1326)   358.704 (943)
1e-5    358.373 (1867)   358.613 (1327)