Replies: 12 comments 3 replies
-
|
Hi ,
a few thoughts
* Have you tried using loglik_kind = "approx_differentiable"? There are
instances in which the analytic likelihood causes divergences but the LAN
does not even for the DDM (the gradient of navarro fuss behaves differently
than a neural net). I'm also curious if you will still bump up against
that odd upper bound of 0.542 on the t parameter that you described before
and I see in your posteriors here
* have you tried the centered specification? The funnel can sometimes be
avoided better with centered than non-centered depending on whether the
likelihood is narrow (roughly related to how many trials you have per
subject) - see here
<https://betanalpha.github.io/assets/case_studies/hierarchical_modeling.html>
for
details but I have confirmed that this is the case in hssm by varying trial
counts.
Michael
…On Mon, Jul 22, 2024 at 10:18 PM hyang336 ***@***.***> wrote:
I am fitting a DDM with the following model specification:
v_intercept_prior = {
"Intercept": {"name": "Normal", "mu": 1, "sigma": 2, "initval": 1},
"1|subj_idx": {"name": "Normal",
"mu": 0, # using non-centered approach so mu's of indiv subject offsets should be 0
"sigma": {"name": "HalfNormal",
"sigma": 1
}, "initval": 0.5
},
}
a_intercept_prior = {
"Intercept": {"name": "Gamma", "mu": 1.5, "sigma": 0.75, "initval": 1},
"1|subj_idx": {"name": "Normal",
"mu": 0,
"sigma": {"name": "HalfNormal",
"sigma": 1, "initval": 0.3
},
},
}
z_intercept_prior = {
"Intercept": {"name": "Beta", "alpha": 5, "beta": 5, "initval": .5},
"1|subj_idx": {"name": "Normal",
"mu": 0,
"sigma": {"name": "HalfNormal",
"sigma": 0.05, "initval": .01
},
},
}
model= hssm.HSSM(
data=data,
prior_settings="safe",
include=[
{
"name": "v",
"formula": "v ~ 1 + (1|subj_idx)",
"prior": v_intercept_prior,
"link": "identity",
},
{
"name": "a",
"formula": "a ~ 1 + (1|subj_idx)",
"prior": a_intercept_prior,
"link": "identity",
},
{
"name": "z",
"formula": "z ~ 1 + (1|subj_idx)",
"prior": z_intercept_prior,
"link": "identity",
}
],
)
The model was run with 4 chains, each having 10000 burn-in and 10000
samples.
The results showed a lot of divergence (in fact, all 40000 samples
diverged), but the posterior traces looked okay for more parameters:
posterior_diagnostic_10000_10000_TA_0.8_trace_median-binarized_t-strat_norandom_recent_v_on_null.png
(view on web)
<https://github.com/user-attachments/assets/c13060a6-bdad-417f-ba5b-f4f308c0392a>
To further diagnose the fit, I plotted for each individual subject
pair-plots of the parameters, which showed a funnel for z.
null_model_sub-1.png (view on web)
<https://github.com/user-attachments/assets/760faa5d-2a6e-4a71-a726-296090258e7c>
My questions are
1. Do other parameter pairs look normal?
2. why are all samples diverging?
3. how do I alleviate the funnel in z?
—
Reply to this email directly, view it on GitHub
<#514>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG7TFFA4WMRJ54A7VVTBL3ZNW4RFAVCNFSM6AAAAABLJNLYWCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWHE3DMNRQGM>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
The model with "approx_differentiable" just finished running. It did get rid of the divergence, in fact, the results reported 0 divergence. But I still get the warnings about rhat and ess: Also the posterior traces look worse than the analytical version, but t is no longer capped at 0.542. I am going to do a pair-plot for this model. Any other recommendation on diagnosing the issue will be highly appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the suggestion. I tried to use the approx_differentiable, but it seems like the code attempted to make a call to huggingface to download the model? I got the following error because I am running HSSM on a computing node with no internet access. Is there a way to pre-download the model and put it somewhere to avoid this? |
Beta Was this translation helpful? Give feedback.
-
|
Yes! You can download any model here https://huggingface.co/franklab/HSSM/tree/main, save it locally, and provide the path to the onnx file when creating the model. Just pass the path to the file to the loglik parameter |
Beta Was this translation helpful? Give feedback.
-
|
Looking at your posteriors it does seem like some of them are different
across chains which relates to the rhat statistic warning. We should sort
out why you had the 0.542 cap in the analytical version, that is quite
strange.
But before going further in diagnosing the problem it would be helpful to
confirm that the data you are modeling do actually conform to predictions
from a DDM and where the parameters should be separately identifiable. have
you looked at the shapes of the RT distributions and choice proportions? Do
you have separate task conditions or just fitting a single v for each
condition etc?
…On Mon, Jul 29, 2024 at 2:00 PM hyang336 ***@***.***> wrote:
The model with "approx_differentiable" just finished running. It did get
rid of the divergence, in fact, the results reported 0 divergence. But I
still get the warnings about rhat and ess:
The rhat statistic is larger than 1.01 for some parameters. This indicates problems during sampling. See https://arxiv.org/abs/1903.08008 for details
The effective sample size per chain is smaller than 100 for some parameters. A higher number is needed for reliable rhat and ess computation. See https://arxiv.org/abs/1903.08008 for details
Also the posterior traces look worse than the analytical version, but t is
no longer capped at 0.542. I am going to do a pair-plot for this model. Any
other recommendation on diagnosing the issue will be highly appreciated.
posterior_diagnostic_10000_10000_TA_0.8_trace_median-binarized_t-strat_norandom_recent_v_on_null.png
(view on web)
<https://github.com/user-attachments/assets/a00c15f3-8dc3-423a-abd6-521554a0114f>
—
Reply to this email directly, view it on GitHub
<#514 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG7TFCDXRXQ4OXXZ5FYUELZOZ7MLAVCNFSM6AAAAABLJNLYWCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJYGI3DGNY>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Here are the RT distribution for the two types of responses. A potential cause is that the behavioral paradigm used was not a binary choice but a rating on a 5-point Likert scale, I binarized the responses around each subject's median rating to fit the DDM after trying the race_4 model with no success. The RT distributions look okay to my eyes, so I thought this should be okay and would just increase the variability in each response type, but now I am less sure about that. I do have separate task conditions (i.e. two types of judgement), but the model fitting was always done within a single task condition. |
Beta Was this translation helpful? Give feedback.
-
|
OK I agree the distributions look ok (at least at the group level), though
it is possible that people treat the likert responses differently with
different speeds, and if the distributions for these choices are different
enough it would be hard to pin down a set of ddm parameters that fits them
all. This is also not something that would really be solved by a race
model (since that would still assume discrete choices whereas likert is
continuous). There are some other models of continuous process like the
circular diffusion model for color, which we hope to include on hssm soon.
Nevertheless in principle the model should still be able to converge even
if there is some collinearity in believable parameters. But if you have two
conditions it might help to fit the two conditions together with only one
parameter varying by condition (e.g. if condition varies difficulty then
probably v will vary by condition and fit the data better but also allow
you to better estimate v separately from a,z).
…On Tue, Jul 30, 2024 at 12:01 PM hyang336 ***@***.***> wrote:
Here are the RT distribution for the two types of responses. A potential
cause is that the behavioral paradigm used was not a binary choice but a
rating on a 5-point Likert scale, I binarized the responses around each
subject's median rating to fit the DDM after trying the race_4 model with
no success. The RT distributions look okay to my eyes, so I thought this
should be okay and would just increase the variability in each response
type, but now I am less sure about that. I do have separate task conditions
(i.e. two types of judgement), but the model fitting was always done within
a single task condition.
RT_distribution_median-binarized.png (view on web)
<https://github.com/user-attachments/assets/8938c94d-7a30-44ae-b287-987d27002788>
—
Reply to this email directly, view it on GitHub
<#514 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG7TFDHHGNTGOYAHWM6TBTZO62GPAVCNFSM6AAAAABLJNLYWCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJZGI2TSNQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for these insights. I think plotting the RT distributions of individual Likert options to see if they differ substantially between subjects may be a way to tell if that is causing the issue. Do you agree? I am not sure if combining the conditions would make sense in my case, because they are not the same task with different difficulties, but quite different tasks. Moreover, each task is associated with different brain activity (fMRI BOLD) in different regions, which will be regressors for the various DDM parameters. So combining the tasks will be hard. |
Beta Was this translation helpful? Give feedback.
-
|
ah i see, so you are looking to estimate a single set of ddm parameters
for all trials without varying difficulty. It could work but the design is
not optimal to tease apart parameters.
re: RTs for different likert options I think you don't want to compare them
between subjects but rather within subjects (and then take the mean of
those differences across subjects). otherwise overall variability between
subjects can mask differences within subjects (and ultimately the model is
predicting individuals)
…On Tue, Jul 30, 2024 at 4:56 PM hyang336 ***@***.***> wrote:
Thanks for these insights. I think plotting the RT distributions of
individual Likert options to see if they differ substantially between
subjects may be a way to tell if that is causing the issue. Do you agree?
I am not sure if combining the conditions would make sense in my case,
because they are not the same task with different difficulties, but quite
different tasks. Moreover, each task is associated with different brain
activity (fMRI BOLD) in different regions, which will be regressors for the
various DDM parameters. So combining the tasks will be hard.
—
Reply to this email directly, view it on GitHub
<#514 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG7TFCDILGQ65Y3FEDLC53ZO74WFAVCNFSM6AAAAABLJNLYWCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJZGQ3TMNI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Yeah I agree that the dataset is not optimal. It is more or less an ad hoc decision to model it with SSM. Regarding RTs, perhaps I am a bit confused. Is the main issue:
I guess 2 would affect how effective the pooling is in the hierarchical model. But for 1, if we assume that after binning responses, the RT distribution is still unimodal, wouldn't it just increase the variance? Is the issue that if a response option (after binning) has too much variance in RT it can no longer constrain the parameter estimates? Or maybe I am not thinking about this correctly... |
Beta Was this translation helpful? Give feedback.
-
|
If the binned response for each subject has an exgaussian shape that is
unimodal I agree it should be possible for a DDM to find parameters that
fit it. But a unimodal distribution when averaging across all subjects
doesnt mean it is unimodal for individuals.
…On Tue, Jul 30, 2024 at 6:48 PM hyang336 ***@***.***> wrote:
Yeah I agree that the dataset is not optimal. It is more or less an ad hoc
decision to model it with SSM.
Regarding RTs, perhaps I am a bit confused. Is the main issue:
1. different response options having different RT distributions within
subject, or
2. different subjects having different RT distributions for the same
response option?
I guess 2 would affect how effective the pooling is in the hierarchical
model. But for 1, if we assume that after binning responses, the RT
distribution is still unimodal, wouldn't it just increase the variance? Is
the issue that if a response option (after binning) has too much variance
in RT it can no longer constrain the parameter estimates? Or maybe I am not
thinking about this correctly...
—
Reply to this email directly, view it on GitHub
<#514 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG7TFE6FSRJAR2S6K6YNR3ZPAJ25AVCNFSM6AAAAABLJNLYWCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJZGUZTINA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I ran a Hartigans' dip test for unimodality on the RT distribution for each of the two response options in each subject (~45 trials). Only one subject had marginally significant result for one response option (p = .062). And the RT histogram for that response option for that subject looks like this: These are based on a relatively small number of trials but I think it suggests that the distribution are mostly unimodal at individual subject level. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @hyang336, apart from the funnel on Being hard bound by One thing that the trace plots make me think about is that funnels can actually be parameter-specific, and we should figure out how to make parameterizations I am a priori not sure if that's currently possible in Bambi, will do some due diligence on that. If yes we will push the functionality through, if not we are going to bring it up with the Bambi folks (@tomicapretto :)). |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Alex, Thanks for the insights. I haven't tried the centered parameterization yet, as I wanted to be reasonably sure first that the data itself does not have weird distributions that DDM cannot model. I thought the centered/non-centered mode could be adjusted solely based on how one specify the priors. And given that the prior of each parameter is specified separately, one can use centered for some parameters and non-centered for others. But now I am not so sure I understand how one can choose to use centered vs. non-centered parameterization in HSSM, could you give an example for one parameter? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I just realized that Michael has actually described how to do this in my issue post here :#471, in case someone run into similar issues. |
Beta Was this translation helpful? Give feedback.
-
I am fitting a DDM with the following model specification:
The model was run with 4 chains, each having 10000 burn-in and 10000 samples.
The results showed a lot of divergence (in fact, all 40000 samples diverged), but the posterior traces looked okay for more parameters:
To further diagnose the fit, I plotted for each individual subject pair-plots of the parameters, which showed a funnel for z.
My questions are
Beta Was this translation helpful? Give feedback.
All reactions