HSSM estimating problem - Inconsistent results.

[JoeSu112]

Hi all,
I’m now using HSSM to fit my experiment data into hierarchical drift-diffusion model (HDDM).
Here is example of my data:

subject	treatment_will	month_delayed	pay_diff	response	rt
1	1	4	20	1	3.556
1	1	3	10	0	2.765
1	1	1	15	1	3.456
2	0	4	20	0	4.678

Here is code for my model:

HDDM = hssm.HSSM(
    data=data,
    model="ddm",
    include=[
        {
            "name": "v",
            "prior": {
                "Intercept": {"name": "Normal", "mu": 0, "sigma": 3},
                "month_delayed": {"name": "Normal", "mu": 0, "sigma": 15},
                "pay_diff": {"name": "Normal", "mu": 0, "sigma": 15},
            },
            "formula": "v ~ 1 + (month_delayed|treatment_will) + (pay_diff|treatment_will)",
            "link": "identity",
        },
	  {
            "name": "a",
            "formula": "a ~ 0 + (1|subject) + (1|treatment_will)",
            "link": "identity",
        },
        {
            "name": "z",
            "formula": "z ~ 0 + (1|subject) + (1|treatment_will)",
            "link": "identity",
        },
    ],
)

Here is code for the posterior sampling:

infer_HDDM = HDDM.sample(
    sampler="mcmc",  # type of sampler to choose,
    cores=2,
    chains=4,
    draws=100000,
    tune=20000, 
    idata_kwargs=dict(log_likelihood=False), 
) 

However, I am encountering a problem that even I increase the posterior samples (draws=10000 -> 30000 -> 50000 -> 100000) , the parameters are still inconsistent in every posterior sampling. For example, the parameter "v_month_delayed|treatment_will[0]" from last 3 sampling are 0.226., -0.004, -0.169 and the rhat is higher than 1.1 also.

I am wondering how to solve this inconsistency. Should I keep increasing samples or there are other solutions?Thanks for any help!

[AlexanderFengler]

Could you post the output of az.plot_trace() function here as well?
What are the values of the other parameters also?

This might just be related to current issues with chain initialization, which we are working to fix in general.

[frankmj]

Also: * It is usually a good idea to use mean-centered (ideally, z-scored) regressors (e.g. for month-delayed and pay, transform this to z-score). Probably better for treatment_will to be -1,1 instead of 0,1. This makes it easier to interpret the coefficients and any interactions but also allows you to use priors that are consistent with the range of values expected for the parameters * The priors that you specify are quite wide. eg. a sigma of 15 on drift rate on top of a regressor that goes up to 20 or 4 means that it will consider drift rates that can be in the hundreds... whereas typical drift rates almost never go larger than 4 or 5. This issue should be partly helped by z-scoring the regressors, but you could also narrow your priors so the sampler doesn't have to consider many very unreasonable values and get stuck. * You should not need 100K samples to get convergence - I would first try some of these issues to debug it with lower samples. As Alex says, some of this might relate to chain initialization but any such issues are exaggerated when the range of values possible is so large. Michael Michael J Frank, PhD | Edgar L. Marston Professor Director, Carney Center for Computational Brain Science <https://www.brown.edu/carney/ccbs> Laboratory of Neural Computation and Cognition <https://www.lnccbrown.com/> Brown University website <http://ski.clps.brown.edu>

…

On Sun, Oct 22, 2023 at 10:15 PM Alexander Fengler ***@***.***> wrote: Could you post the output of az.plot_trace() function here as well? What are the values of the other parameters also? This might just be related to current issues with chain initialization, which we are working to fix in general. — Reply to this email directly, view it on GitHub <#297 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG7TFEZS4ZWCR5WY5OOGA3YAXHLPAVCNFSM6AAAAAA55ZGKACVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNJUGA3DE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[JamesWeiChen]

Dear Michael,

I have the same problem below. Could you help us? Thanks a lot~

Best,

James

[JoeSu112]

Hi @frankmj @AlexanderFengler,
Thanks for your replies and suggestions. I've followed your advices:

1. Let treatment_will be -1,1 instead of 0,1.
2. Transform month-delayed and pay into z-score.
3. Change the sigma of priors of the drift rate from 15 to 3.
4. Samples are 20k instead of 100k.

However, after these corrections, the parameters a and z are still inconsistent.
Here is the trace of a and z:

Is this because the chain initialization problem you've mentioned before? If it is, is there any solutions to the problem?
Thanks for any help!

[JoeSu112]

Hi,
Do you have any more ideas or suggestions on this problem? If you have any advice or method that might work, please let me know!
Thanks again for your help!!!

[frankmj]

Hi James and Joe, I've been informed that we are very close to rolling out a new release that should fix these issues - hold tight! M

…

On Tue, Nov 21, 2023 at 9:52 AM James ***@***.***> wrote: Dear Michael, I have the same problem below. Could you help us? Thanks a lot~ Best, James — Reply to this email directly, view it on GitHub <#297 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG7TFGEYL4GUMCHCFZ2LWDYFS5ZLAVCNFSM6AAAAAA55ZGKACVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMMZRGUYDO> . You are receiving this because you were mentioned.Message ID: ***@***.***>