Calculating DIC for HSSM models

[HugoSnyders]

Hi all,

I'm currently doing some model comparison in HSSM and I wanted to double check whether there is any way to calculate DIC for HSSM models. I know Arviz only allows for calculating LOO and WAIC (and dissuades people from using DIC), but I wanted to use multiple metrics for my model comparison (including DIC) to see if they all lead to the same ranking for my models. I was wondering if you knew of any workaround for this? I had a go at calculating DIC manually but in the end that just wasn't really feasible, especially for the more complex models.

Thanks for any help you can provide!

Best,
Hugo

[AlexanderFengler]

Hey @HugoSnyders ,

I think we have an open issue on this, we will try to get that in as a convenience function. In principle, one should be able to calculate DIC from the idata object returned after sampling without the need for further info.

Not sure what you tried so far, but just to mention it:

To compute the DIC from your traces (idata.posterior), you need to pass the idata_kwargs = {'log_likelihood': True} argument to your call to myhssmmodel.sample(), to include log likelhoods in your trace.

If you did that and ran into trouble, could you share a code-snippet, so we can help you?

Best,
Alex

[kiante-fernandez]

Not sure what @HugoSnyders tried but you could give this a go. Use comes with all the standard dissuading people from using DIC :).

Make sure you set idata_kwargs = {'log_likelihood': True}.

def dic(inference_data):
    """
    Calculate the Deviance Information Criterion (DIC) for a given model.

    Parameters:
    inference_data (arviz.InferenceData): An ArviZ InferenceData object containing the posterior samples

    Returns:
    float: The computed DIC value.
    """

    # Extract log likelihood from the inference data
    log_likelihood = inference_data.log_likelihood

    # Calculate the point-wise deviance
    D_bar = -2 * np.mean(log_likelihood)

    # Calculate the effective number of parameters
    p_D = 2 * (D_bar + 2 * np.mean(log_likelihood) - np.mean(log_likelihood))

    # Calculate DIC
    dic = D_bar + p_D

    return dic

[digicosmos86]

Hey @kiante-fernandez! Thanks for writing up this function! Can we steal it and include it in HSSM as a utility function?

[kiante-fernandez]

@digicosmos86 Yes, of course, I'm happy to PR too if it is helpful. Would just need little direction about where you want it. I assume it would be similar in style to the HSSM.summary function and placed however in hssm.utils? Feel free to add it if it's more of a hassle though.

[HugoSnyders]

Hi both,

Seems like not including the idata_kwargs = {'log_likelihood': True} argument was indeed the culprit. Once included, @kiante-fernandez's function worked like a charm.

Thanks for your help!

Best,
Hugo