I wanna to know how to calculate the results of PSNR/SSIM while evaluating?

[TonJ24]

Branch

master branch https://mmediting.readthedocs.io/en/latest/

📚 The doc issue

I wanna to know how to calculate the results of PSNR/SSIM while evaluating?
I suppose that there may be two branches:
1、Calculate the mean of all clips.
2、Calculate the mean of a specific number of clips, for an example, while evaluting on the Vimeo-90K-T, choosing N clips to mean the PSNR/SSIM.

Thanks sincerely for your reply

Suggest a potential alternative/fix

No response

[LeoXing1996]

Hey @TonJ24 , if you want to calculate the mean of all clips, you can directly use EvalHook in your config. For example:

evaluation = dict(interval=5000, save_image=False, gpu_collect=True)
test_cfg = dict(metrics=['PSNR', 'SSIM'], crop_border=0, convert_to='y')

If you want to evaluate the specific number of clips, perhaps you need to modify the forward_test of your model to select specific clips and calculate metrics for them.

[TonJ24]

Hey @TonJ24 , if you want to calculate the mean of all clips, you can directly use EvalHook in your config. For example:

evaluation = dict(interval=5000, save_image=False, gpu_collect=True)
test_cfg = dict(metrics=['PSNR', 'SSIM'], crop_border=0, convert_to='y')

If you want to evaluate the specific number of clips, perhaps you need to modify the forward_test of your model to select specific clips and calculate metrics for them.

Thank you for your elaborate reply. However, it is a challenge for me to find the default number of evalution clips.Can you tell me what's the number of evalution clips on Vimeo-90K-T as default?

[LeoXing1996]

@TonJ24 Do you mean how to split the test and evaluation set?
If you mean this, you may refer to sep_testlist.txt provided by the official dataset.

[TonJ24]

@TonJ24 Do you mean how to split the test and evaluation set? If you mean this, you may refer to sep_testlist.txt provided by the official dataset.

For an example, I mean that when I leverage your pretrained parameter, your existing model and your default setup to evaluate on Vimeo-90K-T,the PSNR/SSIM=40.0/0.8900. And I wanna to know whether this average result calculated from all (7824*7) pictures.

[ckkelvinchan]

@TonJ24 Do you mean how to split the test and evaluation set? If you mean this, you may refer to sep_testlist.txt provided by the official dataset.

For an example, I mean that when I leverage your pretrained parameter, your existing model and your default setup to evaluate on Vimeo-90K-T,the PSNR/SSIM=40.0/0.8900. And I wanna to know whether this average result calculated from all (7824*7) pictures.

No, they are calculated only on the center frame. That means, the 4th frame.

[TonJ24]

@TonJ24 Do you mean how to split the test and evaluation set? If you mean this, you may refer to sep_testlist.txt provided by the official dataset.

For an example, I mean that when I leverage your pretrained parameter, your existing model and your default setup to evaluate on Vimeo-90K-T,the PSNR/SSIM=40.0/0.8900. And I wanna to know whether this average result calculated from all (7824*7) pictures.

No, they are calculated only on the center frame. That means, the 4th frame.

Thanks very much for your elaborate reply. I assume that calculating the average PSNR/SSIM on (7824*7) is a normal idea. Do you know why this framework just obtain final result from center frame?

[ckkelvinchan]

Since earlier methods fall into the sliding-window framework. Specifically, they use 7 frames to produce the center frame. Therefore, they can only get the 4th frame given a 7-frame sequence. For others, padding is needed and therefore they do not consider them.

[TonJ24]

Since earlier methods fall into the sliding-window framework. Specifically, they use 7 frames to produce the center frame. Therefore, they can only get the 4th frame given a 7-frame sequence. For others, padding is needed and therefore they do not consider the

Since earlier methods fall into the sliding-window framework. Specifically, they use 7 frames to produce the center frame. Therefore, they can only get the 4th frame given a 7-frame sequence. For others, padding is needed and therefore they do not consider them.

Get! Thank you very much!