Reproducibility of ICLR Figure 2

[zhan8855]

Thank you so much for your awesome work!

Currently, I am facing difficulty to reproduce the Figure 2 of the ICLR paper. I followed the guidelines at https://github.com/HazyResearch/zoology/tree/main/zoology/experiments/iclr24_zoology_figure2. However, my results seem to differ from those reported in the paper:


Would it be possible for you to offer some guidance or advice? I really appreciate your help and look forward to your insights. Thank you again in advance!

[simran-arora]

These look highly similar to me! What's the issue

[zhan8855]

Well, it seems that my reproduced Hyena and H3 are better than the results reported in the paper (e.g. H3 reaches ~0.5 with dim=64, while the corresponding results reported in the paper is only ~ 0.25). At the same time, my reproduced BaseConv is slightly worse (e.g. Reproduced BaseConv with dim=256 is less than 0.75 in reproduced results, while in paper it is more than 0.75).