How to visualize Hyena Matrix H(u) from the Paper

[ZHymLumine]

Firstly, I would like to extend my sincere gratitude for your amazing work and the code that you have shared. It's truly impressive.

I was reading through your paper and noticed the visualization of the Hyena Matrix H(u), which I found to be particularly interesting. However, when I went through the codebase, I realized that there isn't a direct implementation for materializing this matrix.

Could you kindly guide me on how I might go about visualizing the Hyena Matrix H(u) using my own dataset? Any pointers or additional information you could provide would be greatly appreciated.

[Zymrael]

Thank you for the kind words. I'll use a notation close to the paper to make it easier to connect the dots, and set the number of projections to 3. In the first version of Hyena, you will have a different matrix for each channel. To visualize H(u) for a target channel you'll need to materialize the following matrices:

• T_v: toeplitz matrix corresponding to the causal short 1d convolution applied to the target channel of the v projection
• T_x1: same as above, but for a different projection
• T_x2: same as above, but for a different projection
• D_v: the L x L matrix with v at the given channel on the diagonal. Note this is also L x L
• D_x1: same as above
• D_x2: same as above
• T: Toeplitz matrix corresponding to the long implicit convolution.

Once you have these matrices, you can multiply them in the same order you would apply them to v, e.g,:

• H(u) := T_x2 D_x2 T T_x1 D_x1 T_v v

Once you have the matrix you can also check whether applying H(u) gives you the same output via a direct matrix multiply. When you generate the Toeplitz matrices with the convolutional filters, be careful to take into account padding to keep all convolutions causal.