About mistakes in the implementation of SMA

[renll]

Thanks for the great work. I think the implementation of SMA has some mistakes.

1. The original SMA takes the logarithm during initialization and the exponential during forward path to ensure that the temperature scaler is larger than zero:
   https://github.com/renll/SeqBoat/blob/5a34aed3c573858630ec64b0f2c4a22948962947/fairseq/modules/seqboat_utils.py#L166

But it seems that this trick is lacked in the zoology implementation:

zoology/zoology/mixers/selective.py

Line 209 in dd43c72

self.temperature = nn.Parameter(torch.ones(1) * math.sqrt(d_model) * alpha)

2. The original SMA selects the inputs tokens for the attention module:
   https://github.com/renll/SeqBoat/blob/5a34aed3c573858630ec64b0f2c4a22948962947/fairseq/modules/seqboat_unit.py#L603
   but the Zoology one only selects the output:
   

   zoology/zoology/mixers/selective.py

   Line 245 in de4e258

   y = attn_output * selection + x