Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Self attention for pooling linear classifier #28

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Self attention for pooling linear classifier #28

wants to merge 3 commits into from

Commits on Jan 6, 2019

  1. Self attention for pooling linear classifier 

    This PR will introduce a `BiAttentionPoolingClassifier` as in [Attention is all you need](https://arxiv.org/abs/1706.03762) following the discussion with @sebastianruder in Teams.

I ran out of memory on my 1060 while testing the attention module, but was able to at least verify that it is functionally correct. Some changes might be required to ensure that the tensor passed to `self.layers` is of the right shape (but I'm not quite sure as of now).

I'll shift all the stuff to Collab for testing and see if it's any help.
    @aayux
    aayux authored Jan 6, 2019
    Configuration menu
    Copy the full SHA
    d0c1547 View commit details
    Browse the repository at this point in the history

  2. Make callable through bicls_head

    @aayux
    aayux authored Jan 6, 2019
    Configuration menu
    Copy the full SHA
    8fe2891 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2019

  1. Fixed self attention, local tests passed

    @aayux
    aayux committed Feb 14, 2019
    Configuration menu
    Copy the full SHA
    03c4a2c View commit details
    Browse the repository at this point in the history