Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Supporting memory efficient dropout in flash attention #23

Merged
merged 10 commits into from
Jun 5, 2024
Merged

Supporting memory efficient dropout in flash attention #23

merged 10 commits into from
Jun 5, 2024

Commits on May 24, 2024

  1. adding dropout to regular flash attention.

    @tongxin
    tongxin committed May 24, 2024
    Configuration menu
    Copy the full SHA
    53747d4 View commit details
    Browse the repository at this point in the history

  2. fixed syntax errors.

    @tongxin
    tongxin committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c23bf88 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. working in progress, forward dropout passes tests.

    @tongxin
    tongxin committed May 27, 2024
    Configuration menu
    Copy the full SHA
    3fa6183 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Flash attn fwd with dropout done.

    @tongxin
    tongxin committed May 28, 2024
    Configuration menu
    Copy the full SHA
    b60017d View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. [dropout] bug fixes, passing forward and backward tests.

    @tongxin
    tongxin committed May 29, 2024
    Configuration menu
    Copy the full SHA
    f7798fa View commit details
    Browse the repository at this point in the history

  2. fixed extra return errors in tests.

    @tongxin
    tongxin committed May 29, 2024
    Configuration menu
    Copy the full SHA
    91a22f3 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. move call to philox_cuda_seed_offset & kernel launch into device guar… 

    …d to ensure execution on device other than cuda:0
    @iclementine
    iclementine committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    cda4b19 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. remove unused parameters

    @iclementine
    iclementine committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    8b4fdbb View commit details
    Browse the repository at this point in the history

  2. Minor refactors.

    @tongxin
    tongxin committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    e49958e View commit details
    Browse the repository at this point in the history

  3. add dropout into feature list in README

    @iclementine
    iclementine committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    e5c408a View commit details
    Browse the repository at this point in the history