Log in
Kye Gomez

Kye Gomez/AttentionIsOFFByOne

Implementation of "Attention Is Off By One" by Evan Miller

Issues looking for funding

Is there a test showing effectiveness of softmax1 removing outliers?

#4 opened in kyegomez/AttentionIsOFFByOne

Fund
Discuss: softmax_one and zero_vector in QuietAttention conflicts?

#1 opened in kyegomez/AttentionIsOFFByOne

1
Fund
how to solve the issue of overflow

#2 opened in kyegomez/AttentionIsOFFByOne

Fund
If you wanna make it fast, just use nn.softmax() and concatenate a zero.

#3 opened in kyegomez/AttentionIsOFFByOne

1
Fund
IMPORTANT：The definition of the softmax one is wrong

#5 opened in kyegomez/AttentionIsOFFByOne

4
👍3
Fund
Is there any evidence that softmax one can takes advantages over normal softmax?

#6 opened in kyegomez/AttentionIsOFFByOne

👍2
Fund
Kye Gomezkyegomez/AttentionIsOFFByOne
Creatorkyegomez
Stars181
LicenseMIT License
RepositoryGitHub