Could you possibly add infini-attention and not this thing whatever this was supposed to be
2
👍
0Describe the solution you'd like
Ideally you should add true Infini-attention implementation there but I suppose you won't
Additional context
https://arxiv.org/pdf/2404.07143 - if you take a look at infini-attention paper, it's something totally different, I get it that the paper mentions both full attention and linear attention, but that doesn't mean you're just supposed to concatenate those, how tf did you even get an idea of doing that and calling it an implementation of infini-attention
kyegomez / Infini-attention
Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTORCH
48
Contributors get 40% of received funds after fees
How does funding with Polar work?
1
Pay now to fund the work behind this issue.
2
Get updates on progress being made.
3
Maintainer is rewarded once the issue is completed.
FAQ
Backer
You're funding impactful open source efforts
Contributor
You want to contribute to this effort
Maintainer
You want to get funding like this too