I could not find a rigorous definition of the feature norms in the paper. Which layer or block do the tokens originate from?
Regarding the attention maps, I assume that the norms are based on the linearly transformed tokens used to calculate the attention matrices. According to LayerNorm, all tokens should have a norm of
Am I misunderstanding something?
kyegomez / Vit-RGTS
Open source implementation of "Vision Transformers Need Registers"
143
Contributors get 40% of received funds after fees
How does funding with Polar work?
1
Pay now to fund the work behind this issue.
2
Get updates on progress being made.
3
Maintainer is rewarded once the issue is completed.
FAQ
Backer
You're funding impactful open source efforts
Contributor
You want to contribute to this effort
Maintainer
You want to get funding like this too