In the module: MambaTransformer/mamba_transformer, you execute the following in class MambaTransformerblock:
# Layernorm
self.norm = nn.LayerNorm(dim)
def forward(self, x: Tensor) -> Tensor:
for mamba, attn, ffn in zip(
self.mamba_blocks,
self.transformer_blocks,
self.ffn_blocks,
):
x = self.norm(x)
x = mamba(x) + x
x = self.norm(x)
x = attn(x) + x
x = self.norm(x)
x = ffn(x) + x
return xSince the layerNorm has trainable parameter, you appear to be calling three layer norms in the forward function with tied parameters. Is that what you really want?
kyegomez / MambaTransformer
Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling
170
Contributors get 40% of received funds after fees
How does funding with Polar work?
1
Pay now to fund the work behind this issue.
2
Get updates on progress being made.
3
Maintainer is rewarded once the issue is completed.
FAQ
Backer
You're funding impactful open source efforts
Contributor
You want to contribute to this effort
Maintainer
You want to get funding like this too