Do two promising mechanisms help each other, or is one doing all the work?
Separate the moving parts
This project compares standard and IBNN feed-forward layers with standard and forgetting attention. A two-by-two experiment makes it possible to ask which change helps and whether the combination behaves differently.
The harness runs locally with PyTorch and includes character-level language modelling on Tiny Shakespeare, alongside further experiments. Correctness checks and repeated seeds matter because a small apparent improvement can easily be noise.
A useful negative result
In the reported three-seed Tiny Shakespeare factorial, forgetting attention improves held-out performance by about 0.19 bits per character. Adding IBNN does not provide an additional gain; its mean result is slightly worse in both attention settings.
The forget mechanism is related to the published Forgetting Transformer, rather than a claim of a new attention invention. The study’s value is the controlled comparison and the willingness to retain an unhelpful result.
Keep the conclusion at the right scale
These results describe the tested architectures, datasets and training budgets. They are not evidence that every form of lateral coupling fails in every model. The repository extends the investigation with additional variants and robustness checks, making the next question testable rather than assuming an answer.
