Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Sasha Rush3 HN recommenders · recommended in 3 months, 2024–2024
lecture · advanced · 2024–2024 on HN
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.
HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.
Up next
2 mo on HNDrawing Conclusions: Is renting really a waste of money?
The Globe and Mail
2 HN recommender · 2018–2019
16 mo on HNThe spelled-out intro to neural networks and backpropagation: building micrograd
Andrej Karpathy
17 HN recommender · 2022–2025
14 mo on HNBut what is a neural network? | Deep learning chapter 1
3Blue1Brown
16 HN recommender · 2017–2025
10 mo on HNAlphaGo - The Movie | Full award-winning documentary
Google DeepMind
10 HN recommender · 2020–2025
10 mo on HNTransformers, the tech behind LLMs | Deep Learning Chapter 5
3Blue1Brown
13 HN recommender · 2024–2026
10 mo on HNUSENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
USENIX
10 HN recommender · 2018–2026
11 mo on HNLecture 1 | Machine Learning (Stanford)
Stanford
11 HN recommender · 2008–2020
10 mo on HNMarI/O - Machine Learning for Video Games
SethBling
10 HN recommender · 2015–2026
8 mo on HNDeep Dive into LLMs like ChatGPT
Andrej Karpathy
10 HN recommender · 2025–2026
10 mo on HNComputer program that learns to play classic NES games
suckerpinch
9 HN recommender · 2015–2025
9 mo on HNRL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning
Google DeepMind
9 HN recommender · 2017–2026
8 mo on HNLet's build GPT: from scratch, in code, spelled out.
Andrej Karpathy
14 HN recommender · 2023–2025
3 comments from Hacker News
To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: ▶ this video
Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: ▶ this video Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).
Would recommend Sasha Rush's lecture ▶ this video , it provides the intuition for state space models via linear RNNs