The Simple Essence of Automatic Differentiation - Conal Elliott
Microsoft Research3 HN recommenders · recommended in 4 months, 2018–2023
talk · advanced · evergreen · 2018–2023 on HN
Conal Elliott presents his paper on automatic differentiation, deriving a simple, generalized AD algorithm from a natural specification. Varying the representation of derivatives yields forward and reverse mode algorithms that avoid the complexity and statefu…
Cited as the more accessible companion to the paper, and as the most interesting treatment of AD via compiling to categories.
Up next
3 mo on HNHow to achieve nanometer precision in space?
Breaking Taps
4 HN recommender · 2022–2026
3 mo on HNTWO Unpickable (?) Locks for Lock Picking Lawyer!
Stuff Made Here
3 HN recommender · 2021–2022
4 mo on HNThis is why we can't have nice things
Veritasium
4 HN recommender · 2021–2024
7 mo on HNThe Future of Mathematics?
Microsoft Research
9 HN recommender · 2019–2025
3 mo on HNFast Inverse Square Root — A Quake III Algorithm
Nemean
3 HN recommender · 2021–2022
3 mo on HN[1299] Unpickable Locks From Stuff Made Here
LockPickingLawyer
5 HN recommender · 2021–2022
5 mo on HNWhy can't you multiply vectors?
Freya Holmér
9 HN recommender · 2023–2025
5 mo on HNFourier Optics used for Optical Pattern Recognition
Huygens Optics
5 HN recommender · 2021–2025
16 mo on HNThe spelled-out intro to neural networks and backpropagation: building micrograd
Andrej Karpathy
17 HN recommender · 2022–2025
14 mo on HNBut what is a neural network? | Deep learning chapter 1
3Blue1Brown
16 HN recommender · 2017–2025
10 mo on HNAlphaGo - The Movie | Full award-winning documentary
Google DeepMind
10 HN recommender · 2020–2025
10 mo on HNTransformers, the tech behind LLMs | Deep Learning Chapter 5
3Blue1Brown
13 HN recommender · 2024–2026
6 comments from Hacker News
The most interesting thing I've seen on AD is "The simple essence of automatic differentiation" (2018) [1]. See past discussion [2], and talk [3]. I think the main idea is that by compiling to categories and pairing up a function with its derivative, the pair becomes trivially composable in forward mode, and the whole structure is easily converted to reverse mode afterwards. [1]: link [2]: link [3]: Talk at Microsoft Research: ▶ this video Other presentations listed here: link
I'm yet to give reverse mode a second try, but my attempts at forward mode (linked in parent post) were on my own without the paper's help. There's a talk that Conal gives that makes the paper a bit more accessible - ▶ this video - but only a bit more. I know it'll take effort to wrap my head around the implementation details ... but the paper certainly helped clarify things I was stumbling around about.
He also gave nice presentation on AD: ▶ this video (surprised that you didn't already mention it, considering the venue ;-))
Another paper related to AD was posted last month: The Simple Essence of Automatic Differentiation [0]. Video presentation [1]. I found it pretty awesome, just in case you hadn't seen it. [0]: https//news.ycombinator.com/item?id=18306860 [1]: ▶ this video
AD is just a Cartesian lens of Jacobians and total derivatives in the category of smooth functions, what's the problem? ▶ this video
This paper won a best paper award at ICFP 2018. Conal Elliott recently gave a longer talk about it at Microsoft Research: ▶ this video