Rust at speed — building a fast concurrent database
Jon Gjengset2 HN recommenders · recommended in 2 months, 2019–2022
lecture · advanced · evergreen · 2019–2022 on HN
A guest lecture Jon Gjengset gave at Two Sigma on using Rust for large, high-performance systems. It covers why Rust suits such projects, the Noria research database, a concurrent data structure used in it, and practical experience.
HN points to it as a detailed, captivating talk by the Noria author, including a good explanation of deduplicating data between maps using unsafe.
Up next
3 mo on HNCrust of Rust: Lifetime Annotations
Jon Gjengset
3 HN recommender · 2020–2023
11 mo on HNCppCon 2015: Andrei Alexandrescu “std::allocator...”
CppCon
7 HN recommender · 2017–2025
12 mo on HNLexical Scanning in Go - Rob Pike
GoogleFOSSSydney
13 HN recommender · 2012–2025
10 mo on HNCppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step”
CppCon
9 HN recommender · 2018–2026
9 mo on HNRustConf 2018 - Closing Keynote - Using Rust For Game Development by Catherine West
Rust Programming Language
9 HN recommender · 2018–2023
10 mo on HNCppCon 2016: Jason Turner “Rich Code for Tiny Computers: A Simple Commodore 64 Game in C++17”
CppCon
13 HN recommender · 2016–2024
9 mo on HNIdeas about a new programming language for games.
Jonathan Blow
7 HN recommender · 2014–2025
9 mo on HNCppCon 2014: Mike Acton "Data-Oriented Design and C++"
CppCon
9 HN recommender · 2015–2026
9 mo on HNHow I program C
Eskil Steenberg
8 HN recommender · 2017–2025
9 mo on HNGoogle I/O 2012 - Go Concurrency Patterns
Google for Developers
9 HN recommender · 2012–2025
7 mo on HNCppCon 2015: Kate Gregory “Stop Teaching C"
CppCon
7 HN recommender · 2016–2025
7 mo on HNZig: A programming language designed for robustness, optimality, and clarity – Andrew Kelley
Recurse Center
6 HN recommender · 2018–2024
4 comments from Hacker News
The author of Noria, Jon Gjengset, gave a great talk about the system: ▶ this video The speeds achieved are impressive but you do not get consistency.
Anecdotal - One of favorite YouTube live-streaming personalities, Jon Gjengset[1] is a Ph.D student at MIT CSAIL. His talks are very detailed[4] and captivating[3] [1] link Talks [2] ▶ this video [3] link Rust Details(my favorites) [4] link
:wave: Author of the paper this work is based on here. I'm so excited to see dynamic, partially-stateful data-flow for incremental materialized view maintenance becoming more wide-spread! I continue to think it's a _great_ idea, and the speed-ups (and complexity reduction) it can yield are pretty immense, so seeing more folks building on the idea makes me very happy. The PlanetScale blog post references my original "Noria" OSDI paper ( link ), but I'd actually recommend my PhD thesis instead ( link ), as it goes much deeper about some of the technical challenges and solutions involved. It also has a chapter (Appendix A) that covers how it all works by analogy, which the less-technical among the audience may appreciate :) A recording of my thesis defense on this, which may be more digestible than the thesis itself, is also online at link , as well as a shorter talk from a few years earlier at ▶ this video . And the Noria research prototype (written in Rust) is on GitHub: link . As others have already mentioned in the comments, I co-founded ReadySet ( link ) shortly after graduating specifically to build off of Noria, and they're doing amazing work to provide these kinds of speed-ups for general-purpose relational databases. If you're using one of those, it's worth giving ReadySet a look to get these kinds of speedups there! It's also source-available @ link if you're curious.
He explains this at roughly the 40 minute mark. [1] "There's 7 more lines of `unsafe` that avoids keeping both maps. You still keep two maps, but you de-duplicate the data between those two maps." ▶ this video