Lexical Scanning in Go - Rob Pike

Related, I love Rob Pike's talk about lexical Scanning in Go (2011). Educational and elegant approach. ▶ this video

Lexical Scanning in Go by Rob Pike ▶ this video I love everything about this talk. It walks you through building a lexer from scratch in a simple and elegant way, through a very interesting use of coroutines. I appreciate the bits of humor in the talk as well.

Rob Pike did a really good talk entitled 'Lexical Scanning In Go'[0], which covers how the lexing for text/template works. It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow. [0] ▶ this video

Personally, I have a penchant for writing my own pull parsers. Its a mind-expanding exercise. The neat thing about Go is that parsers can return functions that consume the next token. Rob Pike has an excellent video about this: ▶ this video

If you enjoyed the state machine + parsing part and also I recommend the older "Lexical Scanning in Go" talk that Rob Pike gave: ▶ this video It's in Go, but you can apply most of it easily in other languages. I'm so happy that modern languages like Rust, Zig, and Go support unicode/runes/graphemes natively. So many problem just disappear compared to Java, .net, C++, or scripting languages.

Rob Pike’s lexer was very nicely done and struck a chord with me. It’s not nearly as magical as some of the other things here. It’s just great engineering: ▶ this video

I would like to mention that most, if not all, of the Go compiler and related utilities is written in Go itself, with high quality, open source code that can be found in the "go" package of the standard library: link I would recommend using that as a template for one's own compiler, because it's written in idiomatic Go (originally by the language creators) and is cleanly split across abstraction boundaries (packages "scanner", "parser", "ast", and so on.) There's also a smaller, simpler parser in the "text/template/parse" package, that is used to parse the templating languages in "text/template" and "html/template". As an introduction to it, I recommend the talk "Lexical Scanning in Go" by Rob Pike: ▶ this video

Wow, perfect timing! I'm currently writing my first lexer/parser in D as well ( link ) for a Scheme-like language used in Halo 3. I found Rob Pike's talk on lexical scanning to be pretty useful ( ▶ this video ) and I've modeled mine pretty heavily after the text/template lexer: link .

For those who don't know who Rob Pike is, he's pretty awesome, helped create UTF-8 among other things, and now works on Go. link He has given some really great Go talks which are available on youtube, about an hour each: Origins of Go Concurrency Style - link Lexical Scanning in Go - ▶ this video Go Concurrency Patterns (One of my favorites) - link

I'm not sure I like the idea of parsing templates with regular expressions. There is a better way, as demonstrated in Robe Pike's video[1] detailing how the Go template language is implemented. Edit: which isn't meant to detract from the fact that this is a great article. [1] ▶ this video

> Building a lexer out of a recursive function type actually sounds like a really cool thing to try Rob Pike did just that for Go's text/template package ( link ). See the source lexer source code at link and note that stateFn is defined recursively. There's a video by Pike describing how he built the parser at ▶ this video .

> What most people don't realize is that for concurrency to be useful, there must be parallelism. No, as far as I understand from this and other talks, Rob Pike argues that concurrency is useful even without parallelism: it's a better way to structure software. A nice example for this is shown in his talk about lexical scanning in Go: ▶ this video