"Performance Matters" by Emery Berger

Nobody should do benchmarking like this anymore. Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit. Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2]. Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal. Coz is pretty trivial to set up with Zig too. 1. ▶ this video 2. link

I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications. [1] ▶ this video

Aside: this is the most amazing optimisation information I have seen: ▶ this video Unfortunate that it is a video... Summary: Modern CPUs have so many hidden causes of performance variation, that you require special tools to actually measure small performance gains. He finds an amazing but unobvious speed up in SQLite using the causal analysis tool they developed. And a surprising difference between -O2 and -O3 implying overfitting.

I haven't had a chance to mess around with it much, but the coz-profiler [1] uses basically the same idea, but the other way around (i.e., slow down everything else to get a "virtual speedup") which helps quantify what the effect of optimizing some code would be. I learnt about this tool from Emery Berger's talk [2] on this (at strangeloop), which I highly recommend. Lots of really nice insight, even outside of this tool. [1] link [2] ▶ this video

"Performance matters" by Emery Berger on StrangeLoop'19 is a great talk - ▶ this video Also, rest of his talks are also fascinating: - link - link

Woah, I can't believe it, I've found the discussion that linked the thing I was looking for. [1] It's called Coz, the causal profiler [2]. I've submitted this talk again, because I think it's super interesting. [1] link [2] ▶ this video

There's a great talk that explains how/why this works for those looking for more info. Also, Go UMass! ▶ this video

I think the profiling section could benefit from a section describing the sort of profiling noise described in this video: ▶ this video The noise source described is that changes to the code typically alter alignments of code and/or data and that is often the cause of the performance difference measured, not the change to the algorithm. The reason this is noise is because unrelated changes to the code might make it change again. IIRC, the speaker suggests improving compiler/profiler tools to randomise these alignments and benchmark many times.

Relevant talk by Emery Berger: Performance matters ▶ this video I'd recommend everyone interested in the subject to watch it.

Try getting them to watch this talk by Emery Berger: ▶ this video And who knows, maybe they'd even be interested in using coz afterwards: link

This[1] talk from Emery Berger at last year's Strange Loop does a good job discussing the impact of alignment and other 'small' changes on performance, and how to estimate the actual impact of changes on performance [1]: ▶ this video

It's mentioned elsewhere in this HN discussion, but I think I'd encourage watching this talk: ▶ this video . It's particularly relevant to what you're seeing.