How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework

I thought this presentation by Armin Ronacher (the creator of Flask and Jinja) was very enlightening: How Python was Shaped by leaky Internals(2016): ▶ this video See also this HN post (Attempts to make Python fast): link

I think "a + b" is a good example. I really recommend the talk I linked ( ▶ this video ), it gets deep into this stuff. I don't think anyone benefits from "a + b" having so many special cases.

Python is difficult to optimize due to complex language semantics. This excellent talk describes how the semantics of the "add" operation are much more complex than generally described. a+b is not just a.__add__(b), you have to read 200+ lines of CPython interpreter code to fully resolve its semantics: ▶ this video Not saying it's impossible, but it's more of an uphill battle than optimizing other languages that have more regular semantics. Python is also constrained by its C API, which is a very low-level and exposes a lot of implementation details to C extensions. Changing the API could improve performance but it would also make Python incompatible with a large ecosystem of C extensions. I sometimes wish for Python to define a new C API that exposes far less, and to begin a long-term migration towards the new API so that eventually the interpreter would be less constrained. It would be a large migration, but unlike Python 2 -> 3 it wouldn't affect everyday users, only extension authors. I don't know if it will ever happen, but a person can dream.

As the OP states: > mutable interpreter frames, global interpreter locks, shared global state, type slots On top of this, Python is extremely dynamic and nothing can be assured without running through the code. So this leads to needing JITs to improve performance which then give a slow start up time and increased complexity. Even with JIT, Python is just not fast thanks to the above issues and it's overall dynamism. It can be optimised and for sure there's some impressive attempts at doing so. However I don't think pure Python will ever be considered "fast" as these things necessarily get in the way. I highly recommend the two videos posted here that go into more detail as to why there are limits to how far optimisation can go: ▶ this video link

I highly recommend this talk by Armin Ronacher as an accompaniment (or prologue, or epilogue) to the article: "How Python was shaped by leaky internals", ▶ this video

As others have mentioned, Python does have JIT compilers. The problem is that havign a JIT doesn't solve the problem. PyPy is often a factor of 10 behind julia performance and projects like Numba, PyTorch (the PyTorch people had to build their own Python JIT compiler yikes!), etc. will always have a more restricted scope than a project like Julia because Python's very semantics make many optimizations impossible. Here's a great talk from the author of the famous Flask library in Python: ▶ this video where he discusses the fundamental problems with Python's semantics. If you fix these problems, you will end up changing Python so fundamentaly that you'll really have a new language. Generic CPython 3._ code will certainly not be compatible with it.

Python needs a new runtime. This talk shows how bad of shape it's really in. ▶ this video Basically, the language doesn't have a "spec" per-se. The language is whatever the defacto CPython implementation happens to do within it's giant eval loop. Another great talk about CPython internals: link

VWWHFSfQ, you may already know this, but: I recommend this talk by Armin Ronacher (Flask creator) on how Python's implementation internals contribute to the difficulties of making Python faster. ▶ this video One case study Ronacher gets into is the torturous path taken through the Python interpreter (runtime?) when you evaluate `__add__`. Fascinating stuff.

You can't have Python run that much faster without loosing lots of features. I guess if they open sourced it, people wouldn't be that impressed with what they implemented. This video [0] from the creator of Flask explains it quite well. Here [1] is Guido talking about Mojo being in early days and sharing some technical details. [0] ▶ this video [1] link

I know very little about Python, but I would suggest these talks about a few ways in which Python is unique. They won't teach you Python, but they might help in making sense of the confusing bits you might encounter: How Python was Shaped by leaky Internals, Armin Ronacher: ▶ this video What Does It Take To Be An Expert At Python?: link and maybe also When Python Practices Go Wrong: link

Python's classes can change dynamically though during runtime. So each iteration of a loop would need to check if the shape has changed. I'm no Python expert though, so please correct me if I'm mistaken. But it's more than that. There's a really nice talk here from the creator of the Flask framework about a bunch of different semantic choices in the language design, leak out into the ecosystem and make many optimizations impossible: ▶ this video

Armin Ronacher, author Flask, actually has a good talk about this. The gist of it the way Python's internals leak into the language makes it very difficult to build a performant JIT that wouldn't break a large amount of userspace code. Python lets you do _far_ more shenanigans that Javascript does; and a lot of large libraries depend on some of that behavior. Breaking it would probably cause a new 2 -> 3 situation. ▶ this video