Shared memory parallelism in Julia | Kiran Pamnany | Cambridge Julia Meetup (May 2018)

By far the most interesting part of this release is the new multi-threading features, I recommend reading link for an overview, and watching ▶ this video for a talk about some unique stuff that's being worked on with Julia's multi-threading (though the depth-first scheduling is not in 1.3 afaik).

Consider a machine with N cores running a program which spawns N parallel tasks, each of which in turn spawns N parallel tasks. Which subtasks do the cores work on initially? If it's the first subtask of each top-level task, that's breadth-first scheduling. If it's all N subtasks of the first top-level task, that's depth-first scheduling. The latter is better for high-performance computing because it tends to be the lower level libraries that are really carefully tuned to make optimal use of the machine (think BLAS/FTT/parallel sum/prefix sum). Previously it was unknown how to implement this kind of scheduling efficiently. See this talk by Kiran Pamnany for more details: ▶ this video

This work has been a monumental collaborative effort and I think the eventual outcome will be pretty spectacular. I gave a talk at an MIT Julia meetup a few months ago that the author of this PR also spoke at; I found his talk to be an insightful overview of this work: ▶ this video

checkout ▶ this video which explains the new features of the task scheduler in a simple fashion. Basically Julia is employing a new algorithm to schedule tasks, where traditional methods would get stuck/blocked. Essentially, if you run a job with parallel tasks and one of the threads finishes, it's extremely difficult to schedulate the free thread, since the data may still be used by the other threads that are busy; thus you need to have clever way to determine how to allocate computational resources. Julia is implementing results of a new paper on how to optimally do this.

Does the Rust Rayon library have the issue the speaker was describing, where parallel inner loops mostly get executed serially once an outer loop is parallelized? (I assume it does, from its Cilk heritage?) Are there workarounds for when that is a problem? Edit: Dumb mistake on my part. I was looking at the video linked in the top comment ( ▶ this video ) and then I came back to this thread and thought it was the topic.