For example another commentor mentions low-latency concerns in finance, and that's something I have zero experience with.
HPC has often also meant writing a lot of C++ to do e.g. MD or something.
These days, I consider myself HPC-adjacent -- I write scientific ML software, often for use on pretty beefy hardware (TPU pods etc.) So at least for that, here's an off-the-cuff list of a few items that come to mind:
- Know JAX. Really, really well: its internals, how its transforms work. It's definitely a bit bumpy in places, but it's still one of the best things we have for easily scaling programs, e.g. through `jax.pmap`, being able to test on CPU and then run on TPU, etc.
- Triton! New(-ish) kid on the block for GPU programming.
- How CPUs work: L1/L2/L3 caches, branch prediction, etc. Parallelism via OpenMP.
- Know autodiff well. E.g. have a read of the Dex paper, and the concerns with doing autodiff through index operations. Modern scientific computing is moving towards a ubiquitously autodifferentiable future.
- ... plus loads more, haha. Probably I'm still missing ten different things that another reader considers crucial.
Depends! HPC is a massive umbrella term. Which part of it are you interested in?
For finance, firewall-to-firewall time was an important concept. (The time it takes a signal to get into your datacenter, be processed, then emit a signal back out.)
But e.g. massive data processing is an entirely different beast. Latency isn't too important, whereas parallelization is crucial.
So it's kind of hard to make a list without being pointed in a vague direction.
Can anyone suggest me something similar for HPC domain?