Why this infrastructure is so tightly coupled with CUDA? CUDA is very specific and closed APIs for NVidia hardware only. Programming languages should focus on more general primitives that might work on NVidia or TPUs or something else. PyTorch also has CUDA all over in its APIs and its frustrating to see such tight binding with closed one company API. Also take a look at OpenCL.
Our view is that to get performance out of a system (here CUDA), it's better not to start abstracting it right away. So we have CUDAnative.jl and CUDAdrv.jl for fairly low-level CUDA programming, albeit in a high-level language. However, with CuArrays.jl we implement the Julia array interface for CUDA GPUs. That means you can write array code for one platform (CPU using Base.Array) and start using hardware accelerators by just switching the array type (CUDA GPU using CuArray). Of course, real-life applications might still need to use CUDA specific functionality for one reason or another, but at least you can get most of the way without platform-specific programming.
Are you certain that the story is as simple as "CUDA performs better"? It's common folklore, but I have seen little evidence. The only situations I know of when CUDA performs better is when CUDA-specific features are used (if they are relevant for whatever problem is at hand). Also, CUDA libraries (like cuBLAS or cuFFT) tend to be more efficient than their OpenCL equivalent, which is likely because much more work has gone into them. I have also noted that the CUDA compiler is willing to use less accurate (but faster) floating-point instructions by default (for things like e.g. inverse square root), where you need to pass options to the OpenCL compiler for it to do the same. This will matter for some programs.
In fact, I have run tens of thousands of lines of essentially equivalent CUDA and OpenCL code (automatically generated) on the same hardware, and performance was in all cases very similar[0]. If anything, CUDA was actually slower than average (but in the cases I investigated, this was down to arbitrary differences like the CUDA compiler not unrolling some loops as aggressively and such).
No; the code we would need to generate would be rather different. Would you expect a significant difference? When we did research on nvrtc before implementing this, we couldn't find any concrete information that nvrtc should generate slower code.
Sure, I don't mind CUDA backend as first class citizen. I'm talking about having my code sprinkled with word "cuda" all over. Why can't I write my code that is bit more abstract and potentially compilable to different backends? That is, think about the primitives instead of tightly getting married to cuda forever. AMD performance might not be good today but how about 10 years later? How about using TPUs instead? or FPGAs (if someone creates backend for it)?
Well, one problem is that you're reading an NVIDIA marketing post on an NVIDIA blog talking in particular about the lowest levels of the stack targeting NVIDIA hardware. Higher level abstractions can and do just work across different hardware backends (not as well as we'd like, but we have some thoughts on how to improve that).
> Why this infrastructure is so tightly coupled with CUDA?
It's not. It uses LLVM, which can easily target AMD GPUs. (Whether the Julia folks have invested in making this work, I dunno, but it's not Extremely Hard.)
Understandably nvidia gives you the wrong impression.
We are indeed interested in targeting AMD GPUs. There is a prototype backend available at https://github.com/JuliaGPU/AMDGPUnative.jl and we are closely following the status of SPIR-V and Intel GPUs in LLVM.
The focus on CUDA comes from the fact that most HPC systems for scientific computing are using Nvidia GPUs. That is finally slowly changing.
Because Khronos up to a little while lived on a bubble that we have to use C, write our own compiler and linking logic to use GPGPUs and collect debugging toolchains from each OEM.
Only when they started getting a beating of PTX bytecode and multi-language deployment on CUDA did they woke up and came up with SPIR (later SPIR-V) and SYCL, which still isn't widely deployed.
OpenCL, hip, Vulkan, what tomorrow? Or alternatively, there were cl* libraries, roc* libraries and hip* libraries for AMD? Which ones are supported?
CUDA doesn't require rewriting code with ${OSS} framework of the year, every year. They need to earn that lock-in with future compatibility guarantees which none of OSS projects has.
Whatever it is, as long as it's not tied to one GPU only, it could be promising. Something that's tied to Nvidia or anyone else exclusively is not good, and surely isn't democratizing anything.
> CUDA doesn't require rewriting code with ${OSS} framework of the year, every year.
How so? Change the GPU from Nvidia, and you are forced to rewrite code. That's the whole point of lock-in, it's a tax on developers. CUDA doens't guarantee you anything, if you don't stick with their GPUs.
Vulkan on the the other hand has conformance requirements.
Yes, but is cuda going to keep its edge 10 years down the line? Do I want to hardcode my algorithms so tightly with today's cuda APIs? Can there be better more generic primitives that are agnostic of propitiatory cuda APIs but would support it as backend without too much perf hit?
If you want performance then yeah. If you are after hypothetical performance in future which may not even materialise, then the choice is yours. Everyone knows where the sensible ground is. Which, unfortunately, is CUDA only
AMD has a search and replace library that's API compatible with many cuda functions now. It hasn't caught on yet, but if they release decent hardware soon, it might.
