100% agree, and there are a number of efforts in the space. mlpack (https://www.github.com/mlpack/mlpack/), Shogun (https://www.shogun-toolbox.org/), and Shark (https://www.shark-ml.org/) are three that have been around for over a decade now. They're a little niche because C++ is not that popular for data science, but they are generally pretty fast (especially mlpack, which focuses on speed).