As another plug for Kaggle - it lets you know what is state of the art. For instance, from my Kaggle experience I know that gradient boosted decision tress (specifically xgboost) are virtually always superior to random forests. They are also basically just as easy to use, in contrast to neural nets which are not as user friendly. The machine learning step was probably the easiest step in this process, but it is also easy to leave simple gains on the table. Gradient boosted decision trees don't get nearly enough hype.
I would even argue it's easier to use that random forests. The standalone R package is super slick, with built in cross validation. And if you want to tune the hyperparameters you can just do it in caret. Really accessible stuff.
Yeah on Kaggle I found myself getting routinely outperformed by people with scripts that simply ran a gradient boosted decision tree and nothing else. And yet the topic was never mentioned in my modeling and stats courses!
That's because it's practically new. Tianqi Chen authored the R package for it (original release Aug 2015) and actually posts about it on the Kaggle forums quite frequently.
