Use Brier score for classification

[ja-thomas]

Might be better as a default compared to missclassification rate especially for tuning to make everything a bit smoother

[PhilippPro]

In my benchmark, it was not always better regarding the MMCE than for example tuning AUC. It had on average the lowest rank but at least for RF it sometimes provided worse results than tuning MMCE directly or MMCE. Important parameter in my opinion.

[ja-thomas]

Thanks for the info. Looks interesting.
So, in essence, it still makes sense for the user to choose the measure that should be optimized.
The question is still what's the best default measure :)
If I look over the average ranks over different measures, the brier score seems to be the best overall, correct?

[PhilippPro]

Yes, for the ranks this is true. But difference to for example logarithmic loss is not big. Probably it would be interesting to benchmark autoxgboost with different target measures. ;)

[ja-thomas]

sounds like a good idea.

@Coorsaa can you add that to the benchmark?