Need content on feature importance for tree-based methods

[davidrosenberg]

Software routinely gives various measures of feature importance and even marginal dependency on features. Explanation of what such things are and pointing out how they fail seems like a potentially very useful Lab topic.

[buj201]

Related subtopic: Tree-based models that require one-hot encoding of categoricals, vs. out-of-box support for categoricals.

1. Example dataset: Consider looking at https://archive.ics.uci.edu/ml/datasets/Adult (categoricals for occupation/education/etc). Ideally we'd find a dataset where the categoricals rank differently for the two runs described in 2.
2. Use catboost or lightgbm with and without one hot encoding preprocessor.
3. Get feature_importance_ from two versions.
4. Compare.

[davidrosenberg]

Somewhat related to feature importance is feature selection. Here's a paper on non-linear feature selection with gradient boosting. Haven't read it yet but could be interesting: http://alicezheng.org/papers/gbfs.pdf