Where to find out more about packages

1. Vignettes
2. Journal Article (R Journal is great)
3. Rstudio Community
4. StackOverflow
5. Twitter

rpart.plot

tree_spec <- decision_tree(
  cost_complexity = 0.1,
  tree_depth = 10,
  mode = "regression") |>
  set_engine("rpart")

wf <- workflow() |>
  add_recipe(
    recipe(Salary ~ Hits + Years + PutOuts + RBI + Walks + Runs,
                   data = baseball)
  ) |>
  add_model(tree_spec)

model <- fit(wf, baseball)

rpart.plot(model$fit$fit$fit,
           roundint = FALSE)

Fitting classification trees

• We use recursive binary splitting to grow the tree

• Instead of RSS, we can use:

• Gini index: \(G = \sum_{k=1}^K \hat{p}_{mk}(1-\hat{p}_{mk})\)

• This is a measure of total variance across the \(K\) classes. If all of the \(\hat{p}_{mk}\) values are close to zero or one, this will be small

• The Gini index is a measure of node purity small values indicate that node contains predominantly observations from a single class

• In R, this can be estimated using the gain_capture() function.

Classification tree - Heart Disease Example

• Classifying whether 303 patients have heart disease based on 13 predictors (Age, Sex, Chol, etc)

2. Create a model specification that tunes based on complexity, \(\alpha\)

tree_spec <- decision_tree(
  cost_complexity = tune(), 
  tree_depth = 10,
  mode = "classification") %>% 
  set_engine("rpart")

wf <- workflow() |>
  add_recipe(
    recipe(HD ~ Age + Sex + ChestPain + RestBP + Chol + Fbs + 
                     RestECG + MaxHR + ExAng + Oldpeak + Slope + Ca,
    data = heart
    )
  ) |>
  add_model(tree_spec)

3. Fit the model on the cross validation set

grid <- expand_grid(cost_complexity = seq(0.01, 0.05, by = 0.01))
model <- tune_grid(wf,
                   grid = grid,
                   resamples = heart_cv,
                   metrics = metric_set(gain_capture, accuracy)) 

5. Choose \(\alpha\) that minimizes the Gini Index

best <- model %>%
  select_best(metric = "gain_capture")

7. Examine how the final model does on the full sample

final_model %>%
  predict(new_data = heart) %>%
  bind_cols(heart) %>%
  conf_mat(truth = HD, estimate = .pred_class) %>%
  autoplot(type = "heatmap")

Bagging

• bagging is a general-purpose procedure for reducing the variance of a statistical learning method (outside of just trees)

• It is particularly useful and frequently used in the context of decision trees

• Also called bootstrap aggregation

Bagging

• Mathematically, why does this work? Let’s go back to intro to stat!

• If you have a set of \(n\) independent observations: \(Z_1, \dots, Z_n\), each with a variance of \(\sigma^2\), what would the variance of the mean, \(\bar{Z}\) be?

• The variance of \(\bar{Z}\) is \(\sigma^2/n\)

• In other words, averaging a set of observations reduces the variance.

• This is generally not practical because we generally do not have multiple training sets

Bagging

Averaging a set of observations reduces the variance. This is generally not practical because we generally do not have multiple training sets.

• Bootstrap! We can take repeated samples from the single training data set.

Bagging process

• generate \(B\) different bootstrapped training sets

• Train our method on the \(b\)th bootstrapped training set to get \(\hat{f}^{*b}(x)\), the prediction at point \(x\)

• Average all predictions to get: \(\hat{f}_{bag}(x)=\frac{1}{B}\sum_{b=1}^B\hat{f}^{*b}(x)\)

• This is bagging!

Bagging regression trees

• generate \(B\) different bootstrapped training sets
• Fit a regression tree on the \(b\)th bootstrapped training set to get \(\hat{f}^{*b}(x)\), the prediction at point \(x\)
• Average all predictions to get: \(\hat{f}_{bag}(x)=\frac{1}{B}\sum_{b=1}^B\hat{f}^{*b}(x)\)

Bagging classification trees

• for each test observation, record the class predicted by the \(B\) trees

• Take a majority vote - the overall prediction is the most commonly occuring class among the \(B\) predictions

Out-of-bag Error Estimation

• You can estimate the test error of a bagged model

• The key to bagging is that trees are repeatedly fit to bootstrapped subsets of the observations

• On average, each bagged tree makes use of about 2/3 of the observations (you can prove this if you’d like!, not required for this course though)

• The remaining 1/3 of observations not used to fit a given bagged tree are the out-of-bag (OOB) observations

Out-of-bag Error Estimation

You can predict the response for the \(i\)th observation using each of the trees in which that observation was OOB

• This will yield \(B/3\) predictions for the \(i\)th observations. We can average this!

• This estimate is essentially the LOOCV error for bagging as long as \(B\) is large 🎉

Random Forest process

• Like bagging, build a number of decision trees on bootstrapped training samples

• Each time the tree is split, instead of considering all predictors (like bagging), a random selection of \(m\) predictors is chosen as split candidates from the full set of \(p\) predictors

• The split is allowed to use only one of those \(m\) predictors

• A fresh selection of \(m\) predictors is taken at each split

• typically we choose \(m \approx \sqrt{p}\)

1. Randomly divide the data in half, 149 training observations, 148 testing

set.seed(77)
heart_split <- initial_split(heart, prop = 0.5)
heart_train <- training(heart_split)

2. Create model specification

model_spec <- rand_forest(
  mode = "classification",
  mtry = ---
) |> 
  set_engine("ranger")

3. Create the workflow

wf <- workflow() |>
  add_recipe(
    recipe(
      HD ~ Age + Sex + ChestPain + RestBP + Chol + Fbs + 
               RestECG + MaxHR + ExAng + Oldpeak + Slope + Ca + Thal,
             data = heart_train
    )
  ) |>
  add_model(rf_spec)

5. Examine how it looks in the test data

heart_test <- testing(heart_split)
model |>
  predict(new_data = heart_test) |>
  bind_cols(heart_test) |>
  conf_mat(truth = HD, estimate = .pred_class) |>
  autoplot(type = "heatmap")

Trade Off