KNN

• In theory we would always like to predict qualitative responses using the Bayes classifier.
• For real data, we do not know the conditional distribution of Y given X, and so computing the Bayes classifier is impossible.
• Many approaches attempt to estimate the conditional distribution of Y given X, and then classify a given observation to the class with highest estimated probability. One such method is the K-nearest neighbors (KNN) classifier.

KNN

• Given positive integer \(K\), and a test observation \(x_0\), KNN identifies the \(K\) points in the training data that are closest to \(x_0\), represented by \(\mathcal{N}_0\).

• KNN then estimates the conditional probabilities for each class \(j\) as the fraction of the points in \(\mathcal{N}_0\) whose response values equal \(j\):

• \[P(Y=j|X=x_0)=\frac{1}{K}\sum_{i\in \mathcal{N}}I(y_i=j)\]

• Lastly, KNN classifies \(x_0\) into the class with the largest probability

KNN Example

The KNN approach, using \(K = 3\), is illustrated in a situation with six blue observations and six orange observations.

KNN Example

The KNN approach, using \(K = 3\), is illustrated in a situation with six blue observations and six orange observations.

KNN Example KNN vs Bayes

KNN Low and High Flexibility

• Comparison of KNN decision boundaries (solid black curves) obtained using \(K = 1\) and \(K = 100\).
  • The \(K = 1\) decision boundary is overly flexible and the \(K = 100\) boundary is not sufficiently flexible.
• The Bayes decision boundary is shown as a purple dashed line.

Naive Bayes

• In LDA and QDA we assumed the predictors were Normal
  • This allow us to find the density functions \(f_k(x)\)’s by optimizing a linear or quadratic function \(\delta_k(x)\).
  • With LDA, we assume the predictors have the same covariance structure between classes
  • With QDA, \(X_j\)s can have any covariance structure
  • With both, the predictors must be quantitative

Naive Bayes

• With Naive Bayes we drop the normality assumption but introduce a must stronger assumption
  • Within a class \(k\),
  • The predictors are independent

Naive Bayes - Why?

• Each \(f\) is one dimensional!
• If \(X_j\) is quantitative can still assume each \(X_j|Y=k\) is univariate normal
• If \(X_j\) is quantitative, then another option is to use a non-parametric estimate for \(f_{kj}\)
  • Make a histogram for the observations of the \(j\)th predictor within each class.
  • Then we can estimate \(f_{kj}(x_j)\) as the fraction of the training observations in the \(k\)th class that belong to the same histogram bin as \(x_j\) .
  • We can use a kernel density estimator, which is kernel density estimator (essentially a smoothed version of a histogram)

Naive Bayes - Why?

• If \(X_j\) is qualitative, then we can simply count the proportion of training observations for the \(j\)th predictor corresponding to each class.
  • Suppose that \(X_j \in \{1, 2, 3\}\), and we have 100 observations in the \(k\)th class.
  • Suppose that the \(j\)th predictor takes on values of 1,2, and 3 in 32, 55, and 13 of those observations, respectively. Then we can estimate \(f_{kj}\) as
• \[\hat{f}_{kj}(x_j) = \begin{cases} .32 \quad \text{if }x_j = 1\\ .55 \quad \text{if }x_j = 2\\ .13 \quad \text{if }x_j = 3\\ \end{cases}\]

Naive Bayes - Statsquest

https://www.youtube.com/watch?v=O2L2Uv9pdDA

Comparing LDA, QDA, and Normal Naive Bayes

https://towardsdatascience.com/differences-of-lda-qda-and-gaussian-naive-bayes-classifiers-eaa4d1e999f6

Naive Bayes - Given Y, the predictors X are conditionally independent.

LDA - LDA assumes that the covariance matrix across classes is the same.

QDA - QDA does not assume constant covariance matrix across classes.