Classification Assessment

Paired t-Test and Bayes Classifier

In this assignment you will evaluate the performance of the Full Bayes and Naive Bayes classifier on the iris.txt dataset. Each point has four numeric dimensions, and the last dimension is the class. You have to assess via the paired t-test whether the classifiers are significantly different or not.

You will implement Algorithm 22.4 in chapter 22, except that instead of the K-fold cross validation, use K rounds of bootstrap resampling (as described in Algorithm 22.3:lines 1-4). For each bootstrap sample \(\mathbf{D}_i\) you need to learn the paramerers of the Bayes classifiers -- full and naive -- and then test on \(\mathbf{D} \setminus \mathbf{D}_i\) and record the error rates. Finally tabulate the difference in the performance of the full and naive Bayes classifier for each round. From the difference values in each of the \(K\) rounds compute the \(z\)-score value (line 9; Algorithm 22.4) and then test if the two classifiers are different or not at \(\alpha=0.95\) and \(\alpha=0.99\) confidence level. Use \(K=30\).

Note that the only difference between the full and naive Bayes classifier is that the former uses the full covariance matrix, whereas the latter uses a diagonal one.

You should implement the Bayes classifiers on your own, but you may use builtin functions to find out the critical values for the t-distribution for a given confidence level \(\alpha\). Note: In python you will find scipy.stats module useful for this; In R, you may use qt).