Summary: Machine Learning on the Rosanne-ABC Firing Incident Dataset

Summary of results: which methodology/modality “wins?”

Vanilla Merged
Algorithm Speed CCI % ROC AUC RMSE F-1 CCI % ROC AUC F-1 RMSE
ZeroR Instant 37.9333 0.4990 0.4144 NULL 47.1000 0.4990 NULL 0.4536
OneR Instant 43.0000 0.5420 0.5339 NULL 52.9833 0.5660 NULL 0.5599
NaiveBayes Fast 63.8500 0.8160 0.3808 0.6410 63.9667 0.8000 0.6430 0.4374
IBK Fast 56.5333 0.6910 0.4386 0.5230 59.5833 0.6510 0.5470 0.4972
RandomTree Fast 59.5833 0.6800 0.4474 0.5920 62.7167 0.6700 0.6210 0.4954
SimpleLogistic Moderate 73.6500 0.8850 0.3065 0.7320 73.6500 0.8730 0.7300 0.3502
DecisionTable Slow too slow for viable computation on consumer-grade hardware
MultilayerPerceptron Slow
RandomForest Slow
Vanilla Merged
Meta-Classifier Speed CCI % ROC AUC RMSE F-1 CCI % ROC AUC F-1 RMSE
Stack (ZR, NB) Moderate 37.9333 0.4990 0.4144 NULL vacuous results, omitted
Stack (NB. RT) Moderate 63.7000 0.8230 0.3795 0.6350 61.9833 0.6980 0.6130 0.4523
Vote (ZR, NB, RT) Moderate 62.0833 0.8430 0.3414 0.6110 64.0500 0.8330 0.6260 0.3830
CostSensitive (ZR) Instant 37.9333 0.4990 0.4144 NULL 36.6667 0.4990 NULL 0.4623
CostSensitive (OR) Instant 42.7000 0.5400 0.5353 NULL 39.6167 0.5170 NULL 0.6345
CostSensitive (NB) Fast 63.8500 0.8160 0.3808 0.6410 64.0833 0.8010 0.6450 0.4365
CostSensitive (IBK) Fast 56.5333 0.6910 0.4386 0.5230 59.5833 0.6510 0.5470 0.4972
CostSensitive (RT) Fast 59.5833 0.6800 0.4474 0.5920 63.3833 0.7050 0.6350 0.4728
CostSensitive (SL) Moderate 73.6500 0.8850 0.3065 0.7320 74.7833 0.8780 0.7450 0.3478

My results are contained in a separate text file in lab journal format. Salient results consisted of:

My methodological strategy began with a wide selection of algorithms. In particular, I was concerned about the trade-off between speed and accuracy. If one algorithm yields >80% accuracy (NB: none of those I queried did so) but takes days to compute (eg. Multilayer Perceptron), it may be unsuitable for rapid analysis of the constantly-changed media landscape in which we work.

An algorithm that takes an hour and returns ~70% accuracy may be more desirable (as was the case with SimpleLogistic). Furthermore, depending on the usage case (such as user interfaces or a mobile app), one might prefer something with lower accuracy but near-instantaneous results (NaiveBayes being an excellent candidate).

Another concern is whether certain modalities can demonstrate the rather cynical scenario where people willing to publicly defend a racist cannot be easily discerned from noise and chatter. I shall return to this concern shortly.

All-in-all, NaiveBayes performed well in terms of worst-case computation time, taking very little time to provide results. Using what I refer to as the “vanilla” dataset, results came with a weighted average ROC AUC just over 0.8 and approximately 64% correctly classified instances. On the other hand, ZeroR performed horribly — basically classifying everything in one class (NB: this is still better than just randomly guessing).

A dark horse contender, however, showed up late in the game: SimpleLogistic. SimpleLogistic took under an hour to build a model and run a 10-fold cross-validation while returning better accuracy (CCI %, ROC AUC, RMSE) than the other algorithms I queried. It took about 30-40 minutes to run this depending such modalities as merged data, penalties, and so on. I find the speed-accuracy trade-off to be reasonable. Furthermore, unlike almost other algorithms investigated, applying CostSensitiveClassifier to merged SimpleLogistic yielded slightly improved accuracy (in terms of CCI% and ROC AUC); however, this came at a cost of a higher RMSE.

Lastly, aside from the observations made in the previous paragraph, using meta-classification strategies such as voting/stacking and introducing penalties did not yield very different results from the algorithms they modulated, but were none-the-less interesting to observe.

Upon further scrutiny, I felt tempted to give merit to the rather bold claim that Pro-Rosanne rhetoric is hard to discern from noise (suggested by running the RandomTree algorithm with penalties). A counter-example to this was discovered when running vanilla IBK with penalties (Anti-Roseanne came back misclassified as Unclear/Unrelated rather often).

This underscores a subtle but critical notion: just as one entertains a moral hazard by cherry-picking data, one may also encounter into a similar, far graver problem by bashing models to support a claim and hiding behind algorithms as a sort of unassailable black box.

One thought on “Summary: Machine Learning on the Rosanne-ABC Firing Incident Dataset

Comments are closed.