Confidence AI & ML | Trustworthy Artificial Intelligence Research Lab

High Confidence and Trustable Machine Learning

Many real-world applications require a high degree of confidence in the correctness of the decisions made by AI/ML models. Since even the classifier models with high accuracy (e.g., >95%) still introduce some misclassification error, it may not be possible to rely on the output of a classifier.

I lead research on the study of a new generation of algorithms to increase human-AI interaction and our confidence level in AI and ML models. I have developed an innovative methodology that allows AI/ML models to learn to identify difficult-to-decide data samples and flag them as ‘rejects.’ Other data are classified as usual. After training, the model can flag difficult-to-decide items, indicating that it rejects to make a decision on these items and defers them to a human expert with necessary precautions. The technique helps the model to yield only the accurate output, thus avoiding misinformation or incorrect decisions.

WisdomeNet: Trustable Machine Learning in Neural Networks

For this method’s approach in neural networks, I created WisdomNet, which can determine what it cannot correctly classify. WisdomNet is a neural network architecture for trustable learning that can determine what it cannot correctly classify. Its name is inspired by Aristotle’s quote, “Knowing yourself is the beginning of all wisdom.’

The WisdomNet is built from a trained neural network by adding a ‘conjugate neuron’ to the output layer (see figure bellow for examples). The conjugate neuron shall be trained to learn the doubtful data that the original network might incorrectly classify. The WisdomNet inherits the original network’s knowledge and then is trained to learn more from the doubtful data samples. WisdomNet has been applied to several network architectures, such as multilayer perceptron, convolutional neural networks, and deep neural networks. WisdomNet can make accurate predictions and reduce the error rate to as low as 0% while indicating the doubtful data as ‘reject’ at a low rate of around 10%.

Else-Tree Classifier to Minimize Misclassification

Traditional decision tree classifiers are based on an impurity measure that identifies the most informative attribute to be selected at the early levels of a decision tree. The class labels are chosen based on the majority of class labels at a leaf node.

We proposed a novel decision tree classifier, Else-Tree, by analyzing pure regions or ranges of an attribute per class. After identifying the longest or most populated contiguous range per class. The rest of the ranges are fed into else branch of the decision tree. Only conflicting or doubtful samples are passed to the lower levels of the decision tree. It does not necessarily assign a class for difficult samples to classify.

We have used our protein crystallization trial data and three other publicly available datasets to evaluate else-tree. The experiments show that the else-tree may reduce the misclassification to 0% by labeling difficult samples as undecided when the training set is a good representation of the dataset.