Jan Hendrik Metzen

My personal blog on python and machine learning

Compare Classifier Predictions using Reliability Diagrams

Reliability diagrams are useful for checking if the predicted probabilities of a binary classifier are well calibrated. For perfectly calibrated predictions, the curve in a reliability diagram should be as close as possible to the diagonal/identity. This post compares the reliability diagrams of different classfiers on artifical data and discusses the respective pros and cons.