With an ever-increasing number of real-world applications of machine learning (ML) methodology, a certain level of self-perception, and especially uncertainty-awareness, of ML agents is becoming indispensable, all the more in safety-critical domains such as medicine or cyber-physical systems. One important concept in this regard is the notion of a reject option, i.e., the possibility for the learner to abstain from a prediction in cases of uncertainty. In fact, it is a common view that a learner should better abstain from making a prediction when not being certain enough, rather than engaging itself in unsubstantiated speculation or random guessing.

In classification settings, corresponding approaches are often referred to as classifier abstention or classification with reject option. In the machine learning community, quite a number of methods for learning and prediction with a reject option have been proposed in recent years. In this tutorial, we aim to provide a comprehensive overview of these methods. We focus on multi-class classification because the overwhelming majority of existing methods addresses that setting in particular. Going beyond the standard reject option, we also cover recent work on set-valued prediction, i.e., the idea of predicting a set of possible classes instead of committing to a single one. Set-valued predictions can be seen as partial abstentions and comprise a complete reject (prediction of the entire set of classes) as a special case.

Structure

Part 1: Introduction - 10 minutes (Eyke)
Part 2: Uncertainty in classification: representation and methods - 40 minutes (Eyke)
Part 3: Classification with reject option - 25 minutes (Titouan)
Part 4: Set-valued prediction - 25 minutes (Titouan)
Part 5: Rejection and set-valued prediction in settings with distribution shift - 40 minutes (Willem)

Presenters

Willem Waegeman, Ghent University, Belgium
Titouan Lorieul, INRIA, University of Montpellier, France
Eyke Hüllermeier, University of Munich, Germany

Target audience

The tutorial intends to cover an overview of existing methods, while focussing on general concepts. With the tutorial we aim to attract both researchers that are already active in one of the above domains, as well as researchers with little or no prior experience in uncertainty-aware classification. As such, we believe that the tutorial will attract ECML/PKDD attendees from diverse subfields of machine learning and with different background.

Practical information

For registration, see the ECML/PKDD 2021 website. The tutorial starts at 2:30pm on September 17th.

Slides

Slides can be downloaded using this link.