Evidential Turing Processes

Introduction

Reliable machine learning models deployed in safety-critical settings (medical diagnostics, autonomous driving, natural language understanding) must handle three distinct kinds of uncertainty: uncertainty in the model’s own fit to the data, uncertainty arising from intrinsic class ambiguity, and uncertainty about whether an input comes from the same domain the model was trained on. Existing Bayesian approaches address these challenges in a piecemeal fashion: Parametric Bayesian Models (PBMs) such as Bayesian Neural Networks (BNNs) capture model-level uncertainty well but poorly represent local class ambiguity; Evidential Bayesian Models (EBMs) such as Evidential Deep Learning (EDL) quantify class overlap well but lack the global structure needed for out-of-distribution detection. Evidential Turing Processes (ETP) introduces a principled unification through the Complete Bayesian Model (CBM) framework, which provably inherits the favorable uncertainty decomposition properties of both approaches, realized via a novel Turing Process: a stochastic process with an external memory that accumulates in-domain evidence without requiring a context set at test time.

In this work (2nd author), I designed and conducted all experiments and wrote the experimental section of the paper.

Problem Statement

The paper formally defines Total Calibration as the simultaneous competence of a discriminative predictor in three distinct tasks:

• Model misfit: how closely the model’s predicted class distribution matches the true class-conditional distribution, measured by Negative Log-Likelihood (NLL). Reducible by seeing more training data; signals systematic errors in model fit.

• Class overlap: how faithfully the model reflects the irreducible ambiguity at decision boundaries (where two classes genuinely overlap), measured by Expected Calibration Error (ECE). Cannot be eliminated with more data; requires a local uncertainty mechanism.

• Domain mismatch: the ability to detect inputs drawn from a distribution unseen during training (out-of-distribution detection), measured by the Area Under the ROC Curve (AUROC). Requires a global uncertainty signal that builds up over the training set.

The key observation: PBMs maintain a posterior over a global parameter θ; this variance shrinks with data and drives good NLL and AUROC, but poorly captures per-sample class ambiguity. EBMs maintain an observation-specific prior over class probabilities π; this captures class overlap via ECE, but has no global variable to detect domain shift. No prior method achieves all three simultaneously.

Methodology

Complete Bayesian Models (CBM)

The CBM framework introduces a global parameter θ (from PBM) alongside a local class-probability variable π (from EBM). Their joint generative model is:

The CBM variance decomposition contains all three required uncertainty components:

The first term recovers the reducible model uncertainty of PBMs (handles model misfit and domain mismatch); the second and third recover the irreducible and data uncertainty of EBMs (handles class overlap). A CBM is the minimal structure that guarantees total calibration.

Turing Processes and the ETP

Equipping the CBM’s empirical prior $p(\pi \mid \theta, x)$ with a mechanism that learns to be accurate on individual samples requires two ingredients: (1) a global random variable connecting samples across the dataset (as in Neural Processes [Garnelo et al., 2018]), and (2) a memory that accumulates evidence from context data without needing to store that context at test time (as in Neural Turing Machines [Graves et al., 2014]).

The Turing Process is a formally defined stochastic process combining both: its prior $p_M(\theta)$ has free parameters $M$ (the memory), and conditioning on a context set $\mathcal{D}_C$ updates the memory parameters via an explicit rule $r$: $M’ = r(M, \mathcal{D}_C)$. Unlike Neural Processes, no context is required at prediction time, since the memory has absorbed the context during training.

Evidential Turing Processes (ETP) instantiate this design as a CBM:

The global parameters $\theta = {w, Z}$ split into a neural network weight $w$ and a memory $Z = {z_1, \ldots, z_R}$ parameterized by $M = (m_1, \ldots, m_R)$. Each input $x$ is encoded and matched to memory via attention, yielding input-specific Dirichlet hyperparameters for π. Memory cells are updated by a convex interpolation rule that writes new uncertainty information (both the ground-truth label and the model’s current soft prediction) into cells weighted by their similarity to the current input, without gradient-based optimization of $M$.

Ablation structure: Deactivating ETP components one at a time recovers established baselines. Removing the external memory and update rule recovers the Evidential Neural Process (ENP) (a novel surrogate for the Attentive Neural Process [Kim et al., 2019]); further removing the local variable π recovers the standard Neural Process; removing the global Z recovers EDL; removing both π and Z recovers a BNN.

Results

Datasets: Five real-world classification benchmarks covering text and images, each with a designated out-of-distribution dataset for AUROC evaluation:

Results are averages over 10 random seeds. Baselines are BNN, EDL, and ENP (curated as the strongest possible ablation). ETP is the only method that consistently ranks among top performers across all three total-calibration metrics on every dataset.

Total Calibration Results (mean ± std, 10 seeds)

Test error ↓ — model fit:

ECE ↓ — class overlap calibration:

NLL ↓ — negative log-likelihood:

ETP achieves the best NLL on all five datasets, a result no other method matches.

AUROC ↑ — out-of-distribution detection:

Bold entries are within three standard deviations of the best result. ETP is the only method that never catastrophically fails on any metric: EDL collapses on IMDB accuracy (38.3% error) and NLL universally; ENP collapses on IMDB (50% error, no better than chance) and NLL; BNN fails at ECE across the board. ETP is the only model that simultaneously achieves top-tier NLL on all five datasets while remaining competitive on ECE and AUROC.

Robustness Against Gradual Domain Shift

Models are evaluated on corrupted variants of three datasets (FMNIST-C, CIFAR10-C, SVHN-C) using 19 corruption types (e.g., motion blur, fog, pixelation) at five severity levels. The ECE metric is tracked at each corruption level.

Computational Cost

Measured on CIFAR10, wall-clock time per epoch: ETP (10.8 ± 0.1 s), BNN (8.2 ± 0.1 s), EDL (8.6 ± 0.1 s), ENP (9.5 ± 0.2 s). ETP incurs a ~14% overhead versus the next-closest baseline (ENP), a modest cost relative to the consistent gain across all three calibration axes.

Conclusion

• Total Calibration formalized: Model misfit, class overlap, and domain mismatch are defined as three formally distinct uncertainty types, each with a designated metric (NLL, ECE, AUROC), providing a rigorous evaluation protocol for uncertainty-aware classifiers.
• CBM as a unifying theory: The Complete Bayesian Model framework shows analytically that a model combining a global (PBM-style) and a local (EBM-style) random variable inherits the favorable uncertainty decomposition of each, a strictly necessary structure for total calibration.
• Turing Process as a practical realization: The external memory mechanism accumulates uncertainty evidence during training without requiring a context set at test time, making ETP suitable for standard (non-meta-learning) classification pipelines.
• Unique simultaneous performance: ETP is the only method among all compared approaches that consistently ranks among top performers on all three calibration metrics across all five real-world datasets.
• Corruption robustness: The ECE advantage holds under 19 types of data corruption at five severity levels, confirming that ETP’s calibration improvements reflect a genuine improvement in uncertainty structure rather than overfitting to clean-domain statistics.

References

1. Kandemir, M., Akgül, A., Haussmann, M., & Unal, G. (2022). Evidential Turing Processes. International Conference on Learning Representations (ICLR 2022). Code
2. Graves, A., Wayne, G., & Danihelka, I. (2014). Neural Turing Machines. arXiv:1410.5401.
3. Garnelo, M., Rosenbaum, D., Maddison, C. J., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D. J., & Eslami, S. M. A. (2018). Neural Processes. ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models.
4. Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, S. M. A., Rosenbaum, D., Vinyals, O., & Teh, Y. W. (2019). Attentive Neural Processes. ICLR 2019.
5. Sensoy, M., Kaplan, L., & Kandemir, M. (2018). Evidential Deep Learning to Quantify Classification Uncertainty. NeurIPS 2018.
6. Malinin, A., & Gales, M. (2018). Predictive Uncertainty Estimation via Prior Networks. NeurIPS 2018.
7. Hendrycks, D., & Dietterich, T. (2019). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR 2019.