publications

2024

1. NeurIPS

   Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

   A. Akgül, M. Haussmann, and M. Kandemir

   In Neural Information Processing Systems 2024

   Abs Bib HTML Code

   Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value iteration, use Monte Carlo sampling to estimate the Bellman target to perform temporal difference based policy evaluation. We find out that the randomness caused by this sampling step significantly delays convergence. We present a theoretical result demonstrating the strong dependency of suboptimality on the number of Monte Carlo samples taken per Bellman target calculation. Our main contribution is a deterministic approximation to the Bellman target that uses progressive moment matching, a method developed originally for deterministic variational inference. The resulting algorithm, which we call Moment Matching Offline Model-Based Policy Optimization (MOMBO), propagates the uncertainty of the next state through a nonlinear Q-network in a deterministic fashion by approximating the distributions of hidden layer activations by a normal distribution. We show that it is possible to provide tighter guarantees for the suboptimality of MOMBO than the existing Monte Carlo sampling approaches. We also observe MOMBO to converge faster than these approaches in a large set of benchmark tasks.

   @inproceedings{akgul2024deterministic,
     title = {Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning},
     author = {Akgül, A. and Haussmann, M. and Kandemir, M.},
     year = {2024},
     booktitle = {Neural Information Processing Systems},
   }

2. AABI

   PAC-Bayesian Soft Actor-Critic Learning

   B. Tasdighi, A. Akgül, M. Haussmann, and 2 more authors

   In Advances in Approximate Bayesian Inference Symposium 2024

   Abs Bib HTML

   Actor-critic algorithms address the dual goals of reinforcement learning, policy evaluation and improvement, via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that the online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably to the state of the art on multiple classical control and locomotion tasks in both sample efficiency and asymptotic performance.

   @inproceedings{bahareh2024pac4sac,
     title = {PAC-Bayesian Soft Actor-Critic Learning},
     author = {Tasdighi, B. and Akgül, A. and Haussmann, M. and Brink, K.K. and Kandemir, M.},
     year = {2024},
     booktitle = {Advances in Approximate Bayesian Inference Symposium},
   }

3. ICLR

   Calibrating Bayesian UNet++ for Sub-Seasonal Forecasting

   B. Asan, Abdullah Akgül, A. Unal, and 2 more authors

   In Tackling Climate Change with Machine Learning at ICLR 2024 2024

   Abs Bib HTML

   Seasonal forecasting is a crucial task when it comes to detecting the extreme heat and colds that occur due to climate change. Confidence in the predictions should be reliable since a small increase in the temperatures in a year has a big impact on the world. Calibration of the neural networks provides a way to ensure our confidence in the predictions. However, calibrating regression models is an under-researched topic, especially in forecasters. We calibrate a UNet++ based architecture, which was shown to outperform physics-based models in temperature anomalies. We show that with a slight trade-off between prediction error and calibration error, it is possible to get more reliable and sharper forecasts. We believe that calibration should be an important part of safety-critical machine learning applications such as weather forecasters.

   @inproceedings{asan2024calibrating,
     title = {Calibrating Bayesian UNet++ for Sub-Seasonal Forecasting},
     author = {Asan, B. and Akgül, Abdullah and Unal, A. and Kandemir, M. and Unal, G.},
     booktitle = {Tackling Climate Change with Machine Learning at ICLR 2024},
     year = {2024},
   }

4. L4DC

   Continual Learning of Multi-modal Dynamics with External Memory

   Abdullah Akgül, G. Unal, and M. Kandemir

   In Proceedings of The 6th Annual Learning for Dynamics and Control Conference 2024

   Abs Bib HTML

   We study the problem of fitting a model to a dynamical environment when new modes of behavior emerge sequentially. The learning model is aware when a new mode appears, but it does not have access to the true modes of individual training sequences. We devise a novel continual learning method that maintains a descriptor of the mode of an encountered sequence in a neural episodic memory. We employ a Dirichlet Process prior on the attention weights of the memory to foster efficient storage of the mode descriptors. Our method performs continual learning by transferring knowledge across tasks by retrieving the descriptors of similar modes of past tasks to the mode of a current sequence and feeding this descriptor into its transition kernel as control input. We observe the continual learning performance of our method to compare favorably to the mainstream parameter transfer approach.

   @inproceedings{akgul2024cddp,
     title = {Continual Learning of Multi-modal Dynamics with External Memory},
     author = {Akgül, Abdullah and Unal, G. and Kandemir, M.},
     booktitle = {Proceedings of The 6th Annual Learning for Dynamics and Control Conference},
     year = {2024},
   }

2023

2022

1. ICLR

   Evidential Turing Processes

   M. Kandemir, Abdullah Akgül, M. Haussmann, and 1 more author

   In International Conference on Learning Representations 2022

   Abs Bib HTML

   A probabilistic classifier with reliable predictive uncertainties i) fits successfully to the target domain data, ii) provides calibrated class probabilities in difficult regions of the target domain (e.g. class overlap), and iii) accurately identifies queries coming out of the target domain and reject them. We introduce an original combination of Evidential Deep Learning, Neural Processes, and Neural Turing Machines capable of providing all three essential properties mentioned above for total uncertainty quantification. We observe our method on three image classification benchmarks to consistently improve the in-domain uncertainty quantification, out-of-domain detection, and robustness against input perturbations with one single model. Our unified solution delivers an implementation-friendly and computationally efficient recipe for safety clearance and provides intellectual economy to an investigation of algorithmic roots of epistemic awareness in deep neural nets.

   @inproceedings{kandemir2022evidential,
     title = {Evidential Turing Processes },
     author = {Kandemir, M. and Akgül, Abdullah and Haussmann, M. and Unal, G.},
     booktitle = {International Conference on Learning Representations},
     year = {2022},
     url = {https://openreview.net/forum?id=84NMXTHYe-},
   }

2. FL-NeurIPS

   How to Combine Variational Bayesian Networks in Federated Learning

   A. Ozer, K.B. Buldu, Abdullah Akgül, and 1 more author

   In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022) 2022

   Abs Bib HTML

   Federated Learning enables multiple data centers to train a central model collaboratively without exposing any confidential data. Even though deterministic models are capable of performing high prediction accuracy, their lack of calibration and capability to quantify uncertainty is problematic for safety-critical applications. Different from deterministic models, probabilistic models such as Bayesian neural networks are relatively well-calibrated and able to quantify uncertainty alongside their competitive prediction accuracy. Both of the approaches appear in the federated learning framework; however, the aggregation scheme of deterministic models cannot be directly applied to probabilistic models since weights correspond to distributions instead of point estimates. In this work, we study the effects of various aggregation schemes for variational Bayesian neural networks. With empirical results on three image classification datasets, we observe that the degree of spread for an aggregated distribution is a significant factor in the learning process. Hence, we present an survey on the question of how to combine variational Bayesian networks in federated learning, while providing computer vision classification benchmarks for different aggregation settings.

   @inproceedings{ozer2022fl,
     title = {How to Combine Variational Bayesian Networks in Federated Learning},
     author = {Ozer, A. and Buldu, K.B. and Akgül, Abdullah and Unal, G.},
     booktitle = {Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022)},
     year = {2022},
     url = {https://openreview.net/forum?id=AkPwb9dvAlP},
   }