Publications HAL de Odalric-Ambrym Maillard

2021

Conference papers

titre
From Optimality to Robustness: Dirichlet Sampling Strategies in Stochastic Bandits
auteur
Dorian Baudry, Patrick Saux, Odalric-Ambrym Maillard
article
Neurips 2021, Dec 2021, Sydney, Australia
resume
The stochastic multi-arm bandit problem has been extensively studied under standard assumptions on the arm's distribution (e.g bounded with known support, exponential family, etc). These assumptions are suitable for many real-world problems but sometimes they require knowledge (on tails for instance) that may not be precisely accessible to the practitioner, raising the question of the robustness of bandit algorithms to model misspecification. In this paper we study a generic Dirichlet Sampling (DS) algorithm, based on pairwise comparisons of empirical indices computed with re-sampling of the arms' observations and a data-dependent exploration bonus. We show that different variants of this strategy achieve provably optimal regret guarantees when the distributions are bounded and logarithmic regret for semi-bounded distributions with a mild quantile condition. We also show that a simple tuning achieve robustness with respect to a large class of unbounded distributions, at the cost of slightly worse than logarithmic asymptotic regret. We finally provide numerical experiments showing the merits of DS in a decision-making problem on synthetic agriculture data.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-03421252/file/main.pdf BibTex
titre
Stochastic bandits with groups of similar arms
auteur
Fabien Pesquerel, Hassan Saber, Odalric-Ambrym Maillard
article
NeurIPS, Dec 2021, Sydney, Australia
resume
We consider a variant of the stochastic multi-armed bandit problem where arms are known to be organized into different groups having the same mean. The groups are unknown but a lower bound q on their size is known. This situation typically appears when each arm can be described with a list of categorical attributes, and the (unknown) mean reward function only depends on a subset of them, the others being redundant. In this case, q is linked naturally to the number of attributes considered redundant, and the number of categories of each attribute. For this structured problem of practical relevance, we first derive the asymptotic regret lower bound and corresponding constrained optimization problem. They reveal the achievable regret can be substantially reduced when compared to the unstructured setup, possibly by a factor q. However, solving exactly the exact constrained optimization problem involves a combinatorial problem. We introduce a lowerbound inspired strategy involving a computationally efficient relaxation that is based on a sorting mechanism. We further prove it achieves a lower bound close to the optimal one up to a controlled factor, and achieves an asymptotic regret q times smaller than the unstructured one. We believe this shows it is a valuable strategy for the practitioner. Last, we illustrate the performance of the considered strategy on numerical experiments involving a large number of arms.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-03427597/file/Neurips_Submission.pdf BibTex
titre
Routine Bandits: Minimizing Regret on Recurring Problems
auteur
Hassan Saber, Léo Saci, Odalric-Ambrym Maillard, Audrey Durand
article
ECML-PKDD 2021, Sep 2021, Bilbao, Spain
resume
We study a variant of the multi-armed bandit problem in which a learner faces every day one of B many bandit instances, and call it a routine bandit. More specifically, at each period h ∈ [1, H] , the same bandit b^h is considered during T > 1 consecutive time steps, but the identity b^h is unknown to the learner. We assume all rewards distribution are Gaussian standard. Such a situation typically occurs in recommender systems when a learner may repeatedly serve the same user whose identity is unknown due to privacy issues. By combining banditidentification tests with a KLUCB type strategy, we introduce the KLUCB for Routine Bandits (KLUCB-RB) algorithm. While independently running KLUCB algorithm at each period leads to a cumulative expected regret of Ω(H log T) after H many periods when T → ∞, KLUCB-RB benefits from previous periods by aggregating observations from similar identified bandits, which yields a non-trivial scaling of Ω(log T). This is achieved without knowing which bandit instance is being faced by KLUCB-RB on this period, nor knowing a priori the number of possible bandit instances. We provide numerical illustration that confirm the benefit of KLUCB-RB while using less information about the problem compared with existing strategies for similar problems.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-03286539/file/ECML2021_RoutineBandits%20%28Camera-Ready%29.pdf BibTex
titre
Learning Value Functions in Deep Policy Gradients using Residual Variance
auteur
Yannis Flet-Berliac, Reda Ouhamma, Odalric-Ambrym Maillard, Philippe Preux
article
ICLR 2021 - International Conference on Learning Representations, May 2021, Vienna / Virtual, Austria
resume
Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic uses a new state-value (resp. state-action-value) function approximation that learns the value of the states (resp. state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02964174/file/iclr_avec.pdf BibTex

2020

Conference papers

titre
Robust-Adaptive Interval Predictive Control for Linear Uncertain Systems
auteur
Edouard Leurent, Denis Efimov, Odalric-Ambrym Maillard
article
CDC 2020 - 59th IEEE Conference on Decision and Control, Dec 2020, Jeju Island / Virtual, South Korea
resume
We consider the problem of stabilization of a linear system, under state and control constraints, and subject to bounded disturbances and unknown parameters in the state matrix. First, using a simple least square solution and available noisy measurements, the set of admissible values for parameters is evaluated. Second, for the estimated set of parameter values and the corresponding linear interval model of the system, two interval predictors are recalled and an unconstrained stabilizing control is designed that uses the predicted intervals. Third, to guarantee the robust constraint satisfaction, a model predictive control algorithm is developed, which is based on solution of an optimization problem posed for the interval predictor. The conditions for recursive feasibility and asymptotic performance are established. Efficiency of the proposed control framework is illustrated by numeric simulations.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-02942414/file/CDC20_Edouard.pdf BibTex
titre
Sub-sampling for Efficient Non-Parametric Bandit Exploration
auteur
Dorian Baudry, Emilie Kaufmann, Odalric-Ambrym Maillard
article
NeurIPS 2020, Dec 2020, Vancouver, Canada
resume
In this paper we propose the first multi-armed bandit algorithm based on re-sampling that achieves asymptotically optimal regret simultaneously for different families of arms (namely Bernoulli, Gaussian and Poisson distributions). Unlike Thompson Sampling which requires to specify a different prior to be optimal in each case, our proposal RB-SDA does not need any distribution-dependent tuning. RB-SDA belongs to the family of Sub-sampling Duelling Algorithms (SDA) which combines the sub-sampling idea first used by the BESA [1] and SSMC [2] algorithms with different sub-sampling schemes. In particular, RB-SDA uses Random Block sampling. We perform an experimental study assessing the flexibility and robustness of this promising novel approach for exploration in bandit models.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02977552/file/sda_hal.pdf BibTex
titre
Robust-Adaptive Control of Linear Systems: beyond Quadratic Costs
auteur
Edouard Leurent, Denis Efimov, Odalric-Ambrym Maillard
article
NeurIPS 2020 - 34th Conference on Neural Information Processing Systems, Dec 2020, Vancouver / Virtual, Canada
resume
We consider the problem of robust and adaptive model predictive control (MPC) of a linear system, with unknown parameters that are learned along the way (adaptive), in a critical setting where failures must be prevented (robust). This problem has been studied from different perspectives by different communities. However, the existing theory deals only with the case of quadratic costs (the LQ problem), which limits applications to stabilisation and tracking tasks only. In order to handle more general (non-convex) costs that naturally arise in many practical problems, we carefully select and bring together several tools from different communities, namely non-asymptotic linear regression, recent results in interval prediction, and tree-based planning. Combining and adapting the theoretical guarantees at each layer is non trivial, and we provide the first end-to-end suboptimality analysis for this setting. Interestingly, our analysis naturally adapts to handle many models and combines with a data-driven robust model selection strategy, which enables to relax the modelling assumptions. Last, we strive to preserve tractability at any stage of the method, that we illustrate on two challenging simulated environments.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-03004060/file/main.pdf BibTex
titre
Monte-Carlo Graph Search: the Value of Merging Similar States
auteur
Edouard Leurent, Odalric-Ambrym Maillard
article
ACML 2020 - 12th Asian Conference on Machine Learning, Nov 2020, Bangkok / Virtual, Thailand. pp.577 - 602
resume
We consider the problem of planning in a Markov Decision Process (MDP) with a generative model and limited computational budget. Despite the underlying MDP transitions having a graph structure, the popular Monte-Carlo Tree Search algorithms such as UCT rely on a tree structure to represent their value estimates. That is, they do not identify together two similar states reached via different trajectories and represented in separate branches of the tree. In this work, we propose a graph-based planning algorithm, which takes into account this state similarity. In our analysis, we provide a regret bound that depends on a novel problem-dependent measure of difficulty, which improves on the original tree-based bound in MDPs where the trajectories overlap, and recovers it otherwise. Then, we show that this methodology can be adapted to existing planning algorithms that deal with stochastic systems. Finally, numerical simulations illustrate the benefits of our approach.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-03004124/file/main.pdf BibTex
titre
Tightening Exploration in Upper Confidence Reinforcement Learning
auteur
Hippolyte Bourel, Odalric-Ambrym Maillard, Mohammad Talebi
article
International Conference on Machine Learning, Jul 2020, Vienna, Austria
resume
The upper confidence reinforcement learning (UCRL2) algorithm introduced in (Jaksch et al., 2010) is a popular method to perform regret minimization in unknown discrete Markov Decision Processes under the average-reward criterion. Despite its nice and generic theoretical regret guarantees , this algorithm and its variants have remained until now mostly theoretical as numerical experiments in simple environments exhibit long burn-in phases before the learning takes place. In pursuit of practical efficiency, we present UCRL3, following the lines of UCRL2, but with two key modifications: First, it uses state-of-the-art time-uniform concentration inequalities to compute confidence sets on the reward and (component-wise) transition distributions for each state-action pair. Furthermore , to tighten exploration, it uses an adap-tive computation of the support of each transition distribution, which in turn enables us to revisit the extended value iteration procedure of UCRL2 to optimize over distributions with reduced support by disregarding low probability transitions, while still ensuring near-optimism. We demonstrate , through numerical experiments in standard environments, that reducing exploration this way yields a substantial numerical improvement compared to UCRL2 and its variants. On the theoretical side, these key modifications enable us to derive a regret bound for UCRL3 improving on UCRL2, that for the first time makes appear notions of local diameter and local effective support, thanks to variance-aware concentration bounds.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-03000664/file/ICML2020_UCRL3_FinalVersion.pdf BibTex
titre
Restarted Bayesian Online Change-point Detector achieves Optimal Detection Delay
auteur
Réda Alami, Odalric-Ambrym Maillard, Raphael Féraud
article
International Conference on Machine Learning, Jul 2020, Wien, Austria
resume
In this paper, we consider the problem of sequential change-point detection where both the changepoints and the distributions before and after the change are assumed to be unknown. For this problem of primary importance in statistical and sequential learning theory, we derive a variant of the Bayesian Online Change Point Detector proposed by (Fearnhead & Liu, 2007) which is easier to analyze than the original version while keeping its powerful message-passing algorithm. We provide a non-asymptotic analysis of the false-alarm rate and the detection delay that matches the existing lower-bound. We further provide the first explicit high-probability control of the detection delay for such approach. Experiments on synthetic and realworld data show that this proposal outperforms the state-of-art change-point detection strategy, namely the Improved Generalized Likelihood Ratio (Improved GLR) while compares favorably with the original Bayesian Online Change Point Detection strategy.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-03021712/file/paper.pdf BibTex

Preprints, Working Papers, ...

titre
Efficient Change-Point Detection for Tackling Piecewise-Stationary Bandits
auteur
Lilian Besson, Emilie Kaufmann, Odalric-Ambrym Maillard, Julien Seznec
article
2020
resume
We introduce GLR-klUCB, a novel algorithm for the piecewise iid non-stationary bandit problem with bounded rewards. This algorithm combines an efficient bandit algorithm, kl-UCB, with an efficient, parameter-free, changepoint detector, the Bernoulli Generalized Likelihood Ratio Test, for which we provide new theoretical guarantees of independent interest. Unlike previous non-stationary bandit algorithms using a change-point detector, GLR-klUCB does not need to be calibrated based on prior knowledge on the arms' means. We prove that this algorithm can attain a $O(\sqrt{TA \Upsilon_T\log(T)})$ regret in $T$ rounds on some ``easy'' instances, where A is the number of arms and $\Upsilon_T$ the number of change-points, without prior knowledge of $\Upsilon_T$. In contrast with recently proposed algorithms that are agnostic to $\Upsilon_T$, we perform a numerical study showing that GLR-klUCB is also very efficient in practice, beyond easy instances.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-02006471/file/BKMS20.pdf BibTex
titre
Optimal Strategies for Graph-Structured Bandits
auteur
Hassan Saber, Pierre Ménard, Odalric-Ambrym Maillard
article
2020
resume
We study a structured variant of the multi-armed bandit problem specified by a set of Bernoulli distributions $ \nu \!= \!(\nu_{a,b})_{a \in \mathcal{A}, b \in \mathcal{B}}$ with means $(\mu_{a,b})_{a \in \mathcal{A}, b \in \mathcal{B}}\!\in\![0,1]^{\mathcal{A}\times\mathcal{B}}$ and by a given weight matrix $\omega\!=\! (\omega_{b,b'})_{b,b' \in \mathcal{B}}$, where $ \mathcal{A}$ is a finite set of arms and $ \mathcal{B} $ is a finite set of users. The weight matrix $\omega$ is such that for any two users $b,b'\!\in\!\mathcal{B}, \text{max}_{a\in\mathcal{A}}|\mu_{a,b} \!-\! \mu_{a,b'}| \!\leq\! \omega_{b,b'} $. This formulation is flexible enough to capture various situations, from highly-structured scenarios ($\omega\!\in\!\{0,1\}^{\mathcal{B}\times\mathcal{B}}$) to fully unstructured setups ($\omega\!\equiv\! 1$). We consider two scenarios depending on whether the learner chooses only the actions to sample rewards from or both users and actions. We first derive problem-dependent lower bounds on the regret for this generic graph-structure that involves a structure dependent linear programming problem. Second, we adapt to this setting the Indexed Minimum Empirical Divergence (IMED) algorithm introduced by Honda and Takemura (2015), and introduce the IMED-GS$^\star$ algorithm. Interestingly, IMED-GS$^\star$ does not require computing the solution of the linear programming problem more than about $\log(T)$ times after $T$ steps, while being provably asymptotically optimal. Also, unlike existing bandit strategies designed for other popular structures, IMED-GS$^\star$ does not resort to an explicit forced exploration scheme and only makes use of local counts of empirical events. We finally provide numerical illustration of our results that confirm the performance of IMED-GS$^\star$.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02891139/file/Optimal%20Strategies%20for%20Graph-Structured%20Bandits.pdf BibTex
titre
Forced-exploration free Strategies for Unimodal Bandits
auteur
Hassan Saber, Pierre Ménard, Odalric-Ambrym Maillard
article
2020
resume
We consider a multi-armed bandit problem specified by a set of Gaussian or Bernoulli distributions endowed with a unimodal structure. Although this problem has been addressed in the literature (Combes and Proutiere, 2014), the state-of-the-art algorithms for such structure make appear a forced-exploration mechanism. We introduce IMED-UB, the first forced-exploration free strategy that exploits the unimodal-structure, by adapting to this setting the Indexed Minimum Empirical Divergence (IMED) strategy introduced by Honda and Takemura (2015). This strategy is proven optimal. We then derive KLUCB-UB, a KLUCB version of IMED-UB, which is also proven optimal. Owing to our proof technique, we are further able to provide a concise finite-time analysis of both strategies in an unified way. Numerical experiments show that both IMED-UB and KLUCB-UB perform similarly in practice and outperform the state-of-the-art algorithms.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02883907/file/Forced-exploration%20free%20Strategies%20for%20Unimodal%20Bandits.pdf BibTex

2019

Conference papers

titre
Budgeted Reinforcement Learning in Continuous State Space
auteur
Nicolas Carrara, Edouard Leurent, Romain Laroche, Tanguy Urvoy, Odalric-Ambrym Maillard, Olivier Pietquin
article
Conference on Neural Information Processing Systems, Dec 2019, Vancouver, Canada
resume
A Budgeted Markov Decision Process (BMDP) is an extension of a Markov Decision Process to critical applications requiring safety constraints. It relies on a notion of risk implemented in the shape of a cost signal constrained to lie below an-adjustable-threshold. So far, BMDPs could only be solved in the case of finite state spaces with known dynamics. This work extends the state-of-the-art to continuous spaces environments and unknown dynamics. We show that the solution to a BMDP is a fixed point of a novel Budgeted Bellman Optimality operator. This observation allows us to introduce natural extensions of Deep Reinforcement Learning algorithms to address large-scale BMDPs. We validate our approach on two simulated applications: spoken dialogue and autonomous driving.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02375727/file/9128-budgeted-reinforcement-learning-in-continuous-state-space.pdf BibTex
titre
Regret Bounds for Learning State Representations in Reinforcement Learning
auteur
Ronald Ortner, Matteo Pirotta, Ronan Fruit, Alessandro Lazaric, Odalric-Ambrym Maillard
article
Conference on Neural Information Processing Systems, Dec 2019, Vancouver, Canada
resume
We consider the problem of online reinforcement learning when several state representations (mapping histories to a discrete state space) are available to the learning agent. At least one of these representations is assumed to induce a Markov decision process (MDP), and the performance of the agent is measured in terms of cumulative regret against the optimal policy giving the highest average reward in this MDP representation. We propose an algorithm (UCB-MS) with O(√ T) regret in any communicating MDP. The regret bound shows that UCB-MS automatically adapts to the Markov model and improves over the currently known best bound of order O(T 2/3).
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02375715/file/9435-regret-bounds-for-learning-state-representations-in-reinforcement-learning.pdf BibTex
titre
Learning Multiple Markov Chains via Adaptive Allocation
auteur
Mohammad Talebi, Odalric-Ambrym Maillard
article
Advances in Neural Information Processing Systems 32 (NIPS 2019), Dec 2019, Vancouver, Canada
resume
We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains sequentially to observe their states. The goal of the learner is to sequentially select various chains to learn transition matrices uniformly well with respect to some loss function. We introduce a notion of loss that naturally extends the squared loss for learning distributions to the case of Markov chains, and further characterize the notion of being uniformly good in all problem instances. We present a novel learning algorithm that efficiently balances exploration and exploitation intrinsic to this problem, without any prior knowledge of the chains. We provide finite-sample PAC-type guarantees on the performance of the algorithm. Further, we show that our algorithm asymptotically attains an optimal loss.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02387345/file/AL_MC_HAL.pdf BibTex
titre
Model-Based Reinforcement Learning Exploiting State-Action Equivalence
auteur
Mahsa Asadi, Mohammad Talebi, Hippolyte Bourel, Odalric-Ambrym Maillard
article
ACML 2019, Proceedings of Machine Learning Research, Nov 2019, Nagoya, Japan. pp.204 - 219
resume
Leveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity between transition probabilities of various state-action pairs of an MDP, which naturally defines an equivalence structure in the state-action space. We present equivalence-aware confidence sets for the case where the learner knows the underlying structure in advance. These sets are provably smaller than their corresponding equivalence-oblivious counterparts. In the more challenging case of an unknown equivalence structure, we present an algorithm called ApproxEquivalence that seeks to find an (approximate) equivalence structure, and define confidence sets using the approximate equivalence. To illustrate the efficacy of the presented confidence sets, we present C-UCRL, as a natural modification of UCRL2 for RL in undiscounted MDPs. In the case of a known equivalence structure, we show that C-UCRL improves over UCRL2 in terms of regret by a factor of SA/C, in any communicating MDP with S states, A actions, and C classes, which corresponds to a massive improvement when C SA. To the best of our knowledge, this is the first work providing regret bounds for RL when an equivalence structure in the MDP is efficiently exploited. In the case of an unknown equivalence structure, we show through numerical experiments that C-UCRL combined with ApproxEquivalence outperforms UCRL2 in ergodic MDPs.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02378887/file/2019%20-%20Model-based%20Reinforcement%20Learning%20Exploiting%20State-Action%20Equivalence.pdf BibTex
titre
Practical Open-Loop Optimistic Planning
auteur
Edouard Leurent, Odalric-Ambrym Maillard
article
European Conference on Machine Learning, Sep 2019, Würzburg, Germany
resume
We consider the problem of online planning in a Markov Decision Process when given only access to a generative model, restricted to open-loop policies-i.e. sequences of actions-and under budget constraint. In this setting, the Open-Loop Optimistic Planning (OLOP) algorithm enjoys good theoretical guarantees but is overly conservative in practice, as we show in numerical experiments. We propose a modified version of the algorithm with tighter upper-confidence bounds, KL-OLOP, that leads to better practical performances while retaining the sample complexity bound. Finally, we propose an efficient implementation that significantly improves the time complexity of both algorithms.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02375697/file/407.pdf BibTex
titre
Sequential change-point detection: Laplace concentration of scan statistics and non-asymptotic delay bounds
auteur
Odalric-Ambrym Maillard
article
Algorithmic Learning Theory, 2019, Chicago, United States. pp.1 - 23
resume
We consider change-point detection in a fully sequential setup, when observations are received one by one and one must raise an alarm as early as possible after any change. We assume that both the change points and the distributions before and after the change are unknown. We consider the class of piecewise-constant mean processes with sub-Gaussian noise, and we target a detection strategy that is uniformly good on this class (this constrains the false alarm rate and detection delay). We introduce a novel tuning of the GLR test that takes here a simple form involving scan statistics, based on a novel sharp concentration inequality using an extension of the Laplace method for scan-statistics that holds doubly-uniformly in time. This also considerably simplifies the implementation of the test and analysis. We provide (perhaps surprisingly) the first fully non-asymptotic analysis of the detection delay of this test that matches the known existing asymptotic orders, with fully explicit numerical constants. Then, we extend this analysis to allow some changes that are not-detectable by any uniformly-good strategy (the number of observations before and after the change are too small for it to be detected by any such algorithm), and provide the first robust, finite-time analysis of the detection delay.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02351665/file/maillard19.pdf BibTex

Habilitation à diriger des recherches

titre
Mathematics of Statistical Sequential Decision Making
auteur
Odalric-Ambrym Maillard
article
Mathematics [math]. Université de Lille, Sciences et Technologies, 2019
resume
In this document, we give an overview of recent contributions to the mathematics of statistical sequential learning. Unlike research articles that start from a motivating example and provide little room to the mathematical tools in the main body of the article, we here give primary focus to these tools, in order to stress their potential as well as their role in the development of improved algorithms and proof techniques in the field. We revisit in particular properties of the log Laplace transform of a random variable, the handling of random stopping time in concentration of measure of empirical distributions, and we highlight the fundamental role of the “change of measure” argument both in the construction of performance lower-bounds as well as near-optimal strategies. We then give focus to obtaining finite-time error guarantees on the parameter estimation in parametric models before highlighting the strength of Legendre-Fenchel duality in the design of risk-averse and robust strategies. Finally, we turn the setting of Markov decision processes where we present some key insights for the development of the next generation of decision strategies. We end this manuscript by providing a more focused presentation of three key contributions in bandit theory, stochastic automata, and aggregation of experts
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/tel-02162189/file/HDR2019LIL01.pdf BibTex
titre
Mathematics of Statistical Sequential Decision Making
auteur
Odalric-Ambrym Maillard
article
Statistics [math.ST]. Université de Lille Nord de France, 2019
resume
In this document, we give an overview of recent contributions to the mathematics of statistical sequential learning. Unlike research articles that start from a motivating example and provide little room to the mathematical tools in the main body of the article, we here give primary focus to these tools, in order to stress their potential as well as their role in the development of improved algorithms and proof techniques in the field. We revisit in particular properties of the log Laplace transform of a random variable, the handling of random stopping time in concentration of measure of empirical distributions, and we highlight the fundamental role of the “change of measure” argument both in the construction of performance lower-bounds as well as near-optimal strategies. We then give focus to obtaining finite-time error guarantees on the parameter estimation in parametric models before highlighting the strength of Legendre-Fenchel duality in the design of risk-averse and robust strategies. Finally, we turn the setting of Markov decision processes where we present some key insights for the development of the next generation of decision strategies. We end this manuscript by providing a more focused presentation of three key contributions in bandit theory, stochastic automata, and aggregation of experts.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/tel-02077035/file/HDR_cr.pdf BibTex

Preprints, Working Papers, ...

titre
Active Roll-outs in MDP with Irreversible Dynamics
auteur
Odalric-Ambrym Maillard, Timothy Mann, Ronald Ortner, Shie Mannor
article
2019
resume
In Reinforcement Learning (RL), regret guarantees scaling with the square root of the time horizon have been shown to hold only for communicating Markov decision processes (MDPs) where any two states are connected. This essentially means that an algorithm can eventually recover from any mistake. However, real-world tasks usually include situations where taking a single "bad" action can permanently trap a learner in a suboptimal region of the state-space. Since it is provably impossible to achieve sub-linear regret in general multi-chain MDPs, we assume a weak mechanism that allows the learner to request additional information. Our main contribution is to address: (i) how much external information is needed, (ii) how and when to use it, and (iii) how much regret is incurred. We design an algorithm that minimizes requests for external information in the form of rollouts of a policy specified by the learner by actively requesting it only when needed. The algorithm provably achieves O(√ T) active regret after T steps in a large class of multi-chain MDPs, by only requesting O(log(T)) rollout transitions. The superiority of our algorithm to standard algorithms such as R-Max and UCRL is demonstrated in experiments on some illustrative grid-world examples. (a) (b) (c) Figure 1: Example of (a) a communicating MDP, (b) a unichain MDP with a single recurrent class, and (c) a multi-chain MDP with two recurrent classes. The circles represent states while the labeled edges represent transitions due to executing actions {a, b, c}.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-02177808/file/maillard16a.pdf BibTex
titre
Approximate Robust Control of Uncertain Dynamical Systems
auteur
Edouard Leurent, Yann Blanco, Denis Efimov, Odalric-Ambrym Maillard
article
2019
resume
This work studies the design of safe control policies for large-scale non-linear systems operating in uncertain environments. In such a case, the robust control framework is a principled approach to safety that aims to maximize the worst-case performance of a system. However, the resulting optimization problem is generally intractable for non-linear systems with continuous states. To overcome this issue, we introduce two tractable methods that are based either on sampling or on a conservative approximation of the robust objective. The proposed approaches are applied to the problem of autonomous driving.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01931744/file/robust_control.pdf BibTex

2018

Journal articles

titre
Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPs
auteur
Mohammad Talebi, Odalric-Ambrym Maillard
article
Journal of Machine Learning Research, Microtome Publishing, In press, pp.1-36
resume
The problem of reinforcement learning in an unknown and discrete Markov Decision Process (MDP) under the average-reward criterion is considered, when the learner interacts with the system in a single stream of observations, starting from an initial state without any reset. We revisit the minimax lower bound for that problem by making appear the local variance of the bias function in place of the diameter of the MDP. Furthermore, we provide a novel analysis of the KL-Ucrl algorithm establishing a high-probability regret bound scaling as O S s,a V s,a T for this algorithm for ergodic MDPs, where S denotes the number of states and where V s,a is the variance of the bias function with respect to the next-state distribution following action a in state s. The resulting bound improves upon the best previously known regret bound O(DS √ AT) for that algorithm, where A and D respectively denote the maximum number of actions (per state) and the diameter of MDP. We finally compare the leading terms of the two bounds in some benchmark MDPs indicating that the derived bound can provide an order of magnitude improvement in some cases. Our analysis leverages novel variations of the transportation lemma combined with Kullback-Leibler concentration inequalities, that we believe to be of independent interest.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01737142/file/ALT18_CameraReady.pdf BibTex
titre
Streaming kernel regression with provably adaptive mean, variance, and regularization
auteur
Audrey Durand, Odalric-Ambrym Maillard, Joelle Pineau
article
Journal of Machine Learning Research, Microtome Publishing, 2018, 1, pp.1 - 48
resume
We consider the problem of streaming kernel regression, when the observations arrive sequentially and the goal is to recover the underlying mean function, assumed to belong to an RKHS. The variance of the noise is not assumed to be known. In this context, we tackle the problem of tuning the regularization parameter adaptively at each time step, while maintaining tight confidence bounds estimates on the value of the mean function at each point. To this end, we first generalize existing results for finite-dimensional linear regression with fixed regularization and known variance to the kernel setup with a regularization parameter allowed to be a measurable function of past observations. Then, using appropriate self-normalized inequalities we build upper and lower bound estimates for the variance, leading to Bersntein-like concentration bounds. The later is used in order to define the adaptive regularization. The bounds resulting from our technique are valid uniformly over all observation points and all time steps, and are compared against the literature with numerical experiments. Finally, the potential of these tools is illustrated by an application to kernelized bandits, where we revisit the Kernel UCB and Kernel Thompson Sampling procedures, and show the benefits of the novel adaptive kernel tuning strategy.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01927007/file/1708.00768.pdf BibTex
titre
Boundary Crossing Probabilities for General Exponential Families
auteur
Odalric-Ambrym Maillard
article
Mathematical Methods of Statistics, Allerton Press, Springer (link), 2018, 27, pp.1-31. ⟨10.3103/S1066530718010015⟩
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01737150/file/BoundaryCrossingForExponentialFamilies_RevisedVersion.pdf BibTex

Poster communications

titre
Memory Bandits: Towards the Switching Bandit Problem Best Resolution
auteur
Réda Alami, Odalric-Ambrym Maillard, Raphaël Féraud
article
MLSS 2018 - Machine Learning Summer School, Aug 2018, Madrid, Spain
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01879251/file/conference_poster_4.pdf BibTex

Preprints, Working Papers, ...

titre
Upper Confidence Reinforcement Learning exploiting state-action equivalence
auteur
Odalric-Ambrym Maillard, Mahsa Asadi
article
2018
resume
Leveraging an equivalence property on the set of states of state-action pairs in an Markov Decision Process (MDP) has been suggested by many authors. We take the study of equivalence classes to the reinforcement learning (RL) setup, when transition distributions are no longer assumed to be known, in a discrete MDP with average reward criterion and no reset. We study powerful similarities between state-action pairs related to optimal transport. We first analyze a variant of the UCRL2 algorithm called C-UCRL2, which highlights the clear benefit of leveraging this equivalence structure when it is known ahead of time: the regret bound scales as ~O(D√KCT) where C is the number of classes of equivalent state-action pairs and K bounds the size of the support of the transitions. A non trivial question is whether this benefit can still be observed when the structure is unknown and must be learned while minimizing the regret. We propose a sound clustering technique that provably learn the unknown classes, but show that its natural combination with UCRL2 empirically fails. Our findings suggests this is due to the ad-hoc criterion for stopping the episodes in UCRL2. We replace it with hypothesis testing, which in turns considerably improves all strategies. It is then empirically validated that learning the structure can be beneficial in a full-blown RL problem.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01945034/file/UCRL_Classes_HAL.pdf BibTex

2017

Journal articles

titre
The Non-stationary Stochastic Multi-armed Bandit Problem
auteur
Robin Allesiardo, Raphaël Féraud, Odalric-Ambrym Maillard
article
International Journal of Data Science and Analytics, Springer Verlag, 2017, 3 (4), pp.267-283. ⟨10.1007/s41060-017-0050-5⟩
resume
We consider a variant of the stochastic multi-armed bandit with K arms where the rewards are not assumed to be identically distributed, but are generated by a non-stationary stochastic process. We first study the unique best arm setting when there exists one unique best arm. Second, we study the general switching best arm setting when a best arm switches at some unknown steps. For both settings, we target problem-dependent bounds, instead of the more conservative problem-free bounds. We consider two classical problems: (1) identify a best arm with high probability (best arm identification), for which the performance measure by the sample complexity (number of samples before finding a near-optimal arm). To this end, we naturally extend the definition of sample complexity so that it makes sense in the switching best arm setting, which may be of independent interest. (2) Achieve the smallest cumulative regret (regret minimization) where the regret is measured with respect to the strategy pulling an arm with the best instantaneous mean at each step.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01575000/file/NonStaStoMAB_final.pdf BibTex

Conference papers

titre
Efficient tracking of a growing number of experts
auteur
Jaouad Mourtada, Odalric-Ambrym Maillard
article
Algorithmic Learning Theory, Oct 2017, Tokyo, Japan. pp.1 - 23
resume
We consider a variation on the problem of prediction with expert advice, where new forecasters that were unknown until then may appear at each round. As often in prediction with expert advice, designing an algorithm that achieves near-optimal regret guarantees is straightforward, using ag-gregation of experts. However, when the comparison class is sufficiently rich, for instance when the best expert and the set of experts itself changes over time, such strategies naively require to maintain a prohibitive number of weights (typically exponential with the time horizon). By contrast, designing strategies that both achieve a near-optimal regret and maintain a reasonable number of weights is highly non-trivial. We consider three increasingly challenging objectives (simple regret, shifting regret and sparse shifting regret) that extend existing notions defined for a fixed expert ensemble; in each case, we design strategies that achieve tight regret bounds, adaptive to the parameters of the comparison class, while being computationally inexpensive. Moreover, our algorithms are anytime , agnostic to the number of incoming experts and completely parameter-free. Such remarkable results are made possible thanks to two simple but highly effective recipes: first the " abstention trick " that comes from the specialist framework and enables to handle the least challenging notions of regret, but is limited when addressing more sophisticated objectives. Second, the " muting trick " that we introduce to give more flexibility. We show how to combine these two tricks in order to handle the most challenging class of comparison strategies.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01615424/file/paper-growing.pdf BibTex
titre
Boundary Crossing for General Exponential Families
auteur
Odalric-Ambrym Maillard
article
Algorithmic Learning Theory, Oct 2017, Kyoto, Japan. pp.1 - 34
resume
We consider parametric exponential families of dimension K on the real line. We study a variant of boundary crossing probabilities coming from the multi-armed bandit literature, in the case when the real-valued distributions form an exponential family of dimension K. Formally, our result is a concentration inequality that bounds the probability that B ψ (θ n , θ) f (t/n)/n, where θ is the parameter of an unknown target distribution, θ n is the empirical parameter estimate built from n observations, ψ is the log-partition function of the exponential family and B ψ is the corresponding Bregman divergence. From the perspective of stochastic multi-armed bandits, we pay special attention to the case when the boundary function f is logarithmic, as it enables to analyze the regret of the state-of-the-art KL-ucb and KL-ucb+ strategies, whose analysis was left open in such generality. Indeed, previous results only hold for the case when K = 1, while we provide results for arbitrary finite dimension K, thus considerably extending the existing results. Perhaps surprisingly, we highlight that the proof techniques to achieve these strong results already existed three decades ago in the work of T.L. Lai, and were apparently forgotten in the bandit community. We provide a modern rewriting of these beautiful techniques that we believe are useful beyond the application to stochastic multi-armed bandits.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01615427/file/15.pdf BibTex
titre
Spectral Learning from a Single Trajectory under Finite-State Policies
auteur
Borja Balle, Odalric-Ambrym Maillard
article
International conference on Machine Learning, Jul 2017, Sidney, France
resume
We present spectral methods of moments for learning sequential models from a single trajec-tory, in stark contrast with the classical literature that assumes the availability of multiple i.i.d. trajectories. Our approach leverages an efficient SVD-based learning algorithm for weighted au-tomata and provides the first rigorous analysis for learning many important models using dependent data. We state and analyze the algorithm under three increasingly difficult scenarios: proba-bilistic automata, stochastic weighted automata, and reactive predictive state representations controlled by a finite-state policy. Our proofs include novel tools for studying mixing properties of stochastic weighted automata.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01590940/file/cr_ICML_HankelMatrices2.pdf BibTex

Lectures

titre
Basic Concentration Properties of Real-Valued Distributions
auteur
Odalric-Ambrym Maillard
article
Doctoral. France. 2017
resume
In this note we introduce and discuss a few concentration tools for the study of concentration inequalities on the real line. After recalling versions of the Chernoff method, we move to concentration inequalities for predictable processes. We especially focus on bounds that enable to handle the sum of real-valued random variables, where the number of summands is itself a random stopping time, and target fully explicit and empirical bounds. We then discuss some important other tools, such as the Laplace method and the transportation lemma.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/cel-01632228/file/Lecture1-IntroductionToBasicConcentration.pdf BibTex

2016

Conference papers

titre
Pliable rejection sampling
auteur
Akram Erraqabi, Michal Valko, Alexandra Carpentier, Odalric-Ambrym Maillard
article
International Conference on Machine Learning, Jun 2016, New York City, United States
resume
Rejection sampling is a technique for sampling from difficult distributions. However, its use is limited due to a high rejection rate. Common adaptive rejection sampling methods either work only for very specific distributions or without performance guarantees. In this paper, we present pliable rejection sampling (PRS), a new approach to rejection sampling, where we learn the sampling proposal using a kernel estimator. Since our method builds on rejection sampling, the samples obtained are with high probability i.i.d. and distributed according to f. Moreover, PRS comes with a guarantee on the number of accepted samples.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-01322168/file/erraqabi2016pliable.pdf BibTex

Preprints, Working Papers, ...

titre
Low-rank Bandits with Latent Mixtures
auteur
Aditya Gopalan, Odalric-Ambrym Maillard, Mohammadi Zaki
article
2016
resume
We study the task of maximizing rewards from recommending items (actions) to users sequentially interacting with a recommender system. Users are modeled as latent mixtures of C many representative user classes, where each class specifies a mean reward profile across actions. Both the user features (mixture distribution over classes) and the item features (mean reward vector per class) are unknown a priori. The user identity is the only contextual information available to the learner while interacting. This induces a low-rank structure on the matrix of expected rewards r a,b from recommending item a to user b. The problem reduces to the well-known linear bandit when either user-or item-side features are perfectly known. In the setting where each user, with its stochastically sampled taste profile, interacts only for a small number of sessions, we develop a bandit algorithm for the two-sided uncertainty. It combines the Robust Tensor Power Method of Anandkumar et al. (2014b) with the OFUL linear bandit algorithm of Abbasi-Yadkori et al. (2011). We provide the first rigorous regret analysis of this combination, showing that its regret after T user interactions is˜O(C √ BT), with B the number of users. An ingredient towards this result is a novel robustness property of OFUL, of independent interest.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01400318/file/1609.01508.pdf BibTex
titre
Random Shuffling and Resets for the Non-stationary Stochastic Bandit Problem
auteur
Robin Allesiardo, Raphaël Féraud, Odalric-Ambrym Maillard
article
2016
resume
We consider a non-stationary formulation of the stochastic multi-armed bandit where the rewards are no longer assumed to be identically distributed. For the best-arm identification task, we introduce a version of SUCCESSIVE ELIMINATION based on random shuffling of the K arms. We prove that under a novel and mild assumption on the mean gap ∆, this simple but powerful modification achieves the same guarantees in term of sample complexity and cumulative regret than its original version, but in a much wider class of problems, as it is not anymore constrained to stationary distributions. We also show that the original SUCCESSIVE ELIMINATION fails to have controlled regret in this more general scenario, thus showing the benefit of shuffling. We then remove our mild assumption and adapt the algorithm to the best-arm identification task with switching arms. We adapt the definition of the sample complexity for that case and prove that, against an optimal policy with N − 1 switches of the optimal arm, this new algorithm achieves an expected sample complexity of O(∆^{−2} sqrt(N Kdelta^{−1} log(K/delta)), where δ is the probability of failure of the algorithm, and an expected cumulative regret of O(∆^{−1} sqrt(N T K log(T K))) after T time steps.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01400320/file/1609.02139v1.pdf BibTex
titre
Self-normalization techniques for streaming confident regression
auteur
Odalric-Ambrym Maillard
article
2016
resume
We consider, in a generic streaming regression setting, the problem of building a confidence interval (and distribution) on the next observation based on past observed data. The observations given to the learner are of the form (x, y) with y = f (x) + ξ, where x can have arbitrary dependency on the past observations, f is unknown and the noise ξ is sub-Gaussian conditionally on the past observations. Further, the observations are assumed to come from some external filtering process making the number of observations itself a random stopping time. In this challenging scenario that captures a large class of processes with non-anticipative dependencies, we study the ordinary, ridge, and kernel least-squares estimates and provide confidence intervals based on self-normalized vector-valued martingale techniques, applied to the estimation of the mean and of the variance. We then discuss how these adaptive confidence intervals can be used in order to detect a possible model mismatch as well as to estimate the future (self-information, quadratic, or transportation) loss of the learner at a next step.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01349727/file/HalSubmittedV2.pdf BibTex

2015

Journal articles

titre
Concentration inequalities for sampling without replacement
auteur
Rémi Bardenet, Odalric-Ambrym Maillard
article
Bernoulli, Bernoulli Society for Mathematical Statistics and Probability, 2015, 21 (3), pp.1361-1385. ⟨10.3150/14-BEJ605⟩
resume
Concentration inequalities quantify the deviation of a random variable from a fixed value. In spite of numerous applications, such as opinion surveys or ecological counting procedures , few concentration results are known for the setting of sampling without replacement from a finite population. Until now, the best general concentration inequality has been a Hoeffding inequality due to ?. In this paper, we first improve on the fundamental result of ?, and further extend it to obtain a Bernstein concentration bound for sampling without replacement. We then derive an empirical version of our bound that does not require the variance to be known to the user.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01216652/file/submittedArxiv.pdf BibTex

Preprints, Working Papers, ...

titre
A note on replacing uniform subsampling by random projections in MCMC for linear regression of tall datasets
auteur
Rémi Bardenet, Odalric-Ambrym Maillard
article
2015
resume
New Markov chain Monte Carlo (MCMC) methods have been proposed to tackle inference with tall datasets, i.e., when the number n of data items is intractably large. A large class of these new MCMC methods is based on randomly subsampling the dataset at each MCMC iteration. We investigate whether random projections can replace this random subsampling for linear regression of big streaming data. In the latter setting, random projections have indeed become standard for non-Bayesian treatments. We isolate two issues for MCMC to apply to streaming regression: 1) a resampling issue; MCMC should access the same random projections across iterations to avoid keeping the whole dataset in memory and 2) a budget issue; making individual MCMC acceptance decisions should require o(n) random projections. While the resampling issue can be satisfyingly tackled, current techniques in random projections and MCMC for tall data do not solve the budget issue, and may well end up showing it is not possible.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01248841/file/arxiv.pdf BibTex

2014

Journal articles

titre
Sub-sampling for Multi-armed Bandits
auteur
Akram Baransi, Odalric-Ambrym Maillard, Shie Mannor
article
Proceedings of the European Conference on Machine Learning, 2014, pp.13
resume
The stochastic multi-armed bandit problem is a popular model of the exploration/exploitation trade-off in sequential decision problems. We introduce a novel algorithm that is based on sub-sampling. Despite its simplicity, we show that the algorithm demonstrates excellent empirical performances against state-of-the-art algorithms, including Thompson sampling and KL-UCB. The algorithm is very flexible, it does need to know a set of reward distributions in advance nor the range of the rewards. It is not restricted to Bernoulli distributions and is also invariant under rescaling of the rewards. We provide a detailed experimental study comparing the algorithm to the state of the art, the main intuition that explains the striking results, and conclude with a finite-time regret analysis for this algorithm in the simplified two-arm bandit setting.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-01025651/file/BESA2_corrected.pdf BibTex
titre
Latent Bandits
auteur
Odalric-Ambrym Maillard, Shie Mannor
article
JFPDA, 2014, pp.05
resume
We consider a multi-armed bandit problem where the reward distributions are indexed by two sets -one for arms, one for type- and can be partitioned into a small number of clusters according to the type. First, we consider the setting where all reward distributions are known and all types have the same underlying cluster, the type's identity is, however, unknown. Second, we study the case where types may come from different classes, which is significantly more challenging. Finally, we tackle the case where the reward distributions are completely unknown. In each setting, we introduce specific algorithms and derive non-trivial regret performance. Numerical experiments show that, in the most challenging agnostic case, the proposed algorithm achieves excellent performance in difficult scenarios.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-00990804/file/jfpda_cr.pdf BibTex

Conference papers

titre
Selecting Near-Optimal Approximate State Representations in Reinforcement Learning
auteur
Ronald Ortner, Odalric-Ambrym Maillard, Daniil Ryabko
article
International Conference on Algorithmic Learning Theory (ALT), Oct 2014, Bled, Slovenia. pp.140-154
resume
We consider a reinforcement learning setting where the learner does not have explicit access to the states of the underlying Markov decision process (MDP). Instead, she has access to several models that map histories of past interactions to states. Here we improve over known regret bounds in this setting, and more importantly generalize to the case where the models given to the learner do not contain a true model resulting in an MDP representation but only approximations of it. We also give improved error bounds for state aggregation.
Accès au bibtex
BibTex

Other publications

titre
Latent Bandits.
auteur
Odalric-Ambrym Maillard, Shie Mannor
article
2014
resume
We consider a multi-armed bandit problem where the reward distributions are indexed by two sets --one for arms, one for type-- and can be partitioned into a small number of clusters according to the type. First, we consider the setting where all reward distributions are known and all types have the same underlying cluster, the type's identity is, however, unknown. Second, we study the case where types may come from different classes, which is significantly more challenging. Finally, we tackle the case where the reward distributions are completely unknown. In each setting, we introduce specific algorithms and derive non-trivial regret performance. Numerical experiments show that, in the most challenging agnostic case, the proposed algorithm achieves excellent performance in several difficult scenarios.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-00926281/file/icml_cr_Arxiv.pdf BibTex

2013

Journal articles

titre
Kullback-Leibler Upper Confidence Bounds for Optimal Sequential Allocation
auteur
Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz
article
Annals of Statistics, Institute of Mathematical Statistics, 2013, 41 (3), pp.1516-1541. ⟨10.1214/13-AOS1119⟩
resume
We consider optimal sequential allocation in the context of the so-called stochastic multi-armed bandit model. We describe a generic index policy, in the sense of Gittins (1979), based on upper confidence bounds of the arm payoffs computed using the Kullback-Leibler divergence. We consider two classes of distributions for which instances of this general idea are analyzed: The kl-UCB algorithm is designed for one-parameter exponential families and the empirical KL-UCB algorithm for bounded and finitely supported distributions. Our main contribution is a unified finite-time analysis of the regret of these algorithms that asymptotically matches the lower bounds of Lai and Robbins (1985) and Burnetas and Katehakis (1996), respectively. We also investigate the behavior of these algorithms when used with general bounded rewards, showing in particular that they provide significant improvements over the state-of-the-art.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-00738209/file/klucb.pdf BibTex

Conference papers

titre
Optimal Regret Bounds for Selecting the State Representation in Reinforcement Learning
auteur
Odalric-Ambrym Maillard, Phuong Nguyen, Ronald Ortner, Daniil Ryabko
article
ICML - 30th International Conference on Machine Learning, 2013, Atlanta, USA, United States. pp.543-551
resume
We consider an agent interacting with an environment in a single stream of actions, observations, and rewards, with no reset. This process is not assumed to be a Markov Decision Process (MDP). Rather, the agent has several representations (mapping histories of past interactions to a discrete state space) of the environment with unknown dynamics, only some of which result in an MDP. The goal is to minimize the average regret criterion against an agent who knows an MDP representation giving the highest optimal reward, and acts optimally in it. Recent regret bounds for this setting are of order $O(T^{2/3})$ with an additive term constant yet exponential in some characteristics of the optimal MDP. We propose an algorithm whose regret after $T$ time steps is $O(\sqrt{T})$, with all constants reasonably small. This is optimal in $T$ since $O(\sqrt{T})$ is the optimal regret in the setting of learning in a (single discrete) MDP.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-00778586/file/icml1_iblb_cr-corrected.pdf BibTex
titre
Competing with an Infinite Set of Models in Reinforcement Learning
auteur
Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, Ronald Ortner
article
AISTATS, 2013, Arizona, United States. pp.463-471
resume
We consider a reinforcement learning setting where the learner also has to deal with the problem of finding a suitable state-representation function from a given set of models. This has to be done while interacting with the environment in an online fashion (no resets), and the goal is to have small regret with respect to any Markov model in the set. For this setting, recently the \BLB~algorithm has been proposed, which achieves regret of order $T^{2/3}$, provided that the given set of models is finite. Our first contribution is to extend this result to a countably infinite set of models. Moreover, the \BLB~regret bound suffers from an additive term that can be exponential in the diameter of the MDP involved, since the diameter has to be guessed. The algorithm we propose avoids guessing the diameter, thus improving the regret bound.
Accès au bibtex
BibTex

Other publications

titre
Robust Risk-averse Stochastic Multi-Armed Bandits
auteur
Odalric-Ambrym Maillard
article
2013
resume
We study a variant of the standard stochastic multi-armed bandit problem when one is not interested in the arm with the best mean, but instead in the arm maximizing some coherent risk measure criterion. Further, we are studying the deviations of the regret instead of the less informative expected regret. We provide an algorithm, called RA-UCB to solve this problem, together with a high probability bound on its regret.
Accès au texte intégral et bibtex
https://hal.inria.fr/hal-00821670/file/RiskAwareKLMAB_Arxiv.pdf BibTex

2012

Preprints, Working Papers, ...

titre
Hierarchical Optimistic Region Selection driven by Curiosity
auteur
Odalric-Ambrym Maillard
article
2012
resume
This paper aims to take a step forwards making the term ''intrinsic motivation'' from reinforcement learning theoretically well founded, focusing on curiosity-driven learning. To that end, we consider the setting where, a fixed partition \P of a continuous space \X being given, and a process \nu defined on \X being unknown, we are asked to sequentially decide which cell of the partition to select as well as where to sample \nu in that cell, in order to minimize a loss function that is inspired from previous work on curiosity-driven learning. The loss on each cell consists of one term measuring a simple worst case quadratic sampling error, and a penalty term proportional to the range of the variance in that cell. The corresponding problem formulation extends the setting known as active learning for multi-armed bandits to the case when each arm is a continuous region, and we show how an adaptation of recent algorithms for that problem and of hierarchical optimistic sampling algorithms for optimization can be used in order to solve this problem. The resulting procedure, called Hierarchical Optimistic Region SElection driven by Curiosity (HORSE.C) is provided together with a finite-time regret analysis.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-00740418/file/horsec_nips_supplementary.pdf BibTex
titre
Online allocation and homogeneous partitioning for piecewise constant mean-approximation
auteur
Odalric-Ambrym Maillard, Alexandra Carpentier
article
2012
resume
In the setting of active learning for the multi-armed bandit, where the goal of a learner is to estimate with equal precision the mean of a finite number of arms, recent results show that it is possible to derive strategies based on finite-time confidence bounds that are competitive with the best possible strategy. We here consider an extension of this problem to the case when the arms are the cells of a finite partition P of a continuous sampling space X \subset \Real^d. Our goal is now to build a piecewise constant approximation of a noisy function (where each piece is one region of P and P is fixed beforehand) in order to maintain the local quadratic error of approximation on each cell equally low. Although this extension is not trivial, we show that a simple algorithm based on upper confidence bounds can be proved to be adaptive to the function itself in a near-optimal way, when |P| is chosen to be of minimax-optimal order on the class of \alpha-Hölder functions.
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-00742893/file/nips965supplementary.pdf BibTex

2011

Conference papers

titre
Selecting the State-Representation in Reinforcement Learning
auteur
Odalric-Ambrym Maillard, Rémi Munos, Daniil Ryabko
article
Neural Information Processing Systems, Dec 2011, Granada, Spain
Accès au bibtex
BibTex
titre
A Finite-Time Analysis of Multi-armed Bandits Problems with Kullback-Leibler Divergences
auteur
Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz
article
24th Annual Conference on Learning Theory : COLT'11, Jul 2011, Budapest, Hungary. pp.18
resume
We consider a Kullback-Leibler-based algorithm for the stochastic multi-armed bandit problem in the case of distributions with finite supports (not necessarily known beforehand), whose asymptotic regret matches the lower bound of \cite{Burnetas96}. Our contribution is to provide a finite-time analysis of this algorithm; we get bounds whose main terms are smaller than the ones of previously known algorithms with finite-time analyses (like UCB-type algorithms).
Accès au texte intégral et bibtex
https://hal.inria.fr/inria-00574987/file/66-Maillard-Munos-Stoltz.pdf BibTex

Reports

titre
Adaptive Bandits: Towards the best history-dependent strategy
auteur
Odalric-Ambrym Maillard, Rémi Munos
article
[Technical Report] 2011, pp.14
resume
We consider multi-armed bandit games with possibly adaptive opponents. We introduce models Theta of constraints based on equivalence classes on the common history (information shared by the player and the opponent) which dene two learning scenarios: (1) The opponent is constrained, i.e. he provides rewards that are stochastic functions of equivalence classes dened by some model theta*\in Theta. The regret is measured with respect to (w.r.t.) the best history-dependent strategy. (2) The opponent is arbitrary and we measure the regret w.r.t. the best strategy among all mappings from classes to actions (i.e. the best history-class-based strategy) for the best model in Theta. This allows to model opponents (case 1) or strategies (case 2) which handles nite memory, periodicity, standard stochastic bandits and other situations. When Theta={theta}, i.e. only one model is considered, we derive tractable algorithms achieving a tight regret (at time T) bounded by ~O(sqrt(TAC)), where C is the number of classes of theta. Now, when many models are available, all known algorithms achieving a nice regret O(sqrt(T)) are unfortunately not tractable and scale poorly with the number of models |Theta|. Our contribution here is to provide tractable algorithms with regret bounded by T^{2/3}C^{1/3} log(|Theta|)^{1/2}.
Accès au texte intégral et bibtex
https://hal.inria.fr/inria-00574999/file/AdaptiveBandits_HAL.pdf BibTex

Theses

titre
APPRENTISSAGE SÉQUENTIEL : Bandits, Statistique et Renforcement.
auteur
Odalric-Ambrym Maillard
article
Machine Learning [cs.LG]. Université des Sciences et Technologie de Lille - Lille I, 2011. English
resume
This thesis studies the following topics in Machine Learning: Bandit theory, Statistical learning and Reinforcement learning. The common underlying thread is the non-asymptotic study of various notions of adaptation: to an environment or an opponent in part I about bandit theory, to the structure of a signal in part II about statistical theory, to the structure of states and rewards or to some state-model of the world in part III about reinforcement learning. First we derive a non-asymptotic analysis of a Kullback-Leibler-based algorithm for the stochastic multi-armed bandit that enables to match, in the case of distributions with finite support, the asymptotic distribution-dependent lower bound known for this problem. Now for a multi-armed bandit with a possibly adaptive opponent, we introduce history-based models to catch some weakness of the opponent, and show how one can benefit from such models to design algorithms adaptive to this weakness. Then we contribute to the regression setting and show how the use of random matrices can be beneficial both theoretically and numerically when the considered hypothesis space has a large, possibly infinite, dimension. We also use random matrices in the sparse recovery setting to build sensing operators that allow for recovery when the basis is far from being orthogonal. Finally we combine part I and II to first provide a non-asymptotic analysis of reinforcement learning algorithms such as Bellman-residual minimization and a version of Leastsquares temporal-difference that uses random projections and then, upstream of the Markov Decision Problem setting, discuss the practical problem of choosing a good model of states.
Accès au texte intégral et bibtex
https://tel.archives-ouvertes.fr/tel-00845410/file/thesis_Maillard.pdf BibTex

2010

Conference papers

titre
Finite-Sample Analysis of Bellman Residual Minimization
auteur
Odalric-Ambrym Maillard, Rémi Munos, Alessandro Lazaric, Mohammad Ghavamzadeh
article
Asian Conference on Machine Learning, 2010, Japan
resume
We consider the Bellman residual minimization approach for solving discounted Markov decision problems, where we assume that a generative model of the dynamics and rewards is available. At each policy iteration step, an approximation of the value function for the current policy is obtained by minimizing an empirical Bellman residual defined on a set of n states drawn i.i.d. from a distribution, the immediate rewards, and the next states sampled from the model. Our main result is a generalization bound for the Bellman residual in linear approximation spaces. In particular, we prove that the empirical Bellman residual approaches the true (quadratic) Bellman residual with a rate of order O(1/sqrt((n)). This result implies that minimizing the empirical residual is indeed a sound approach for the minimization of the true Bellman residual which guarantees a good approximation of the value function for each policy. Finally, we derive performance bounds for the resulting approximate policy iteration algorithm in terms of the number of samples n and a measure of how well the function space is able to approximate the sequence of value functions.)
Accès au texte intégral et bibtex
https://hal.archives-ouvertes.fr/hal-00830212/file/brm_acml2010.pdf BibTex

Reports

titre
Linear regression with random projections
auteur
Odalric-Ambrym Maillard, Rémi Munos
article
[Technical Report] 2010, pp.22
resume
We consider ordinary (non penalized) least-squares regression where the regression function is chosen in a randomly generated sub-space GP \subset S of finite dimension P, where S is a function space of infinite dimension, e.g. L2([0, 1]^d). GP is defined as the span of P random features that are linear combinations of the basis functions of S weighted by random Gaussian i.i.d. coefficients. We characterize the so-called kernel space K \subset S of the resulting Gaussian process and derive approximation error bounds of order O(||f||^2_K log(P)/P) for functions f \in K approximated in GP . We apply this result to derive excess risk bounds for the least-squares estimate in various spaces. For illustration, we consider regression using the so-called scrambled wavelets (i.e. random linear combinations of wavelets of L2([0, 1]^d)) and derive an excess risk rate O(||f*||_K(logN)/sqrt(N)) which is arbitrarily close to the minimax optimal rate (up to a logarithmic factor) for target functions f* in K = H^s([0, 1]^d), a Sobolev space of smoothness order s > d/2. We describe an efficient implementation using lazy expansions with numerical complexity ˜O(2dN^3/2 logN+N^2), where d is the dimension of the input data and N is the number of data.
Accès au texte intégral et bibtex
https://hal.inria.fr/inria-00483014/file/jmlr_blsr.pdf BibTex
titre
Brownian Motions and Scrambled Wavelets for Least-Squares Regression
auteur
Odalric-Ambrym Maillard, Rémi Munos
article
[Technical Report] 2010, pp.13
resume
We consider ordinary (non penalized) least-squares regression where the regression function is chosen in a randomly generated sub-space GP \subset S of finite dimension P, where S is a function space of infinite dimension, e.g. L2([0, 1]^d). GP is defined as the span of P random features that are linear combinations of the basis functions of S weighted by random Gaussian i.i.d. coefficients. We characterize the so-called kernel space K \subset S of the resulting Gaussian process and derive approximation error bounds of order O(||f||^2_K log(P)/P) for functions f \in K approximated in GP . We apply this result to derive excess risk bounds for the least-squares estimate in various spaces. For illustration, we consider regression using the so-called scrambled wavelets (i.e. random linear combinations of wavelets of L2([0, 1]^d)) and derive an excess risk rate O(||f*||_K(logN)/sqrt(N)) which is arbitrarily close to the minimax optimal rate (up to a logarithmic factor) for target functions f* in K = H^s([0, 1]^d), a Sobolev space of smoothness order s > d/2. We describe an efficient implementation using lazy expansions with numerical complexity ˜O(2dN^3/2 logN+N^5/2), where d is the dimension of the input data and N is the number of data.
Accès au texte intégral et bibtex
https://hal.inria.fr/inria-00483017/file/blsr.pdf BibTex

2009

Conference papers

titre
Compressed Least-Squares Regression
auteur
Odalric-Ambrym Maillard, Rémi Munos
article
NIPS 2009, Dec 2009, Vancouver, Canada
resume
We consider the problem of learning, from K data, a regression function in a linear space of high dimension N using projections onto a random subspace of lower dimension M. From any algorithm minimizing the (possibly penalized) empirical risk, we provide bounds on the excess risk of the estimate computed in the projected subspace (compressed domain) in terms of the excess risk of the estimate built in the high-dimensional space (initial domain). We show that solving the problem in the compressed domain instead of the initial domain reduces the estimation error at the price of an increased (but controlled) approximation error. We apply the analysis to Least-Squares (LS) regression and discuss the excess risk and numerical complexity of the resulting ``Compressed Least Squares Regression'' (CLSR) in terms of N, K, and M. When we choose M=O(\sqrt{K}), we show that CLSR has an estimation error of order O(\log K / \sqrt{K}).
Accès au texte intégral et bibtex
https://hal.inria.fr/inria-00419210/file/cls_nips.pdf BibTex