Bandit Algorithms

BANDIT ALGORITHMS
Max Pagels, Data Science Specialist
max.pagels@sc5.io, @maxpagels
13.10.2016
Learning about the world

PROGRAMMING, IN A NUTSHELL
OUTPUTINPUT
ALGORITHM

MACHINE LEARNING, IN A NUTSHELL
PROGRAMINPUT & OUTPUT
LEARNING ALGORITHM

REINFORCEMENT LEARNING, IN A NUTSHELL
ACTION
WORLD
OBSERVATION
AGENT
REWARD

LEARNING HOW TO PLAY TETRIS
ACTION
WORLD
OBSERVATION
AGENT
REWARD
(Rotate/Up/Down/
Left/Right etc)
Score since
previous action

THE (STOCHASTIC)
MULTI-ARMED
BANDIT PROBLEM
Let’s say you are in a Las Vegas casino, and
there are three vacant slot machines (one-
armed bandits). Pulling the arm of a machine
yields either $0 or $1.
Given these three bandit machines, each with
an unknown payoff strategy, how should we
choose which arm to pull to maximise* the
expected reward over time? 
 
* Equivalently, we want to minimise the expected
regret over time (minimise how much money we
lose).

FORMALLY(ISH)
Problem
We have three one-armed bandits. Each arm
yields a reward of $1 according to some fixed
unknown probability Pa , or a reward of $0 with
probability 1-Pa.
Objective
Find Pa, ∀a ∈ {1,2,3}

OBSERVATION #1
This is a partial information problem: when we
pull an arm, we don’t get to see the rewards of
the arms we didn’t pull.
(For math nerds: a bandit problem is an MDP
with a single, terminal, state).

OBSERVATION #2
We need to create some approximation, or best
guess, of how the world works in order to
maximise our reward over time.
We need to create a model.
“All models are wrong but some are useful” —
George Box

OBSERVATION #3
Clearly, we need to try (explore) the arms of all
the machines, to get a sense of which one is
best.

OBSERVATION #4
Though exploration is necessary, we also need
to choose the best arm as much as possible
(exploit) to maximise our reward over time.

THE EXPLORE/EXPLOIT TUG OF WAR

HOW DO WE SOLVE THE BANDIT PROBLEM?

A NAÏVE ALGORITHM
Step 1: For the first N pulls
Evenly pull all the arms, keeping track of the
number of pulls & wins per arm
Step 2: For the remaining M pulls
Choose the arm the the highest expected
reward

A NAÏVE ALGORITHM
reward
Arm 1: {trials: 100, wins: 2}
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17

A NAÏVE ALGORITHM
reward
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17
Explore
Exploit

EPOCH-GREEDY [1]
reward
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17
Explore
Exploit
[1]: http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf

EPSILON-GREEDY/Ε-GREEDY [2]
Choose a value for the hyperparameter ε
Loop forever
With probability ε:
Choose an arm uniformly at random, keep track of trials and wins
With probability 1-ε:
Choose the arm the the highest expected reward
Explore
Exploit
[2]: people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf

OBSERVATIONS
For N = 300 and M = 1000, the Epoch-Greedy algorithm will choose
something other than the best arm 200 times, or about 15% of the time.
At each turn, the ε-Greedy algorithm will choose something other than the
best arm with probability P=(ε/n) × (n-1), n=number of arms. It will always
explore with this probability, no matter how many time steps we have
done.

Ε-GREEDY: SUB-OPTIMAL LINEAR REGRET*
- O(N)Cumulativeregret
4
Timesteps
*Assuming no annealing

THOMPSON SAMPLING ALGORITHM
For each arm, initialise a uniform probability distribution (prior)
Loop forever
Step 1: sample randomly from the probability distribution of each
arm
Step 2: choose the arm with the highest sample value
Step 3: observe reward for the chosen and update the
hyperparameters of its probability distribution (posterior)

EXAMPLE WITH TWO ARMS (BLUE & GREEN)
Assumption: rewards follow a Bernoulli process, which allows us to use a 𝛽-distribution as a
conjugate prior.
FIRST, INITIALISE A UNIFORM RANDOM PROB.
DISTRIBUTION FOR BLUE AND GREEN
0 10 1

FIRST, INITIALISE A UNIFORM RANDOM PROB.
DISTRIBUTION FOR BLUE AND GREEN
0 1

RANDOMLY SAMPLE FROM BOTH ARMS (BLUE GETS THE
HIGHER VALUE IN THIS EXAMPLE)
0 1

PULL BLUE , OBSERVE REWARD (LET’S SAY WE GOT A
REWARD OF 1)
0 1

UPDATE DISTRIBUTION OF BLUE
0 1

REPEAT THE PROCESS: SAMPLE, CHOOSE ARM WITH
HIGHEST SAMPLE VALUE, OBSERVE REWARD, UPDATE.
0 1

0 1
AFTER 100 TIMESTEPS, THE DISTRIBUTIONS MIGHT LOOK
LIKE THIS

0 1
AFTER 1,000 TIMESTEPS, THE DISTRIBUTIONS MIGHT LOOK
LIKE THIS: THE PROCESS HAS CONVERGED.

THOMPSON SAMPLING: LOGARITHMIC REGRET -
O(LOG(N))Cumulativeregret
100
Timesteps

MULTI-ARMED BANDITS ARE
GOOD…
So far, we’ve covered the “vanilla” multi-armed bandit problem. Solving this
problem lets us find the globally best arm to maximise our expected reward
over time.
However, depending on the problem, the globally best arm may not always
be the best arm.

…BUT CONTEXTUAL BANDITS
ARE THE HOLY GRAIL
The the contextual bandit setting, once we pull an arm and receive a reward,
we also get to see a context vector associated with the reward. The
objective is now to maximise the expected reward over time given the
context at each time step.
Solving this problem gives us a higher reward over time in situations where
the globally best arm isn’t always the best arm.
Example algorithm: Thompson Sampling with Linear Payoffs [3]
[3]: https://arxiv.org/abs/1209.3352

…ACTUALLY, ADVERSARIAL
CONTEXTUAL BANDITS ARE THE
HOLY GRAIL
Until now, we’ve assumed that each one-armed bandit gives a $100 reward
according to some unknown probability P. In real-world scenarios, the world
rarely behaves this well. Instead of a simple dice roll, the bandit may pay out
differently depending on the day/hour/amount of money in the machine. It
works as an adversary of sorts.
A bandit algorithm that is capable of dealing with changes in payoff
structures in a reasonable amount of time tend to work better than
stochastic bandits in real-world settings.
Example algorithm: EXP4 (a.k.a the “Monster” algorithm) [4]
[4]: https://arxiv.org/abs/1002.4058

CONTEXTUAL BANDIT
ALGORITHMS ARE 1) STATE-OF-
THE-ART & 2) COMPLEX
• Many algorithms are <5 years old. Some of the most interesting ones
are less than a year old.
• Be prepared to read lots of academic papers and struggle
implementing the algorithms.
• Don’t hesitate to contact the authors if there is some detail you don’t
understand (thanks John Langford and Shipra Agrawal!)
• Always remember to simulate and validate bandits using real data.

REAL-WORLD APPLICATIONS OF BANDITS

BIVARIATE & MULTIVARIATE TESTING
Vanilla MAB

… & COUNTLESS OTHER POSSIBILITIES

TIP: CONTEXTUAL BANDIT IMPLEMENTED IN
PYTHON + LAMBDA + API GATEWAY + REDIS
= OPTIMISATION ENGINE

DEMO: HTTPS://
MWTDS.AZUREWEBSITES.NET/HOME/
APITESTDRIVE

LAUNCHED YESTERDAY (!): AZURE
CUSTOM DECISION SERVICE

IF YOU CAN MODEL IT AS A
GAME, YOU CAN USE A BANDIT
ALGORITHM TO SOLVE IT*.
*Given rewards that aren’t delayed.

Bandit Algorithms

More Related Content

Similar to Bandit Algorithms (20)

More from SC5.io (12)

Recently uploaded (20)

Bandit Algorithms