Practical AI for Business: Bandit Algorithms

BANDIT ALGORITHMS
Max Pagels, Machine Learning Specialist
max.pagels@sc5.io, @maxpagels
Understanding the world using reinforcement learning
DATA & AI

ABOUT ME
MAX PAGELS
MSc, BSc, Computer Science (University of Helsinki),
former CS researcher
Currently a machine learning specialist at Helsinki-
based software consultancy SC5, working on applied AI
& cloud-native solutions
Specific areas of interest include probabilistic
supervised learning, deep neural networks and
(algorithmic) complexity theory
Other hats I sometimes wear: full-stack developer,
technical interviewer, architect

TRADITIONAL PROGRAMMING, IN A NUTSHELL
OUTPUTINPUT
ALGORITHM

MACHINE LEARNING, IN A NUTSHELL
PROGRAMINPUT & OUTPUT
LEARNING ALGORITHM

REINFORCEMENT LEARNING, IN A NUTSHELL
ACTION
WORLD
OBSERVATION
AGENT
REWARD

ACTION
WORLD
OBSERVATION
AGENT
REWARD
(Rotate/Up/Down/
Left/Right etc)
Score since
previous action
LEARNING HOW TO PLAY TETRIS

ACTION
WORLD
STATE
AGENT
REWARD
(Rotate/Up/Down/
Left/Right etc)
Score since
previous action
If fully observable
LEARNING HOW TO PLAY TETRIS

IF REWARDS AREN’T DELAYED, IT’S A BANDIT PROBLEM

THE (STOCHASTIC)
MULTI-ARMED
BANDIT PROBLEM
Let’s say you are in a Las Vegas casino, and
there are three vacant slot machines (one-
armed bandits). Pulling the arm of a machine
yields either $0 or $1.
Given these three bandit machines, each with
an unknown payoff strategy, how should we
choose which arm to pull to maximise* the
expected reward over time? 
 
* Equivalently, we want to minimise the expected
regret over time (minimise how much money we
lose).

FORMALLY(ISH)
Problem
We have three one-armed bandits. Each arm
yields a reward of $1 according to some fixed
unknown probability Pa , or a reward of $0 with
probability 1-Pa.
Objective
Find Pa, ∀a ∈ {1,2,3}, play argmax(Pa, ∀a ∈ {1,2,3})

OBSERVATION #1
This is a partial information problem: when we
pull an arm, we don’t get to see the rewards of
the arms we didn’t pull.
(For math aficionados: a bandit problem is an
MDP with a single, terminal, state).

OBSERVATION #2
We need to create some approximation, or best
guess, of how the world works in order to
maximise our reward over time.
We need to create a model.
“All models are wrong but some are useful” —
George Box

OBSERVATION #3
Clearly, we need to try (explore) the arms of all
the machines, to get a sense of which one is
best.

OBSERVATION #4
Though exploration is necessary, we also need
to choose the best arm as much as possible
(exploit) to maximise our reward over time.

THE EXPLORE/EXPLOIT TUG OF WAR
< ACQUIRING KNOWLEDGE MAXIMISING REWARD >

HOW DO WE SOLVE THE BANDIT PROBLEM?

A NAÏVE ALGORITHM
Step 1: For the first N pulls
Evenly pull all the arms, keeping track of the
number of pulls & wins per arm
Step 2: For the remaining M pulls
Choose the arm the the highest expected
reward (ie. mean reward)

A NAÏVE ALGORITHM
Arm 1: {trials: 100, wins: 2}
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17

A NAÏVE ALGORITHM
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17
Explore
Exploit

EPOCH-GREEDY [1]
Arm 1: 2/100 = 0.02
Arm 2: 21/100 = 0.21
Arm 3: 17/100 = 0.17
Explore
Exploit
[1]: http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf

EPSILON-GREEDY/Ε-GREEDY [2]
Choose a value for the hyperparameter ε (e.g. 0.1 = 10%)
Loop forever
With probability ε:
Choose an arm uniformly at random, keep track of trials and wins
With probability 1-ε:
Choose the arm the the highest expected reward
Explore
Exploit
[2]: people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf

OBSERVATIONS
For N = 300 and M = 1000, the Epoch-Greedy algorithm will choose
something other than the best arm 200 times, or about 15% of the time.
At each turn, the ε-Greedy algorithm will choose something other than the
best arm with probability P=(ε/n) × (n-1), n=number of arms. It will always
explore with this probability, no matter how many time steps we have
done.

Ε-GREEDY: SUB-OPTIMAL LINEAR REGRET*
- O(N)Cumulativeregret
4
Timesteps
*Assuming no annealing

THOMPSON SAMPLING ALGORITHM
For each arm, initialise a uniform probability distribution (prior)
Loop forever
Step 1: sample randomly from the probability distribution of each
arm
Step 2: choose the arm with the highest sample value
Step 3: observe reward for the chosen and update the
hyperparameters of its probability distribution (posterior)

EXAMPLE WITH TWO ARMS (BLUE & GREEN)
Assumption: rewards follow a Bernoulli process, which allows us to use a 𝛽-distribution as a
conjugate prior.
FIRST, INITIALISE A UNIFORM RANDOM PROB.
DISTRIBUTION FOR BLUE AND GREEN
0 10 1

FIRST, INITIALISE A UNIFORM RANDOM PROB.
DISTRIBUTION FOR BLUE AND GREEN
0 1

RANDOMLY SAMPLE FROM BOTH ARMS (BLUE GETS THE
HIGHER VALUE IN THIS EXAMPLE)
0 1

PULL BLUE , OBSERVE REWARD (LET’S SAY WE GOT A
REWARD OF 1)
0 1

UPDATE DISTRIBUTION OF BLUE
0 1

REPEAT THE PROCESS: SAMPLE, CHOOSE ARM WITH
HIGHEST SAMPLE VALUE, OBSERVE REWARD, UPDATE.
0 1

0 1
AFTER 100 TIMESTEPS, THE DISTRIBUTIONS MIGHT LOOK
LIKE THIS

0 1
AFTER 1,000 TIMESTEPS, THE DISTRIBUTIONS MIGHT LOOK
LIKE THIS: THE PROCESS HAS CONVERGED.

THOMPSON SAMPLING: LOGARITHMIC REGRET -
O(LOG(N))Cumulativeregret
100
Timesteps

MULTI-ARMED BANDITS ARE
GOOD…
So far, we’ve covered the “vanilla” multi-armed bandit problem. Solving this
problem lets us find the globally best arm to maximise our expected reward
over time.
However, depending on the problem, the globally best arm may not always
be the best arm.

…BUT CONTEXTUAL BANDITS
ARE THE HOLY GRAIL
The the contextual bandit setting, once we pull an arm and receive a reward,
we also get to see a context vector associated with the reward. The
objective is now to maximise the expected reward over time given the
context at each time step.
Solving this problem gives us a higher reward over time in situations where
the globally best arm isn’t always the best arm.
Example algorithm: Thompson Sampling with Linear Payoffs [3]
[3]: https://arxiv.org/abs/1209.3352

…ACTUALLY, ADVERSARIAL
CONTEXTUAL BANDITS ARE THE
HOLY GRAIL
Until now, we’ve assumed that each one-armed bandit gives a $100 reward
according to some unknown probability P. In real-world scenarios, the world
rarely behaves this well. Instead of a simple dice roll, the bandit may pay out
differently depending on the day/hour/amount of money in the machine. It
works as an adversary of sorts.
A bandit algorithm that is capable of dealing with changes in payoff
structures in a reasonable amount of time tend to work better than
stochastic bandits.
Example algorithm: EXP4 (a.k.a the “Monster” algorithm) [4]
[4]: https://arxiv.org/abs/1002.4058

CONTEXTUAL BANDIT
ALGORITHMS ARE 1) STATE-OF-
THE-ART & 2) COMPLEX
• Many algorithms are <5 years old. Some of the most interesting ones
are less than a year old.
• Be prepared to read lots of academic papers whilst implementing the
algorithms.
• Don’t hesitate to contact the authors if there is some detail you don’t
understand (thanks John Langford and Shipra Agrawal!).
• Always remember to simulate and validate bandits using real data.
• Always remember to log actions and rewards of deployed bandits.

PRACTICAL TIPS & CONSIDERATIONS
FOR BANDITS
• If you are implementing your own online-learnable MAB and serving
predictions/receiving rewards over an API, a single process may suffice
• If using Thompson Sampling, sampling 100 arms won’t take more than
a few milliseconds -> possible to implement a single-threaded real-
time learning bandit that can scale to hundreds of requests per
second
• Thousands of arms or thousands of requests per second will require
multiple processes + synchronisation
• For contextual bandits, multiple processes + batching is required
• For some applications, you may need some heuristic for what counts as a
“zero reward”
• Remember to leverage caching on the web, otherwise the UX suffers
• In some cases, you might not want convergence -> tweak algorithm to
always explore with some nonzero probability
• In most applications, the number of arms can vary at any given timestep

REAL-WORLD APPLICATIONS OF BANDITS

BIVARIATE & MULTIVARIATE TESTING
Vanilla MAB

CODEC SELECTION
Image credit: Microsoft/Skype

ONLINE MULTIPLAYER
Image credit: Microsoft/Bungie

PERSONALISED RECOMMENDATIONS
Contextual
MAB

PERSONALISED RECOMMENDATIONS
Multiple-
play MAB

SELFIE OPTIMISATION
Vanilla MAB
(one per
user)
Image: Tinder Tech Blog

… & COUNTLESS OTHER POSSIBILITIES

DIY: CONTEXTUAL BANDIT IMPLEMENTED IN
PYTHON + AZURE FUNCTIONS + API + REDIS
= OPTIMISATION ENGINE

AS-A-SERVICE: AZURE CUSTOM
DECISION SERVICE

IF YOU CAN MODEL IT AS A
GAME, YOU CAN USE A BANDIT
ALGORITHM TO SOLVE IT*.
*Given rewards that aren’t delayed.

THANK YOU
A special thanks to Microsoft for hosting & Practical AI for organising!
QUESTIONS?

Practical AI for Business: Bandit Algorithms

More Related Content

What's hot (20)

Similar to Practical AI for Business: Bandit Algorithms (20)

More from SC5.io (12)

Recently uploaded (20)

Practical AI for Business: Bandit Algorithms