![The Contextual Bandit Setting
For t = 1, . . . , T:
1. The world produces some context xt
∈ X
2. The learner chooses an action at
∈ {1, . . . ,K}
3. The world reacts with reward rt
(at
) ∈ [0, 1]
Goal: Learn a good policy for choosing actions given context.
What does learning mean?](https://image.slidesharecdn.com/final-slides-171112210832/85/Personalized-News-Recommendation-Stream-Data-Based-6-320.jpg)
![LinUCB (Disjoint Linear Model)
Assumption: The expected reward for action a is a linear function in the features of the context, i.e.:
1. In each trial t, for each a ∈ At
estimate θa
via regularized linear regression using feature matrix Da
.
E[rt, a
| xt, a
] = xT
t, a
θa
*
2. Choose at
such that,](https://image.slidesharecdn.com/final-slides-171112210832/85/Personalized-News-Recommendation-Stream-Data-Based-12-320.jpg)
![LinUCB (Hybrid Model)
Assumption: The expected reward for action a is the sum of two linear terms, one that is independent of
the action and one that is specific to each action, i.e.:
E[rt, a
| xt, a
] = zT
t, a
β*
+ xT
t, a
θa
*
Algorithm works similar to the previous LinUCB algorithm.](https://image.slidesharecdn.com/final-slides-171112210832/85/Personalized-News-Recommendation-Stream-Data-Based-13-320.jpg)
Personalized News Recommendation (Stream Data Based)
- 1. IR & E Personalized News Article Recommendation (Stream Data Based) Monsoon 17, IIIT Hyderabad
- 2. Keywords ● Contextual Bandit ● Web Service ● Personalization ● Recommender Systems ● Exploration/Exploitation dilemma
- 3. Example of Learning through Exploration Repeatedly: 1. A user comes to Yahoo! (with history of previous visits, IP addresses, data related to his Yahoo! account) 2. Yahoo! chooses information to present (from URLs, Ads, news stories) 3. The user reacts to the presented information (clicks on something, clicks, comes back and clicks again, etc.) Yahoo! wants to interactively choose content and use the observed feedback to improve future content choices.
- 4. Another Example: Clinical Decision Making Repeatedly: 1. A patient comes to a doctor with symptoms, medical history, test results 2. The doctor chooses and suggests a treatment 3. The patient responds to it The doctor wants a policy for choosing targeted treatments for individual patients.
- 5. Current Scenario Which article to feature? Challenges: ● A lot of new users and articles. ● Incorporation of content. ● Changing relevance of articles. Goal: "Quickly" identify relevant news stories on personal level.
- 6. The Contextual Bandit Setting For t = 1, . . . , T: 1. The world produces some context xt ∈ X 2. The learner chooses an action at ∈ {1, . . . ,K} 3. The world reacts with reward rt (at ) ∈ [0, 1] Goal: Learn a good policy for choosing actions given context. What does learning mean?
- 7. The Contextual Bandit Setting (Contd.) What does learning mean? Efficiently competing with a large reference class of possible policies Π = { π : X → {1, ..., K} }
- 8. Some Remarks This is not a supervised learning problem. ● We don’t know the reward of actions not taken, ○ loss function is unknown even at training time. ● Exploration is needed to succeed. ● Simpler than reinforcement learning, ○ We know which action is responsible for each reward.
- 9. Some Remarks (Contd.) This is not a bandit problem. ● In the bandit setting, there is no x, and the goal is to compete with the set of constant actions. ○ Too weak in practice. ● Generalization across x is required to succeed.
- 10. Mapping to our current problem For each time t = 1, 2, 3, … , T, the news page is loaded: 1. Arms or actions are the articles, which can be shown to the user. The environment could be user and article information. 2. If the article a is clicked, rt, a = 1, otherwise 0. 3. Improve new article selection. Goal: Maximize expected Click-through-rate, i.e.,
- 11. Balancing Exploration and Exploitation
- 12. LinUCB (Disjoint Linear Model) Assumption: The expected reward for action a is a linear function in the features of the context, i.e.: 1. In each trial t, for each a ∈ At estimate θa via regularized linear regression using feature matrix Da . E[rt, a | xt, a ] = xT t, a θa * 2. Choose at such that,
- 13. LinUCB (Hybrid Model) Assumption: The expected reward for action a is the sum of two linear terms, one that is independent of the action and one that is specific to each action, i.e.: E[rt, a | xt, a ] = zT t, a β* + xT t, a θa * Algorithm works similar to the previous LinUCB algorithm.
- 14. Evaluation ● Testing on Live Data? ○ TOO EXPENSIVE. ● Then, testing offline? ○ DIFFERENT LOGGING POLICY ● Then, simulator-based approach? ○ BIASED.
- 15. Results ● Training Set: 4.7 million events ● Test Set: 36 million events ● Articles and users clustered into 5 clusters: ○ Two 6-dimensional (one constant) feature vectors
- 16. Questions? Ask in the comment section.