Tutorial on People Recommendations in Social Networks - ACM RecSys 2013,Hong Kong

Editor's Notes

• #12: SamSuggestions drive more people through this loop, faster.This is the KEY to virality
• #21: Here we consider failure as a user making a decision to transit to a new job. Let p(y) denote the probability density function of such an event.
• #23: This probability density function represents the basic pro- portional hazards model that models the tenure before a transition with associated covariates.
• #24: Unreasonable effectiveness of Big Data.. This chart shows the probability of holding a title across all titles, plotted vs number of months after graduation. Notice the spikes.. They are ~12 month almost perfectly aligned.. Remember the itch that you had when you finished 2 years at your company 
• #26: This probability density function represents the basic pro- portional hazards model that models the tenure before a transition with associated covariates.
• #35: Understand this graph better
• #36: Talent Match: job posting flow: When recruiters post jobs we in real time suggest top candidates fit for the job
• #37: So, Intuitively, it makes sense to suggest users who are job seekers in TalentMatch. But we confirmed our intuition, we ran the numbers, and saw that users with a high job seeking intent (actives and passives) have a much higher rate of reply to career related emails when compared to non-job-seekers (16 times the reply rate). And this is exactly the facet of the utility function of TalentMatch that we are interested in improving. So, what we want to do is incorporate the job seeker intent into the TalentMatch model, and we want to do so without negatively affecting the booking rate and the email rate.
• #38: So, what we want is a controlled perturbation of the ranking output by the talent match model, and this is how we are gonna do it: given the talent match ranking, we run a perturbation function on it that generates another ranking, the perturbed ranking, which optimizes for a metric we’re interested in (in the case of TalentMatch, it’s number of users with high-job seeking intent in the top-12 recommendations). Given the 2 rankings and their distribution of match scores, we can compute the distance between them using a variety of metrics, for example KL divergence or Euclidean distance. This divergence score is what will help us to make sure we are not negatively affecting the quality of the recommendations. Notice how, in the perturbed ranking, item Z was bumped from its original third position, below the cutoff line, to the second position, and so whereas before we had 2 non-seekers above the cutoff, meaning they would be recommended, now we have a non-seeker and an active. Also notice, that the perturbation is minimal. We should feel comfortable bumping item Z to the second position, but not to the first position.There are then 3 functions that we need to define: the perturbation function, the divergence function, and the objective function. The parameters of the perturbation function is what we will be estimating based the performance established by the divergence and objective measures: we want high scores on the objective and low scores on the divergence.
• #39: Here is theinstantiation of those functions for the TalentMatch case. The perturbation function simply applies a small boost to the match score, denoted by the letter “y”, and we allow that boost to be different for active and passive job seekers (as denoted by the alpha and the beta parameters). The divergence function is simply the Euclidean distance between the distribution of scores in the talent match ranking and the distribution of scores in the perturbed ranking. This is simply a measure of how match quality was affected (a divergence score of 0 means that the quality of the matches remained unaffected). The objective is the average number of actives and passives in the top-12.
• #40: To find a good perturbation function, we can construct a typical loss function, where the effect of the divergence is governed by a regularization parameter lambda, and then optimize this loss function to find the parameters of the perturbation function, alpha and beta, which correspond respectively to the boost of active and passive job seekers. However, there is a complicating factor: both the divergence and objective functions depend on a ranking, which depends on a sorting operation, and therefore, traditional gradient based approaches are not readily applicable. Also, what should we set lambda to? We don’t just want to use the lambda that generates the lowest loss, we are actually more interested in what our options are regarding what our tradeoff is going to be between the objective and the divergence function.
• #41: We will discuss computational strategies for optimizing the perturbation function in a moment, but before that, we need to discuss the kind of optimization we are actually interested in. What we really want is Pareto optimization, where there is not one optimal solution, instead, there are some solutions which are better in one objective, while other solutions are better in others. In this plot, we have the objective, the average number of actives and passives in the top-12 results, on the y-axis, and the divergence on the x-axis. The original ranking has on average 4 actives and passives in the top-12, as shown in the table in the top left corner. Also, by definition, the divergence of this original ranking, is 0. Each point (or bubble) in the plot represents a specific assignment to the parameters of the perturbation function: alpha and beta. We see on the plot that the only way to increase the objective on the y-axis, is to also allow an increase on the divergence on the x-axis. We also see that for a given divergence, say 50, there are many assignments of alpha and beta with that divergence, with varying scores on the objective. We want the maximum objective for each divergence, and those are the points in the pareto frontier, which are the red points in the plot. So, no matter what divergence you allow, you should pick a point on the pareto frontier. Back to the table of sample plans, we see that if we set alpha and beta to 1.15, we can double the the number of actives and passives in the top-12 (from 4 to 8) while paying the cost of having a divergence of 64, and that this is a point in the pareto frontier.
• #42: Here we can get a better idea of what the divergence scores actually mean, the top left has the distribution of the original, unperturbed model, and as we move across the quadrant, we see how the divergence increases (0, 27, 54, and 100). In the top left histogram, we see the bump around the 0.9’s, and with each histogram, the bump is gradually attenuated, until there is no more bump in the bottom right. So, we would probably be willing to accept a divergence in the 50-60’s range (as shown in the bottom left), but not in the 100’s, which is what’s shown in the bottom right.
• #43: Given that we only had 2 parameters in our perturbation function, grid search was a satisfactory approach and so that’s what we used. When you have a set of pareto optimal values, typically what it’s done is that you look for the proverbial knee of the curve, a point after which you have to pay too much in one objective to get increases in another, and our curve actually displays this characteristic: the Pareto tradeoff is constant up to a divergence of about 60, which as we saw earlier in the histogram slide, was not too bad. Still we did not know exactly what a given divergence would do to the booking and email rate, so we picked a couple of values to A/B test. We picked the maximum value on that line, the one at the knee, and a point in the middle, which corresponded to a boost of 1.15 and 1.07 respectively.So, what did we expect from the tests? Since we knew the rate of reply to career-related emails of users with high-job seeking intent, as well as the expected proportion of those users in the top-12 recommendations, it was easy to get a ball park figure of how much of an increase in reply rate we would obtain: we expected a 50% increase over control for the 1.07 treatment and a 100% increase over control for the 1.15 treatment. Regarding the other 2 facets of the utility function, the booking rate and the email rate of job posters to candidates, what we hoped was that they would remain unchanged or only be minimally affected.
• #44: So, how did we do? Let’s see how facet of the utility was affected. The booking rate remained mostly unchanged, with possibly a very slight dip of 0.4% on the 1.15 treatment. The email rate, to our surprise actually increased in both treatments. This tells us that somehow, the profiles of users with high-job seeking intent were more appealing to job posters than those who weren’t. Specifically what about their profile was more appealing is something we have yet to look into. This also tells us that maybe the snippets that we show job posters were not a great representation of the value for them, and that perhaps better snippets would lead to higher booking rates. Finally, we see that we were able to increase the reply rate, which is what we had originally set out to do, and that the increase for the 1.15 treatment was double that of the 1.07 treatment: 42% and 22%, which was in line with our expectations. Now, these numbers are pretty good, but why weren’t they as high as we had expected? Well, we had thought that job posters contacted all the recommendations, since it did not cost them more to contact all than to contact one, but as we observed in the email rate, which we were able to improve, job posters do not, in fact, contact all of the recommendations.
• #47: A brand new Recommendation Delivery paradigm – Tested on LinkedIn Groups to generate 2X Group Join rate. Applicable to advertising as well..The idea is simple - Reverse the Social Proof idea . Ask the actor to recommend their connections to interact with this item. - The message comes from the individual not LinkedInInherently socially endorsedTimely and contextualCan be applied to other recommendation paradigms as well. Using social recommendations to drive engagement on products on a network/website
• #48: A brand new Recommendation Delivery paradigm – Tested on LinkedIn Groups to generate 2X Group Join rate. Applicable to advertising as well..The idea is simple - Reverse the Social Proof idea . Ask the actor to recommend their connections to interact with this item. - The message comes from the individual not LinkedInInherently socially endorsedTimely and contextual
• #49: Incredibly powerful whetted paradigm that we are excited to try to rope into our Ads offerings
• #50: A brand new Recommendation Delivery paradigm – Tested on LinkedIn Groups to generate 2X Group Join rate. Applicable to advertising as well..The idea is simple - Reverse the Social Proof idea . Ask the actor to recommend their connections to interact with this item. - The message comes from the individual not LinkedInInherently socially endorsedTimely and contextual
• #51: Incredibly powerful whetted paradigm that we are excited to try to rope into our Ads offerings
• #53: Solve the impedance mismatch by creating the Group representation in the user space. This concept is used extensively at LinkedIn for all kinds of user recommendations, not just groups.
• #55: SamNow we can prompt your connections to validate your skills and expertise through an endorsementThis moves more people through the loop faster
• #56: SamHow would you think about this problem? How do you decide what people and skills to show?
• #57: Now we have all the pieces…To reinforce how this works so well,limited adopted by asking manual entry;accelerate by asking them to confirm, but no validation;social tagging, viral loops, and crowdsourcing -> provides the biggest winYou have a skills section -> people may enter their own skills, though not validatedYou recommend skills to add -> more people add skills, still not validatedYou provide a viral endorsement system -> don’t have the catalyst to get adoptionYou need recommendations as a core piece of this ecosystemSo we have the data, what are the applications? Why is this important?
• #58: PeteTO ADD: “Reid endorsed you for Venture Capital.”It’s not just the number of endorsements, it’s the nature.