Improving Enterprise Findability: Presented by Jayesh Govindarajan, Salesforce
1 like901 views
1) Jayesh Govindarajan presented on improving enterprise search and findability at Salesforce. He discussed how enterprise search differs from consumer search, challenges with enterprise findability, and machine learning algorithms like LETOR that can be used. 2) Govindarajan explained that diversity of data, intentions, and customers makes enterprise search more complex than consumer search. Most enterprise search relies on simple ranking functions that may not reflect relevance well. 3) Machine learning algorithms like logistic regression and learning to rank can learn relevance from user behavior data like clicks and views. These algorithms output ranking models that can be deployed to search engines like Solr.
![Most ranking functions start off with a few boosts
and end up like….this
Form
1. Query independent signals - multiplicative boosts in range [1-3]
2. Entity specific signals - additive boosts in the range of [1-12]
a. Accounts, Contacts, Leads - LastActivityScore, LastModifiedScore
b. Cases - CaseStatus, CaseEscalationScore
3. ...](https://image.slidesharecdn.com/jayeshgovindarajanimprovingenterprisefindabilityrev2016-161104162452/85/Improving-Enterprise-Findability-Presented-by-Jayesh-Govindarajan-Salesforce-12-320.jpg)
Improving Enterprise Findability: Presented by Jayesh Govindarajan, Salesforce
- 1. Jayesh Govindarajan Search Relevance @ Salesforce Improving Enterprise findability
- 2. Jayesh Govindarajan Senior Director Search Relevance, Data Science Salesforce 1. How is search in the enterprise different ? 2. Enterprise findability problem 3. Relevance, LETOR algorithms 4. Deploying models in solr 5. A model for every customer 6. Putting the pieces together
- 3. Forward-Looking Statements Statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
- 4. Largest Enterprise Search Service! 1.6 300TB+ 600M+ Queries / Week <2min Incremental Index Latency 500B+ Average Click Rank Index Size Documents in the Index <120ms Query Latency on Search Server 7B+ Index Updates / Day
- 5. Empower enterprise users to effortlessly find all the information they need in order to be successful with Salesforce Intelligent, Fast and Powerful Be a competitive differentiator for Salesforce The Search Vision What information do you need?
- 6. Demo time
- 7. 1. How is enterprise search different ?
- 8. Diversity of Data is a Challenge! Sales Cloud Structured data SFA, B2C Service Cloud Unstructured data Case Mgmt, KB, Field Svc Community Cloud Enterprise Social data Q&A, Chatter, Files App Cloud Search APIs Person Search People data
- 9. Diversity of Intentions: A Service agent exploring a community forum to educate himself: Recall A Service agent looking for a case similar to the one she is currently assigned: Precision A Sales rep looking for a named account to call: Precision A Sales rep looking for contacts in an industry within a certain geo: Recall Patterns of search and discovery differ by user roles, and searched entity
- 10. Customer diversity: one size doesn’t fit all Matching models to Customer Orgs Some Orgs want a lower coefficient in some cases Some Orgs want a higher coefficient
- 11. 2. Understanding enterprise findability problem
- 12. Most ranking functions start off with a few boosts and end up like….this Form 1. Query independent signals - multiplicative boosts in range [1-3] 2. Entity specific signals - additive boosts in the range of [1-12] a. Accounts, Contacts, Leads - LastActivityScore, LastModifiedScore b. Cases - CaseStatus, CaseEscalationScore 3. ...
- 13. Getting to a machine learned function has challenges Constraint 1. Customers build apps on enterprise search platforms. One cannot simply cutover to a new ranking system. Key Lessons 2. Understanding the current search equation is key to anticipating customer breakage/impact. 3. Important to formalize the “Human Intelligence” equation behind a working system.
- 14. 3. Machine learning and Learning to Rank methods
- 15. Build a probabilistic model of relevance. Chance that user clicks on the ith record: pr(r,q,i)= L(r,q)*Rb(i) - L() is a function which maps the (record,query) pair to the likelihood that a user clicks on r in response to q - Rb() is a function which corrects for the positional bias probabilities
- 16. Master Relevance Equation Goal: learn the best linear function of these variables From queries and clicks.
- 17. Logistic Regression docPV score Clicked queryId -hnjnxlbxd 0 10.892 1 1ttuuy6n3 5 0.230 0 1ttuuy6n3 0 0.232 0 1ttuuy6n3 0 0.230 0 1ttuuy6n3 0 0.230 1 1ttuuy6n3 0 0.244 0 1ttuuy6n3 0 0.231 0 1ttuuy6n3 6 0.228 0 1ttuuy6n3 5 0.228 0 1ttuuy6n3 0 0.231 0 If this was the data, the simplest approach would be logistic regression P(clicked) = sigmoid (a0 + a1 (docPV) + a2 (score)) Incremental effects of docPV and score on relevance Bias term that affects all observations equally The only thing to change about this is that we want a separate bias term for each positional rank.
- 18. Page views and Lucene score for result in position 1 Page views and Lucene score for result in position 3 Position that was clicked Example: Query/Click Data Goal:
- 19. b1 b2 b3 b4 b5 1 Result 1 Result 2 Result 3 Result 4 Result 5 = ● Five logistic regressions with shared weights, but different biases. ● Coefficients and biases are learned via MLE (SGD). Learning
- 20. Results ---------- ACCOUNTS ---------- Coefficients ------------------------- docPC 0.203 docPV 0.312 doclm_score 0.642 lastAccessed_score 0.34 score 0.251 Rank Bias ------------------------- Rank 1 1.0 Rank 2 0.884 Rank 3 0.843 Rank 4 0.788 Rank 5 0.94 The model is much better at predicting which of the five results will be clicked. Detect 50% of occurrences where the 5th result was clicked. Wrong on 1 out of 8 attempts All else being equal, the odds of clicking position 2 are about .884 compared to the odds of clicking position 1 Opportunities are the subject of more general searches. E.g. “Which opportunities are John Smith working on?” Searches for accounts or cases are more likely to be very specific. E.g. “I have a specific account in mind…”
- 21. Results: Coefficients (normalized by relative influence) q1 / docPV q2 / docPC d1 / doclm_score d2 / lastaccessed d3 / oppclosedate d4 / oppclosed d5 / caseEscState d6 / caseClosed Lucene Score users 3.12 7.24 groups 4.59 4.59 files 1.32 3.74 cases 0.85 0.15 0.49 0.33 1.21 leads 2.01 2.30 1.01 1.62 contacts 1.87 1.42 1.15 0.76 2.76 accounts 1.64 1.07 1.65 0.96 4.17 oppy 2.27 0.53 0.53 0.62 0.81 3.64 kb 0.50 0.32 2.00 ● Lucene score is always most important (except for leads) ● LastModified is extremely important for accounts and leads, but not at all for cases.
- 22. 4. Implementing Model representation in SOLR
- 23. Relevance Metadata JSON format { "schema": 1.0, "global": { "pc_s": 1.5, Boost Parent Child scores by 1.5 "pv_s": 2.0, Boost PageView counts by 2 "lm_s": 1.333, Last Modified }, "entity": { "500": { Specific for Cases (key prefix 005) "cc_s": 4.0, Boost Open Cases "lm_s": 1.0, Apply a different boost for Last Modified }
- 24. Relevance Metadata (RMD) Relevance Model { schemaVersion:"V1", orgId:"00D1234567", Account:{ PV:2.1, LastMod:0.5 },... QIR:"Solr", DBRerank:"CoreApp" } Model Store JSON is stored in a Blob field in Setup BPO or HBase table Changes to format / schema won't affect table (it's just a blob) AB Experiment Name: Org: Params: JSON Model Deploy Org: Params: JSON Querying Pass the same JSON to Query layer and Solr Server, each should have code that knows what to do given the JSON Model Builder (offline) solutions to help us (Devs / PM etc) build the model JSON files Pass entire JSON to Solr, or just boost function The same JSON is used to run AB experiment and eventually deploy into production Relevancy coefficients are expressed in a JSON data structure, so that we can easily specify per-entity or global-to-org coefficients
- 25. 5. Stacking base and custom models
- 26. Recap: And one size doesn’t fit all Cluster Orgs based on their ACR response curves Green orgs are hurt badly by increasing coefficient changes Blue orgs are hurt badly by decreasing coefficient changes Reds are hurt badly either way Three distinct clusters observed in hierarchical clustering
- 27. Stacking models RELEVANCE PIPELINE Base Model (All orgs, all entities) ...Accounts Model Case Model Knowledge Article Model Feeds Model } Entity Signals Org 1 Org 2 ... Org n
- 28. Putting the pieces together: Relevance ML Pipeline, Runtime
- 29. Relevance ML Pipeline RELEVANCE PIPELINE Common representation of ranking model. Infra to automate training, A/B testing and deployment. FEATURE DETAILS ● Config driven Model Deployment ● Automated model generation ● Training and A/B experimentation Core App Solr Model Deployment Model Evaluation A/B Experiments Model Building ML Training Infrastructure Search Query Logs RMD JSON Training Models RMD JSON RMD JSON RMD JSON Relevance Runtime RMD JSON LEARN TEST SHIP RMD JSON Relevance Model Representation Ranker RMD JSON Training Models
- 30. Relevance Runtime Infrastructure RELEVANCE RUNTIME Executes machine learning models as a service in solr, at scale FEATURE DETAILS ● Ranking functions in solr (linear and non-linear) ● Support for org, entity specific models ● Query Understanding Enterprise Data Index Training Data Signals - Interaction, Behavior Clicks, Likes, Mentions TF/IDF Query Understand ing (NLP, Q&A) Level 1 Top K Ranker Level 2 ML Ranker Model Salesforce Cloud Feature Engineering Snippet Generation Level 3 Post-Ranking Model MachineLearning Pipeline Search Results Conte nt Users/Acti ons Query /Intent query
- 31. Thank you
- 32. We are Hiring ! ML Engineers Engineering Managers Software EngineersData Scientists Join Salesforce Search Cloud Mining Intent @ Work
- 33. Results: Positional Bias Opportunities are the subject of more general searches. E.g. “Which opportunities are John Smith working on?” Searches for accounts or cases are more likely to be very specific. E.g. “I have a specific account in mind…”