South Big Data Hub: Text Data Analysis Panel
Download as PPTX, PDF3 likes4,939 views
The document outlines the expertise of Trey Grainger, SVP of Engineering at Lucidworks, who has a strong background in information retrieval and search technology. Lucidworks facilitates data acquisition and search-driven analytics through its Solr platform, offering advanced features like machine learning, relevancy ranking, and natural language processing. It emphasizes building intelligent applications and leveraging user interactions to improve search results and recommendations.
South Big Data Hub: Text Data Analysis Panel
- 1. Text Data Analysis Panel: South Big Data Hub Trey Grainger SVP of Engineering, Lucidworks
- 2. Trey Grainger SVP of Engineering • Previously Director of Engineering @ CareerBuilder • MBA, Management of Technology – Georgia Tech • BA, Computer Science, Business, & Philosophy – Furman University • Information Retrieval & Web Search - Stanford University Other fun projects: • Co-author of Solr in Action, plus numerous research papers • Frequent conference speaker • Founder of Celiaccess.com, the gluten-free search engine • Lucene/Solr contributor About Me
- 3. what do you do?
- 6. Lucidworks enables Search-Driven Everything Data Acquisition Indexing & Streaming Smart Access API Recommendations & Alerts Analytics & InsightsExtreme Relevancy CUSTOMER SERVICE RESEARCH PORTAL DIGITAL CONTENT CUSTOMER INSIGHTS FRAUD SURVEILLANCE ONLINE RETAIL • Access all your data in a number of ways from one place. • Secure storage and processing from Solr and Spark. • Acquire data from any source with pre-built connectors and adapters. Machine learning and advanced analytics turn all of your apps into intelligent data-driven applications.
- 12. how do you do it?
- 13. Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene™.
- 14. Key Solr Features: ● Multilingual Keyword search ● Relevancy Ranking of results ● Faceting & Analytics (nested / relational) ● Highlighting ● Spelling Correction ● Autocomplete/Type-ahead Prediction ● Sorting, Grouping, Deduplication ● Distributed, Fault-tolerant, Scalable ● Geospatial search ● Complex Function queries ● Recommendations (More Like This) ● Graph Queries and Traversals ● SQL Query Support ● Streaming Aggregations ● Batch and Streaming processing ● Highly Configurable / Plugins ● Learning to Rank ● Building machine-learning models ● … many more *source: Solr in Action, chapter 2
- 15. The standard for enterprise search. of Fortune 500 uses Solr. 90%
- 16. Reference Architecture (Lucidworks Fusion)
- 17. Bay Area Search Type-ahead Prediction Building an Intent Engine Search Box Semantic Query Parsing Intent Engine Spelling Correction Entity / Entity Type Resolution Machine-learned Ranking Relevancy Engine (“re-expressing intent”) User Feedback (Clarifying Intent) Query Re-writing Search Results Query Augmentation Knowledge Graph Contextual Disambiguation
- 19. what’s next?
- 20. Basic Keyword Search (inverted index, tf-idf, bm25, query formulation, etc.) Taxonomies / Entity Extraction (entity recognition, ontologies, synonyms, etc.) Query Intent (query classification, semantic query parsing, concept expansion, rules, clustering, classification) Relevancy Tuning (signals, AB testing/genetic algorithms, Learning to Rank, Neural Networks) Self-learning
- 21. The Three C’s Content: Keywords and other features in your documents Collaboration: How other’s have chosen to interact with your system Context: Available information about your users and their intent Reflected Intelligence “Leveraging previous data and interactions to improve how new data and interactions should be interpreted”
- 22. Feedback Loops User Searches User Sees Results User takes an action Users’ actions inform system improvements
- 23. ● Recommendation Algorithms ● Building user profiles from past searches, clicks, and other actions ● Identifying correlations between keywords/phrases ● Building out automatically-generated ontologies from content and queries ● Determining relevancy judgements (precision, recall, nDCG, etc.) from click logs ● Learning to Rank - using relevancy judgements and machine learning to train a relevance model ● Discovering misspellings, synonyms, acronyms, and related keywords ● Disambiguation of keyword phrases with multiple meanings ● Learning what’s important in your content Examples of Reflected Intelligence
- 24. Key Technologies • Keyword Search - Lucene/Solr • Taxonomies / Entity Extraction - Solr Text Tagger - Word2Vec / Dice Conceptual Search - SolrRDF • Query Intent - Probabilistic Query Parser (SOLR-9418) - Semantic Knowledge Graph (SOLR-9480) • Relevancy Tuning - Solr Learning to Rank Plugin (SOLR-8542) • General Needs: a solid log processing framework (Apache Spark, Lucidworks Fusion, or Solr Daemon Expression)
- 26. Source: Trey Grainger, Khalifeh AlJadda, Mohammed Korayem, Andries Smith.“The Semantic Knowledge Graph: A compact, auto-generated model for real-time traversal and ranking of any relationship within a domain”. DSAA 2016. Knowledge Graph Semantic Knowledge Graph Traversal software engineer* (materialized node) Java C# .NET .NET Developer Java Developer Hibernate ScalaVB.NET Software Engineer Data Scientist Skill Nodes has_related_skillStarting Node Skill Nodes has_related_skill Job Title Nodes has_related_job_title 0.90 0.88 0.93 0.93 0.34 0.74 0.91 0.89 0.74 0.89 0.780.72 0.48 0.93 0.76 0.83 0.80 0.64 0.61 0.780.55
- 27. Knowledge Graph
- 28. Knowledge Graph
- 30. Contact Info Trey Grainger trey.grainger@lucidworks.com @treygrainger http://solrinaction.com Other presentations: http://www.treygrainger.com