Fried sp techcon hybrid search deeper dive
1 like669 views
The document discusses hybrid SharePoint with the new Cloud Hybrid Search feature. It describes how Cloud Hybrid Search allows for a unified search experience across on-premises and cloud content through a single index. It provides an overview of the logical architecture including crawling, query processing, and failure modes. Benefits discussed include reduced infrastructure needs and support for cloud features like Office Graph/Delve.
Fried sp techcon hybrid search deeper dive
- 1. Hybrid SharePoint with the new Cloud Hybrid Search Jeff Fried CTO, BA Insight
- 2. Cloud Search Service Application • Unified index with on-premises and cloud content • Feeds Office Graph/Delve experiences • Supports Search as a Service • Reduces search server footprint Audio text And search indexpropertiessignals Metadata extraction and processing
- 4. 2 Hybrid SharePoint sessions on Monday 4-5:15 – pick one, review the other later SharePoint Hybrid: The Sure Path Forward - Ben Curry Debunking the Hybrid SharePoint Infrastructure Dilemma - Jill Hannemann & Adam Levithan 2 Hybrid search sessions on Tuesday – go to one or both 11:45am The Future of Microsoft Search is Here! Cloud SSA - Jeff Fried & Ben Curry 3:45pm Hybrid SharePoint with the new Cloud Hybrid Search - Jeff Fried Hybrid Sessions at
- 5. Focused on Search and SharePoint since 2004 Longtime Search Nerd • CTO, BA Insight • Senior PM, Microsoft • VP, FAST • SVP, LingoMotors About Jeff Fried Passionate About • Search • SharePoint • Search-driven applications • Information Strategy Blog: DoMoreWithSearch.com Technet Column “A View from the Crawlspace” jeff.fried@bainsight.com
- 6. About BA Insight – Connectivity – Applications - Im – Classification - – Analytics
- 8. Approaches to Hybrid – by Workload Split Workload different tools in different places Split User task uses content or sites across ‘the divide’ Exchange, SharePoint, Lync OneDrive, Yammer, PowerBI, Delve Extranet, Mysites, Team Sites, Project Sites Portals, Intranet, Services/Applications Links Search
- 9. Online On-Prem Cloud Hybrid Search Cloud SSA Text & Metadata
- 10. Online On-Prem Logical Architecture: Crawling Cloud SSA Cloud SSA ParseCrawl SCS ACL Map Process Blob store queue
- 12. Online On-Prem Logical Architecture: Query Cloud SSA Query processing
- 13. Online On-Prem Failure mode: what if you can’t reach the cloud? Cloud SSA
- 14. Online On-Prem Combination: double crawling Cloud SSA Text & Metadata
- 15. Mechanisms Cloud SSA Remote Result Source Cloud App model Add-ins External Content Federation Identity and Directory Sync
- 16. Benefits of Cloud Hybrid Search
- 18. External Content (on-premises and/or in the cloud) SharePoint Server (On-premises or Hosted) Office 365 SharePoint Online Content Onedrive for Business Content Connectors SharePoint Content Adding External Content
- 19. • • • • • • • • • • • • • • • • BA Insight Connectors Mailbox and Archiving Systems • Microsoft Exchange • Microsoft Exchange Online • IBM Lotus Notes • Symantex Evault • Autonomy EAS / (Zantaz) • • • • • • • • • • • • • • • • • ERP and Portal Systems • • • • • • • • • • • • Plus a proven architecture and process for creating new connectors to complex systems
- 20. External Content in O365 UX Unified view across all content - on-premises and on-line - inside and outside SharePoint
- 21. Scaling
- 22. External Content (on-premises and/or in the cloud) Custom Processing CEWS Bottlenecks: 1) Source systems 2) Content Processing 3) Indexer ….
- 23. External Content (on-premises and/or in the cloud) Bottlenecks: 1) Uplink 2) Source systems ….
- 24. 24 Performance
- 25. External Content (on-premises and/or in the cloud) CEWS Custom Processing Bottlenecks: 1) Uplink 2) Source systems 3) Content Processing ….
- 26. Performance Monitoring and Bandwidth (Get-Counter -ListSet "Search Gatherer Azure Plugin - SharePointServerSearch").counter
- 27. 500K items crawled on an Azure D3 50 DPS 100 DPS 1 hour
- 28. Less servers is OK • • •
- 30. • • Directory Synchronization SID S-1-5-21-1212121212-1212121212-1212 jaden@corp.hybridsearch.com msOnline- OnPremiseSecurity Identifier S-1-5-21-1212121212-1212121212-1212 PUID PUID-XXXX-XXXXXXXXXX
- 31. Mapping of Access Control Lists Allow: S-1-5-21-1212121212-1212121212-1212 PUID-XXXX-XXXXXXXXXX • User SIDs are mapped to PUIDs • Group SIDs are mapped to Object IDs • «Everyone» and «Authenticated users» are mapped to «Everyone except external users»
- 32. SUPPORTED – Custom IFilter – BCS connectors – Partner connectors Customizations: Supported & Unsupported SUPPORTED – Tenant level schema mapping – Query rules – Result sources On-premises In the cloud NOT SUPPORTED • Content that requires custom security trimming NOT SUPPORTED • Site collection level schema mapping • Custom security trimming • Custom entity extraction • Content enrichment web service
- 33. 1) 2) 3) Cloud Hybrid Search Limitations + Workarounds 33 Feature OOB Limitation BA Insight CEWS not available with Cloud SSA available via connector framework Entity Extraction not available with Cloud SSA available via autoclassifier Custom Security Trimming not available with O365 index can 'map down' to AD groups Thesaurus SharePoint Online doesn't support a thesaurus can use Federator - with SP server- based search center Removal of on-premises search results not available with Cloud SSA (could provide a custom solution)
- 35. External Content (on-premises and/or in the cloud) SharePoint Server (On-premises or Hosted) SPO Content OneDrive Content Connectors SharePoint Content Connector Framework Office 365 AutoClassifier (app version) CEWS Custom Processing
- 36. External Content (on-premises and/or in the cloud) SharePoint Server (On-premises or Hosted) SPO Content OneDrive Content Connectors SharePoint Content Connector Framework AutoClassifier Office 365 AutoClassifier (app version) CEWS
- 37. DLP Sensitive Data Search works with hybrid Search for sensitive data across on-premises and SharePoint Online All Built-in sensitive types Identification and export Extends to data in OneDrive Sensitive Information type detection through KQL searches Get instant statistics Preview & export results
- 38. Right now: only when you query for it
- 39. A global single index solution Cloud SSA Cloud SSA Cloud SSA Cloud SSA Cloud SSA
- 40. – – – NOT OOB …. but there’s a way to handle them all Scenarios 40
- 42. OOB Federated Search User Experience Results from Cloud Results from SharePoint On-Premise Refiners from Cloud only No termset synchronization Result Blocks (not interleaved)
- 45. 45 Full Range of Hybrid Search Configurations Scenario Most systems and portals hosted on-premises Most systems and portals hosted in the cloud Must work across borders but maintain data residency Single Single MultiSearch Search Index Search Index Across Multiple in SP Server in SP Online * Search Indices How it works Crawls SP Online and other sources from SP Server Crawls SP Server and other sources from Cloud SSA, pushes text & metadata to SP Online Searches SP Online and SP Server simultaneously; combines the results Advantages Simplest approach; best search experience Low footprint on-premises; can use online features (Delve, DLP) Only solution for some scenarios BA Insight Improvement over OOB no OOB solution Adds content outside SP Preview content outside SP2013 Supports content enrichment Provides single interleaved result set and refiners * requires Microsoft Cloud SSA Approaches for Hybrid SharePoint Configuration
- 47. Should I run index reset? NO!
- 48. Best Practice: Content Source Naming & Deletion{ { { {
- 49. Action CrawlDB state Office 365 index state User’s view Create contentsource1_v1 Crawl doc1 doc1 in crawldb Doc1 indexed Doc1 is searchable “index reset” <empty> Doc1 indexed Doc1 is searchable Create a result source to exclude contentsource1_v1 from the tenant & search center site collections <empty> Doc1 indexed Doc1 is no longer searchable Tenant Admin opens SR to delete ALL cloud SSA content. <empty> <empty> All external content has been removed Create contentsource1_v2 Crawl doc1 doc1 in crawldb Doc1 indexed Doc1 searchable again Orphaned Content
- 50. 50 Customer Example: ACE Built on SharePoint 2013 – but couldn’t run as-is in O365
- 51. SharePoint Server in Azure in hybrid configuration with O365 Tenant Virtual Network Cloud Service Availability Set Active Directory & DNS Cloud ServiceCloud Service Availability Set Front End Availability Set App server Availability Set Database Microsoft Azure Gateway subnet Active VPN On-premises environment Optional!
- 52. Example: Using Search-First Migration with Hybrid Cloud Service Availability Sets SharePoint Services Farm Microsoft Azure SharePoint Online Site collections Office 365 Tenant SharePoint 2013 Content Farms SharePoint 2010 Farm(s) 2) Migrate / Upgrade Content Farms Each site collection can be moved independently Can be on-premises, in O365, or hosted in Azure 3) Decommission old farm(s) 1) Establish Search Service (using Azure IaaS)
- 56. Key Considerations for Hybrid: Workloads, Environment, Data, Customizations Availability of features Online versus On-Premises on particular workloads Significant investments in customization of On-Premises workloads Concerns over global network performance with remote sites Regulatory considerations Manageability concerns
- 61. Tools 62