II-SDV 2017: Datafari - Building an Open Source Enterprise Search Solution from Popular Building Blocks
2 likes1,216 views
Datafari is an open-source enterprise search solution that leverages Apache Solr and Elasticsearch to index and analyze documents from various sources. It addresses issues found in third-party proprietary and open-source solutions by offering flexible functionalities and a robust framework for data crawling. The solution includes components like Apache ManifoldCF, a user-friendly UI, and a focus on enterprise search capabilities, all under an Apache v2 license.
II-SDV 2017: Datafari - Building an Open Source Enterprise Search Solution from Popular Building Blocks
- 1. Datafari - Building an Open Source Enterprise Search Solution from Popular Building Blocks CEDRIC ULMER FRANCE LABS II-SDV 25/04/17
- 2. Datafari So what is Datafari? • « Packaged solution » to analyse and search for documents and data • Can index heterogeneous data formats from multiple sources • Federated search interface • Apache v2 licence
- 4. Why Datafari ? Choice of the Apache Solr and Elasticsearch technologies (more about this later...) Three possibilities to answer a customer requirements : • Use a packaged solution available on the market from a 3rd party • Starting from Apache Solr or Elasticsearch (or others) • Develop, gather necessary components for each customer needs • Ensure « production-ready » material: docs, processes, tests. • Create our own packaged solution (yeah!)
- 5. Why Datafari ? Problems with 3rd party proprietary solutions: • Black box • Roadmap not clear • Resilience (bankrupt, acquisition…) Problems with 3rd party open source solutions: • Lack of technical documentation • Difficulty to setup an understandable debug environment • Delay in the embedded components updates: In particular Solr or ES • License issues (mostly viral ones) • Lack of resilience from the makers => Required us to develop our own solution to better address our customer needs
- 6. Why Datafari Idea: • Gather the best of both worlds : • The “packaged” aspect of existing solutions • Many functionalities • All in one • The flexibility of a solution based on Solr and ES • All of that with an Apache v2 licence ☺ • Focus on Enterprise Search: • Admin for search experts • Admin for search admin • Eased AD/LDAP management • Search and data analytics
- 7. Based on 4 building blocks: • Apache Solr • The heart of the search engine • Apache Manifold CF • Crawling documents • Ajax FranceLabs • UI • Elasticsearch • Data analytics Ajax FranceLabs
- 8. Datafari 3.1 Apache Tomcat 7 Data Sources Datafari Search / Admin Apache ManifoldCF CMS DB Fileshares Web Security (AD, LDAP) PostgreSQL Apache Solr 5.5 Document Index Statistics Index Apache ManifoldCF 2.5 Crawler Service Autorization Service ELK Cassandra (User Management)
- 9. Apache Solr Lucene based Full text search engine Apache Top Level project Large communauty (users/devs) Efficient/Reliable Scalable • High availability • Queries • Index volume
- 10. Apache Solr Webapp Java REST APIs XML/HTTP • Indexing • Querying Caching Web admin interface Configuration through XML config files or APIs
- 11. Apache Lucene/Solr – Some refs
- 12. Apache Solr for Datafari Search core of Datafari Preconfigured index for rich documents • Language detection • Standard facets • Autocomplete • Spellchecker Indexing user queries • Enables analytics on search users behavior
- 13. Datafari 3.1 Apache Tomcat 7 Data Sources Datafari Search / Admin Apache ManifoldCF CMS DB Fileshares Web Security (AD, LDAP) PostgreSQL Apache Solr 5.5 Document Index Statistics Index Apache ManifoldCF 2.5 Crawler Service Autorization Service ELK Cassandra (User Management)
- 14. Apache Manifold CF Framework for data crawling Management of incremental crawling Authorization management Programmable crawls (time windows, loads, regex…)
- 15. Apache Manifold CF Many off the shelf connectors: • FileShare (Samba) • JDBC • Website • Alfresco • CMIS • Sharepoint • Mail • Dropbox • LDAP/AD
- 16. Apache Manifold CF for Datafari Manages data crawling Manages authentication Preconfigured integration with our Solr
- 17. Datafari 3.1 Apache Tomcat 7 Data Sources Datafari Search / Admin Apache ManifoldCF CMS DB Fileshares Web Security (AD, LDAP) PostgreSQL Apache Solr 5.5 Document Index Statistics Index Apache ManifoldCF 2.5 Crawler Service Autorization Service ELK Cassandra (User Management)
- 18. Datafari Search Front-End User UI • AjaxFrance Labs Authentication Interactions with Solr (SolrJ) Indexing users queries Admin UI • Solr • ManifoldCF • Statistics
- 19. AjaxFranceLabs Inspired by AjaxSolr Javascript/Ajax client Provides several components: • Manager: backend connection • Widgets • Graphical/Logical components • (Advanced) Search • Facet • Geolocalisation (Based on OpenStreetMap)
- 20. Browser Datafari Server Datafari Search Manager SearchBarWidget ResultWidget FacetWidget Datafari Search Servlet Ajax
- 21. Use case 1 – Oil and Gas Sources: • Sharepoint • Documentum • Fileshare • DB Volume: 28 TB Users: Geoscientists
- 22. Use case 2 – Nuclear Sources: • Fileshare • Oracle • DB Volume: 15 M docs Users: Maintenance operators
- 23. Démo!!!
- 24. Technical Roadmap (1/2) New advanced search Solr 6 Graphical SolrCloud management Always more documentation Annotator
- 25. Technical roadmap (2/2) New languages Consolidation Unit test framework More dashboards in ELK Learning-to-Rank
- 26. Where can I find Datafari Main hub: http://www.datafari.com/en Source code available on Github: • https://code.google.com/p/datafari/ Install packages for Debian 7 and Windows available on: • www.datafari.com Forum: • https://groups.google.com/forum/#!forum/datafari Documentation on Confluence • Technical and functional Tickets and releases on Jira
- 27. Want to follow Datafari ? @francelabs #datafari francelabs francelabs
- 28. Become a Datafarian ! ☺ We are always open to suggestions • “Reorganise your docs…” Contribution • What about a German version ?! • UI widgets ? Most important: your use cases and usage feedback
- 29. CONTACT Don’t hesitate to reach out to us for any info Our corporate website: www.francelabs.com Email: contact@francelabs.com Tél: 09 72 43 72 85 Fax: 09 72 29 28 14