2020-02 Solr's New Plugin System
Download as PPTX, PDF1 like64 views
David Smiley's presentation discusses the importance of pluggable architecture in Apache Solr, detailing the types of plugins available and their management. It highlights the transition from built-in to contrib and third-party plugins for improved security and maintenance, as well as the introduction of a package manager for easier plugin installation and configuration. Key developments in plugin management from 2013 to the release of Solr 8.4 are outlined, alongside resources for further information.
2020-02 Solr's New Plugin System
- 1. By David Smiley On February 2020 At a NEST Meetup
- 2. › A Principal Software Engineer at Salesforce › Apache Lucene/Solr committer & PMC member › Author of Solr Enterprise Search Server books
- 3. › Quiz: how many plugin types? ____ – … in Lucene & Solr combined? – A class/interface whose implementation can be specified via configuration › Many more actual plugins exist › Search platforms need to be pluggable
- 4. › Where do you find out about plugins today? – Solr Ref Guide – Colleagues – Conferences, Meetups – Blogs – Search results › Plugin sources – built-in: Comes with Solr; no installation step – contrib: Comes with Solr; need to install JARs(*) – 3rd party: External; compile/download, install
- 5. › Place JARs into a suitable directory – <solr-home>/lib – <solr-core>/lib – solrconfig.xml <lib … /> Sometimes necessary: – server/solr-webapp/webapp/WEB-INF/lib Jetty/JVM plugins: – server/lib and server/lib/ext
- 6. › Read documentation for how to configure › Sometimes a plugin is always configured the same: – <queryparser name=“foo” class=“com.acme.FooQParser”/> › Often not: – TokenFilters: to which field type? Where in chain?
- 7. Consider RPM, Docker Hub, IDEs, many other examples Why: › Discoverability via a common registry › Ease of installation via CLI commands › And especially the 2nd order effects… (break up the Solr monolith)
- 8. › Some built-in plugins become contribs – Better security – Fewer dependencies of Solr core (JAR hell deterrent) – Usable in older Solr releases – Ex: SolrTextTagger? › Some contrib plugins become 3rd-party – Better security – Less burden on maintaining Solr (test time, docs, dev) – Smaller Solr distribution / Docker image – Separate release cycles from Solr – Ex: DIH, Velocity/Solritas
- 9. › Support common plugin types › A public plugin registry › CLI & Web API commands – to install a plugin – to configure a plugin! – to update a plugin (new version) › Plugin management at runtime › Plugin dependency isolation (JAR hell deterrent) Easy to find & install plugins!
- 10. Once upon a time… › 2013-08, SOLR-5103, Grant Ingersoll – Mostly discussion › 2015-02 SOLR-7073, Noble Paul – runtimeLib & .system collection / blob store › 2017-05, SOLR-10665, Jan Høydahl – Designs, POC, PF4J based, conf talk, … abandoned › 2019-07, SOLR-13661, Noble Paul & Ishan C. – Wider feedback; committed to v8.4
- 11. New in 8.4
- 12. › Plugin type – A base class/interface that Lucene/Solr loads dynamically › Plugin – An implementation of a plugin type › Package – A collection of plugins, plus metadata › Repository – A web location (URL) hosting plugins
- 13. › Enable package manager › Add a repository › List packages › Install a package › Deploy a package – CLI vs manual
- 15. Likely more typical than automated… <requestHandler name="/myhandler” class="mypackage:full.path.to.MyClass” /> curl "http://localhost:8983/api/collections/collection1/ config/params" -H 'Content-type:application/json’ -d "{set: {PKG_VERSIONS: {mypackage: '1.0.0'}}}"
- 16. › Where packages (JARs) are actually placed › Distributed peer-to-peer synchronized directory › Potential for other uses – Large resources – machine learning models › Compare to “.system” collection… › Discuss: Good, bad, …
- 17. › Update a plugin live, in-place – No core reload! › Discuss: Good, bad, …
- 18. › Everything is deemed experimental › Support more plugin types – Schema plugins! – Also hot-loading for more types; some can’t do this › Convert contribs to a local plugin repository › Create an official/primary plugin repository › Better documentation – See https://lucene.apache.org/solr/guide/8_4/package- manager.html
- 19. › Who runs the plugin repository / decides inclusion › Self-configuring plugins › Hot-loading plugins › Assign different package versions to different collections › How a configSet declares it needs a package › Eliminating <lib/> in solrconfig.xml › Security (keys, signing, opting-out)
- 20. Resources: › Ref guide – https://lucene.apache.org/solr/guide/8_4/package-manager- internals.html › Solritas packaging example – https://github.com/erikhatcher/solritas/ › Activate Conference Presentation – https://youtu.be/YQBhEojf358 “Dynamic Plugin Loading & Safe Mode Solr”
Editor's Notes
- #4: 114 via ResourceLoader 34 Classloader.forName Solr probably has a greater proportion of those abstractions.
- #6: Most plugins, first section. Examples: JTS dependency… not really a 1st class plugin but 2nd order dependency. Examples: logger. Or SSL related thing.
- #7: Consequently, there are some steps to do in your configuration.
- #20: “Contentious” : subject to debate