Lucene, Solr and java 9 - opportunities and challenges
Download as PPTX, PDF1 like893 views
Apache Lucene and Solr needed to be updated to work with Java 9's new module system. This introduced challenges around strong encapsulation and reflective access. The talk discussed changes like compact strings and performance improvements from intrinsics and the G1 garbage collector. It also recommended using multi-release JARs to include Java 9 specific implementations of utils classes for compatibility. Migrating to Java 9 could improve security and performance in some cases for Elasticsearch users.
Lucene, Solr and java 9 - opportunities and challenges
- 1. Apache Lucene, Solr and Java 9: Opportunities and Challenges Uwe Schindler SD DataSolutions GmbH / Apache Software Foundation thetaph1 – https://www.thetaphi.de
- 2. My Background • Committer and PMC member of Apache Lucene and Solr - main focus is on development of Lucene Core. • Implemented fast numerical search and maintaining the new attribute-based text analysis API. Well known as Generics and Sophisticated Backwards Compatibility 👮. • Elasticsearch lover. • Working as consultant and software architect at SD DataSolutions GmbH in Bremen, Germany. • Maintaining PANGAEA (Data Publisher for Earth & Environmental Science) where I implemented the portal's geo-spatial retrieval functions with Apache Lucene Core and Elasticsearch.
- 3. What is this talk about? • History of Java 9 and Apache Lucene/Solr • Migration and testing your with Java 9’s module system (Jigsaw) • Common pitfalls with Java 7 / Java 8 code, that just used to work • Performance?
- 4. History Oracle & Apache Lucene
- 6. Chronology: Friday, July 29, 2011 6
- 7. Reaction Oracle (Rory O’Donnell) contacted Lucene PMC. Weekly preview builds. Other Open Source projects started to test with preview builds of Java 8 – and later Java 9. Easy and fast bug reporting!
- 9. Java 9 and Apache Lucene Going forward...
- 11. What changed in Jigsaw? (module system) • Strong encapsulation: – Code only sees classes from packages exported to your code – Private APIs are private – especially those in the JDK! • Your code behaves as if it will be executed with a security manager! ☺
- 12. What else is wrong with Jigsaw? #ReflectiveAccessToNonExportedTypes #AwkwardStrongEncapsulation • New since build 148 of Java 9 • Prevents reflective access to any class from Java runtime #AwkwardStrongEncapsulation: A non-public element of an exported package can still be accessed via the AccessibleObject::setAccessible method of the core reflection API. The only way to strongly encapsulate such an element is to move it to a non-exported package. This makes it awkward, at best, to encapsulate the internals of a package that defines a public API.
- 13. Undocumented APIs? Unsafe, Byte Buffers & Co.
- 16. Unsafe got a new method!?
- 18. Other Changes Compact Strings & Co.
- 19. Compact Strings Java 9 internally stores strings in compact form, if they only contain ISO-8859-1 characters
- 20. Indyfied String Concat "Hallo " + 123 + ' ' + object + " is a concatted string"; • Java 1.0 to 1.8: a chain of StringBuilder.appends() • Java 9: Invokedynamic with StringConcatFactory: String concat(String,int,char,Object,String)
- 21. Just a funny detail...
- 24. Intrinsics java.util.Objects / java.util.Arrays classes: • Bounds checks • Array comparisons (signed / unsigned) • Array differences
- 25. Know this type of code?
- 26. Solution: Multi-Release JAR (JEP 238) • Lucene adds plain Java implementations of java.util.Objects and java.util.Arrays to own codebase (with exact same signatures) • After compilation all class files are “patched” to use Java 9 signatures and stored in separate folder • Builds MR-JAR with: – unmodified Java 8-compatible classes – Patched classes with Java 9 signatures in extra folder
- 28. New Default Garbage Collector • G1GC is now the default – Previously it was ParallelGC – No need to care in most cases, as Solr uses a hardcoded default • CMS collector deprecated! – Warning on start of Solr! – Migrate to G1GC? $ java -XX:+UseConcMarkSweepGC Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
- 29. Why update your cluster to Java 9 ? Improvements?
- 31. Even more security with Elasticsearch! No risk of bad plugins hacking Java internals!
- 32. • Slightly improved performance for some queries! • With coming Lucene patch (LUCENE- 7966): – Compression of large blobs during indexing (Elasticsearch JSON “_source”) – Sorting with MMapDirectory