What's new in Lucene and Solr 4.x

luceneSolr = new
LuceneSolr(4.x)

Grant Ingersoll
CTO, LucidWorks

Confidential © Copyright 2012

Search is dead, long live search

• Embrace fuzziness!

• Search is a system building
block

• If the algorithms fit,
use them!

• Search use leads to search
abuse
- Denormalization frees your mind
- Scoring is just a sparse matrix
multiply
http://cheezburger.com/5243950080
• Scoring features are
everywhere
Confidential and Proprietary
2 © 2012 LucidWorks

Search (R)evolution

• “T’ain’t your father’s search engine”
- Non free text usages abound

• NoSQL before NoSQL was cool
- Many DB-like features

• Flexibility during indexing and scoring

• Finite State Transducers FTW!

• Scale

© 2012 LucidWorks

Agenda

• What’s new In Lucene 4?

• What’s new in Solr 4?

• Sneak Peek: what’s ahead?

© 2012 LucidWorks

Lucene 4


Up and to the Right

• http://people.apache.org/~mikemccand/lucenebench/in
dexing.html


Lucene: Flexibility

• Flexible Index Formats
- New posting list codecs: Block, Simple Text, Append (HDFS..),
etc
- Pulsing codec: improves performance of primary key searches,
inlining docs, positions, and payloads, saves disk seeks

• Pluggable Scoring
- Decoupled from TF/IDF
- Built in alternatives include BM25 & DFR
» http://en.wikipedia.org/wiki/Okapi_BM25
» http://terrier.org/docs/v3.5/dfr_description.html

© 2012 LucidWorks

Lucene: Speed and Memory

• Native Near Real Time (NRT) support
- Per segment
- FieldCache can be controlled to only load new segments
• Soft commit
- Faster without fsync, allows quicker update visibility
• DWPT (Document Writer per Thread)
- Faster more consistent index speed
• Faster fuzzy & wildcard query processing
- Higher performance searching
• String -> BytesRef
- Much improved data structure
- … means less memory and less garbage collection effort

© 2012 LucidWorks

BytesRef memory management improvements

• On a Wikipedia index (11M documents)
- Time to perform the first query with sorting (no warmup queries)
Solr 3x: 13 seconds, Solr 4: 6 seconds.

- Memory consumption Solr 3x: 1,040M, Solr 4: 366M.

- Number of objects on the heap. Solr 3x: 19.4M, Solr 4: 80K. No,
that’s not a typo.

- http://searchhub.org/2012/04/06/memory-comparisons-between-
solr-3x-and-trunk/


FuzzyQuery

• http://people.apache.org/~mikemccand/lucenebench/F
uzzy2.html


QPS (primary key lookup)

• http://people.apache.org/~mikemccand/lucenebench/P
KLookup.html


Lucene: Features

• Doc Values • DirectSpellChecker
- Store data in column order - No more sidecar index!
- Tradeoffs when using vs. • Geospatial improvements
FieldCache (more later)

- http://searchhub.org/2013/04
/02/fun-with-docvalues-in-
solr-4-2/
- http://www.slideshare.net/luc
enerevolution/willnauer-
simon-doc-values-column-
stride-fields-in-lucene

© 2012 LucidWorks

Solr 4


Solr 4: Features

• Search/Faceting/Relevance
- New Relevance Function Queries (tf, df, others)
- Pivot Faceting
- Pseudo-join
- DirectSpellChecker support
- Improved Spatial (more later)
• Indexing
- New Update Processors, including scripting option
- NRT
• Other
- DocTransformer pluggability
- New Admin UI

© 2012 LucidWorks

Geospatial improvements

• Multiple values per field
• Index shapes other than points (circles, polygons, etc)
• More complex interactions than point in a circle

• Indexing:
- "geo”:”43.17614,-90.57341”
- “geo”:”Circle(4.56,1.23 d=0.0710)”
- “geo”:”POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30))”
• Searching:
- fq=geo:"Intersects(-74.093 41.042 -69.347 44.558)"
- fq=geo:"Intersects(POLYGON((-10 30, -40 40, -10 -20, 40 20, 0
0, -10 30)))”

© 2012 LucidWorks

/solr

© 2012 LucidWorks

SolrCloud

• Distributed/sharded indexing & search
- Auto distributes updates and queries to appropriate shards
- Near Real Time (NRT) indexing capable
• Dynamically scalable
- New SolrCloud instances add indexing and query capacity
• Reliable
- No single point of failure
- Transactions logged
- Robust, automatic recover
• http://wiki.apache.org/solr/SolrCloud

© 2012 LucidWorks

SolrCloud’s capabilities

• Transaction log
- All updates are added to the transaction log. The tlog provides support for: durability for
updates that have not yet been committed, peer syncing, real-time get (retrieve documents
by unique id) always up to date because it checks the tlog first, does not require opening a
new searcher to see changes
• Near Real Time (NRT) indexing
- Soft commits make updates visible
- Hard commits make updates durable
• Durability
- Updates to Solr may be in several different states: buffered in memory, flushed, but not
committed or viewable, soft committed (flushed and viewable), committed (durable)
- The transaction log ensures data is not lost in any of these states if Solr crashes.
• Recovery
- Solr uses the transaction log for recovery; on startup Solr checks to see if the tlog is in a
committed state, if not updates since the last commit are applied
• Optimistic locking
- Solr maintains a document version (_version_ field); updates can now specify _version_;
updates to incorrect version will fail

© 2012 LucidWorks

SolrCloud details

• “Leaders” and “replicas”
- Leaders are automatically elected
• Leaders are just a replica with some coordination
responsibilities for the associated replicas
• If a leader goes down, one of the associated replicas is
elected as the new leader
• New nodes are automatically assigned a shard and
role, and replicate/recover as needed
• SolrJ’s CloudSolrServer
• Replication in Solr 4
- Used for new and recovering replicas
- Or for traditional master/slave configuration


Solr as NoSQL

• Characteristics
- Non-traditional data stores
- Not designed for SQL type queries
- Distributed fault tolerant architecture
- Document oriented, data format agnostic(JSON, XML, CSV,
binary)
• Updated durability via transaction log
• Real-time /get fetches latest version w/o hard commit
• Versioning and optimistic locking
- w/ Real Time GET, allows read/write/update w/o conflicts
• Atomic updates
- Can add/remove/change and increment a field in existing doc
w/o re-indexing

© 2012 LucidWorks

Distributed Key / Value Pair Database

• Real-time Get combined with Solr Cloud make a very
powerful key/value pair database
- Durable (tlog)
- Isolated (Optimistic locking)
- Redundant (Solr Cloud Replicas)
- Distributed & scalable (billions of keys, Solr Cloud Sharding)
- Efficient Multi-tenant (Solr Cloud document routing, Solr 4.1)
- Fast (milli-second response time, Pulsing Codec)
- Real-time (tlog)


Routing

• Allows you to route documents and queries to a subset
of shards
• Provides efficient multi-tenancy
• Indexing:
- A shard key can be prepended to the unique document id:
shard_key!unique_id
- Documents with the same shard_key will reside on the same
shard.
• Querying: shard.keys=shard_key1!...
- Much more efficient then searching the entire collection.


Looking ahead

• Automatic shard splitting
• Query parsing: rich query tree control via JSON/XML

• “Schemaless”
- Marketing term meaning convention over configuration for fields

• More programmatic control over system

• Continually improving performance, scalability, and
robustness

© 2012 LucidWorks

• Want to learn more?

• Join us in San Diego April 29 – May 2, 2013

• http://lucenerevolution.org/

• http://lucenerevolution.org/2013/agenda


Resources

• Lucene/Solr
- http://lucene.apache.org

• Me
- @gsingers, grant@lucidworks.com
- http://www.manning.com/ingersoll

• Company
- http://www.lucidworks.com
- http://www.searchhub.org
- Products, Support, Training on and around Lucene and Solr

© 2012 LucidWorks

What's new in Lucene and Solr 4.x

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to What's new in Lucene and Solr 4.x (20)

More from Grant Ingersoll (11)

Recently uploaded (20)

What's new in Lucene and Solr 4.x

Editor's Notes