AEM GEMs Session Oak Lucene Indexes

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Oak Lucene Indexes
Chetan Mehrotra | Senior Computer Scientist
Alex Parvulescu | Senior Developer

Content
 Lucene Index Definitions
 Anatomy of a Query (Restrictions, Sorting, Aggregation)
 Query Diagnostics and Troubleshooting
 Lucene Index Internals (Oak Directory, JMX, Luke)
 Asynchronous Indexing
 Q&A
2

Lucene Index Definition

Index Definition
 Stored under oak:index node
 Define how content gets indexed
 type oak:QueryIndexDefinition
 Required properties
 compatVersion = 2
 type = “lucene”
 async = “async”
4
/oak:index/assetType (oak:QueryIndexDefinition)
- compatVersion = 2
- type = "lucene"
- async = "async"
+ indexRules (nt:unstructured)
+ dam:Asset
+ properties (nt:unstructured)
+ assetType
- propertyIndex = true
- name = "jcr:content/metadata/type"

Index Definition – Index Rules
 Defines which types of node and properties are
indexed
 Rules are defined per nodeType
 Rule consist of one or more property definitions
 Index selected based on match between type used
in Query and presence of indexRule for that type
 Multiple indexRules in same index
 Order important – nodeType matching honors
inheritance
5
SELECT *
FROM [dam:Asset] AS a
WHERE ISDESCENDANTNODE([/content/en])
AND a.[jcr:content/metadata/type] = 'image'
- compatVersion = 2
- type = "lucene"
- async = "async"
+ dam:Asset
+ assetType
- name = "jcr:content/metadata/assetType"
https://jackrabbit.apache.org/oak/docs/query/lucene.html#Indexing_Rules

Index Definition – Property Definitions
 Defines how a property gets indexed
 One or more property definition per indexRule
 Definition mapping done based on matching
property name or regex pattern
 Supports relative property name by there relative
paths
 Order important (if regex are used)
6
SELECT *
WHERE ISDESCENDANTNODE([/content/en])
AND a.[jcr:content/metadata/type] = 'image'
- compatVersion = 2
- type = "lucene"
- async = "async"
+ dam:Asset
+ assetType
- name = "jcr:content/metadata/assetType"
https://jackrabbit.apache.org/oak/docs/query/lucene.html#Property_Definitions

Index Definition – Best Practices
 Precise Index Definition - That indexes just the right amount of content based on your query requirement. Precise index is
happy index!
 Make use of nodetype to achieve a “cohesive index” - This would allow multiple queries to make use of same index and
also evaluation of multiple property restrictions natively in Lucene
 For people familiar with Relational Databases - Nodetype is your Table in your DB and all the direct or relative
properties as columns in that table. Various property definitions are like indexes on those columns.
7
https://jackrabbit.apache.org/oak/docs/query/lucene.html#Design_Considerations

Sample Content to Query Against
8
/content/dam/assets/december/banner.png (dam:Asset)
+ metadata (dam:AssetContent)
- dc:format = "image/png"
- status = "published"
- jcr:lastModified = "2009-10-9T21:52:31"
- app:tags = ["properties:orientation/landscape",
"marketing:interest/product"]
- size = 450
- comment = "Image for december launch"
- jcr:title = "December Banner"
+ xmpMM:History
+ 1
- softwareAgent = "Adobe Photoshop"
- author = "David"
+ renditions (nt:folder)
+ original (nt:file)
+ jcr:content
- jcr:data = ...

Anatomy of Query
9
SELECT
*
WHERE ISDESCENDANTNODE([/content/public/platform])
AND a.[jcr:content/metadata/status] = 'published'
AND CONTAINS([jcr:content/metadata/comment], 'december')
ORDER BY
a.[jcr:content/metadata/jcr:lastModified] DESC
• Nodetype restriction on dam:Asset
• Path restriction on /content/public/platform
• Property restriction on jcr:content/metadata/status
• Fulltext property restriction on jcr:content/metadata/comment
• Sorting done on jcr:content/metadata/jcr:lastModified

Nodetype Restrictions
10
SELECT
*
FROM
[dam:Asset] AS a
WHERE
ISDESCENDANTNODE([/content/public/platform])
AND
a.[jcr:content/metadata/status] = 'published'
AND
CONTAINS([jcr:content/metadata/comment], 'december')
ORDER BY
/oak:index/damAsset (oak:QueryIndexDefinition)
- compatVersion = 2
- type = "lucene"
- async = "async"
+ dam:Asset (nt:unstructured)
+ properties
...
Create index definition node at /oak:index/damAsset with indexRule for dam:Asset

Path Restriction
11
SELECT
*
FROM
[dam:Asset] AS a
WHERE
AND
AND
ORDER BY
Enable evaluatePathRestrictions for indexing paths
Bonus Tip – If all indexable content is under /content/public and query always specify the path restriction then it would be better to define index definition under
/content/public/oak:index (more details)
- compatVersion = 2
- type = "lucene"
- async = "async"
- evaluatePathRestrictions = true
+ properties
...

Property Restriction
12
SELECT
*
FROM
[dam:Asset] AS a
WHERE
AND
AND
ORDER BY
Create property definition node with propertyIndex enabled and name set to relative path
of property
- compatVersion = 2
- type = "lucene"
- async = "async"
+ properties
+ status
- name = "jcr:content/metadata/status"

Fulltext Property Restriction
13
SELECT
*
FROM
[dam:Asset] AS a
WHERE
AND
AND
ORDER BY
Create property definition node with analyzed enabled
- compatVersion = 2
- type = "lucene"
- async = "async"
+ properties
+ status
+ comment
- name = "jcr:content/metadata/comment"
- analyzed = true

Sorting
14
SELECT
*
FROM
[dam:Asset] AS a
WHERE
AND
AND
ORDER BY
- compatVersion = 2
- type = "lucene"
- async = "async"
+ properties
+ status
+ comment
- name = "jcr:content/metadata/comment"
- analyzed = true
+ lastModified
- name = "jcr:content/metadata/jcr:lastModified"
- ordered = true
- type = Date
Create property definition node with ordered enabled and type set to property type. Also
enable propertyIndex if you plan to have some restrictions on it

Fulltext Node Restriction
 Searches for ‘christmas’ in all nodes of type dam:Asset
 Fulltext index for a node is made up fulltext terms made up from
 Node properties – Properties with nodeScopeIndex set to true
 Properties of relative nodes defined by Aggregation Rules
 Aggregation Rules
 Define path patterns for selecting the relative nodes
 Are bound to specific type
 Can be recursive – Relative path refers to nt:file and nt:file has its own aggregation rule defined
 For aggregated nodes all properties whose type are part of includePropertyTypes are included unless a property
definition is defined with nodeScopeIndex=false
15
SELECT * FROM [dam:Asset] WHERE CONTAINS(., 'christmas')

Fulltext - Aggregation
16
/content/dam/assets/december/banner.png
(dam:Asset)
+ metadata (dam:AssetContent)
- dc:format = "image/png"
- status = "published"
- jcr:lastModified = "2009-10-9T21:52:31"
- app:tags =
["properties:orientation/landscape",
"marketing:interest/product"]
- size = 450
- comment = "Image for Christmas launch"
- jcr:title = "December Banner"
+ xmpMM:History
+ 1
- softwareAgent = "Adobe Photoshop"
- author = "David"
+ renditions (nt:folder)
+ original (nt:file)
+ jcr:content
- jcr:data = ...
+ aggregates
+ dam:Asset
+ include0
- path = "jcr:content"
+ include1
- path = "jcr:content/metadata"
+ include2
- path = "jcr:content/metadata/*"
+ include3
- path = "jcr:content/metadata/*/*"
+ include4
- path = "jcr:content/renditions"
+ include5
- path = "jcr:content/renditions/original"
+ nt:file
+ include0
- path = "jcr:content"
image/png
Published
properties:orientation/landscape
marketing:interest/product
December Banner
Image for Christmas launch
Adobe Photoshop
David
Content Aggregation Rules Extracted Terms for Fulltext Index
https://jackrabbit.apache.org/oak/docs/query/lucene.html#Aggregation

Query Result Size
 Oak Fast Result Size
 By default NodeIterator.getSize() returns -1 if result is large as size estimate cost is O(n) due to ACL checks
 ACL Checks can be relaxed (check first ‘k’ only). Enable via system property oak.fastQuerySize.
 OSGi config support with next release
 AEM Query Builder and Pagination
 Make use of p.guessTotal query parameter to avoid costly operation for determining result size
 Use progressive pagination
17

Other Features
 Composing Analyzer – For configuring Stemming, Synonyms, Stop words etc
 Boost – Improving search relevancy
 Tika Config – Control how and which types of binary files are indexed
 Suggestions
 Spell Check
 Pre Extracting Text from Binaries – To speedup reindexing time for repositories having marge number of
binaries having text
18

Tools

Query Explain Tool
 Shipped with AEM 6.1
 Tools -> Operations -> Dashboard -> Diagnosis -> Query Performance
 http://localhost:4502/libs/granite/operations/content/diagnosis/tool.html/_granite_queryperformance
 Shows Slow Query, Popular Query and Explain Query
 ACS Tools (more upto date)
 https://adobe-consulting-services.github.io/acs-aem-tools/explain-query.html
20

Query Explain Tool
21
• Shows logs from various index consulted
• Shows the actual Lucene query fired
• Path Restriction
+:ancestors:/content/public/platform
• Fulltext Restriction
+full:jcr:content/metadata/comment:december
• Property Restriction
+jcr:content/metadata/status:published
• Ordering

Lucene Index Internals

Lucene Index Internals - Directory
 Lucene Directory is stored in the repository
(the source of truth)
 Copy on Read & Copy on Write maintain local
copies for faster access (index content to disk
location mappings are exposed via JMX)
23

Lucene Index Internals - JMX
org.apache.jackrabbit.oak: Lucene Index statistics (LuceneIndex)
 provides a listing of the existing lucene indexes
 http://localhost:4502/system/console/jmx/org.apache.jackrabbit.oak%3Aname%3DLucene+Index+statistics%2Ctype%3
DLuceneIndex
24

Lucene Index Internals - JMX continued
25
org.apache.jackrabbit.oak: IndexCopier support statistics (IndexCopierStats)
 Copy on Read and Copy on Write related stats, of interest is the mapping between index content and location on disk
 http://localhost:4502/system/console/jmx/org.apache.jackrabbit.oak%3Aname%3DIndexCopier+support+statistics%2Ctype%3DIndexCopi
erStats

Lucene Index Internals - JMX continued
26
org.apache.jackrabbit.oak: TextExtraction statistics (TextExtractionStats)
 Very relevant stats related to how much work is done extracting text from binaries
 http://localhost:4502/system/console/jmx/org.apache.jackrabbit.oak%3Aname%3DTextExtraction+statistics%2Ctype%3DTextExtractionSt
ats
 Make sure you remember this one for our experiment with ‘Luke’

Lucene Index Internals - Luke
Let’s run a small experiment: upload a pdf file to the repository, verify if full-text
search works
27
It works! And now let’s see why...

Setting up ‘Luke’ to look at the Lucene index:
1. Why Luke? Luke is a dedicated Lucene index tool, has no alternatives for
viewing content
2. Identify which index you want to look at
3. Export Index Contents
 (easy/online) Lookup the Copy on Read mappings in the JMX console and grab a copy of the index
 (harder/possibly offline) Use the oak console to export the index to a specific location
4. Open ‘Luke’ and make sure you pass in the oak-lucene jar as a classpath entry
(as documented on the docs)
28

29
For the given token ‘mongomk’
there are 2 matching lucene
docs, pointing to the pdf file.
Why 2? Because of index time
aggregation: the parent node
will inherit the ‘:fulltext’
information from its child node.

The default Lucene index defines aggregation for ‘nt:file’s, meaning they will
inherit all extracted full-text information from the ‘nt:resource’ child nodes.
This means that the following search
/jcr:root//element(*, nt:file)[jcr:contains(., 'mongomk')]
Will return a single item:
/granite-gems-lucene/AEM 6 Oak - MongoMK and Queries.pdf
even though the nt:file node itself contains no full-text information
30

Asynchronous Indexing

Asynchronous Indexing - Overview
 AsyncIndexUpdate class is the glue for all existing index implementations (all logging comes from this place)
 Runs as a background job every 5 seconds, for clusters this runs on a single cluster node
 Used mainly with full-text indexes: lucene/solr, also for ordered property indexes (deprecated)
 Efficient: takes care of processing only new content since last successful cycle, uses a fast diff based on checkpoints
 Resilient: in case of error, it will try again on next cycle (no data loss)
 Status exposed via JMX “ IndexStats”
 You can change an index definition to be asynchronous by setting the async property: async=“async”
32

Asynchronous Indexing - Checkpoints
 Checkpoints are a form of read-only tagging of the current
state of the repository
 Each checkpoint has an expected lifetime provided at
creation time, after which it will be removed, as well as
some metadata related to its creation
33
/checkpoints/a6fe070e-deef-4582-85fb-b96b57ecd1a9
- created = 1450285984929
- timestamp = 1536685984929
+ properties
- creator = "AsyncIndexUpdate”
- name = "async”
- thread = "pool-75-thread-4”
+ root // entire repository content
+ libs
+ content
+ apps
....
[SegmentMK representation of a checkpoint]
 The link between the async indexing process and a checkpoint is
established via the /:async node
 /:async@async property must point to an existing checkpoint, otherwise a
full reindex will happen
 /:async@async-LastIndexedTo stores the timestamp up to which the
repository was indexed
 /:async@async-temp is the list of checkpoints to be cleaned up after all
processing is done
/:async
- async = “a6fe070e-deef-4582-85fb-b96b57ecd1a9”
- async-LastIndexedTo = 2015-12-
16T18:13:04.929+01:00
- async-temp = ["6766f0ec-600f-4b8e-95d3-
9b4d04f5877e",
"a6fe070e-deef-4582-85fb-
b96b57ecd1a9”]

Asynchronous Indexing - JMX
org.apache.jackrabbit.oak: "async" ("IndexStats”)
 Start / Done timestamps
 Checkpoints (reference, temp)
 Execution Count & Time, Indexed Nodes Count series
 Errors: failing flag, latest seen error with its timestamp
34

Useful Links
Oak Lucene Docs
https://jackrabbit.apache.org/oak/docs/query/lucene.html
AEM 6 Oak: MongoMK and Queries Gem session
http://dev.day.com/content/ddc/en/gems/aem-6-oak--mongomk-and-queries.html
AEM Docs on Oak Queries and Indexing
https://docs.adobe.com/docs/en/aem/6-1/deploy/platform/queries-and-indexing.html
https://docs.adobe.com/docs/en/aem/6-1/deploy/best-practices/best-practices-for-queries-and-indexing.html
The Index Manager
https://docs.adobe.com/docs/en/aem/6-1/administer/operations/operations-dashboard.html#The Index
Manager
35

AEM GEMs Session Oak Lucene Indexes

AEM GEMs Session Oak Lucene Indexes

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to AEM GEMs Session Oak Lucene Indexes (20)

Recently uploaded (20)

AEM GEMs Session Oak Lucene Indexes