Document relations

Document relations
Martijn van Groningen
@mvgroningen

Tuesday, November 6, 12

Overview
• Background
• Document relations with joining.
• Various solutions in Lucene and Elasticsearch


Background - Lucene model
• Lucene is document based.
• Lucene doesn’t store information about relations
between documents.

• Data often holds relations.
• Good free text search over relational data.


Background - Common solutions
• Compound documents.
• May result in documents with many ﬁelds.

• Subsequent searches.
• May cause a lot of network overhead.

• Non Lucene based approach:
• Use Lucene in combination with a relational database.


Background - Example
• Product
• Name
• Description

• Product-item
• Color
• Size
• Price


Background - Example
• Compound Product & Product-items document.
• Each product-item has its own ﬁeld preﬁx.


Background - Other solutions
• Lucene offers solutions to have a 'relational' like
search.
• Joining
• Result grouping

• Elasticsearch builds on top of the joining
capabilities.

• These solutions aren't naturally supported.


Joining


Joining
• Join support available since Lucene 3.4
• Not a SQL join!

• Two distinct joining types:
• Index time join
• Query time join

• Joining provides a solution to handle document
relations.


Joining - What is out there?
• Index time:
• Lucene’s block join implementation.
• Elasticsearch’s nested ﬁlter, query and facets.
• Built on top of the Lucene’s block join support.

• Query time:
• Lucene’s query time join utility.
• Solr’s join query parser.
• Elasticsearch’s various parent-child queries and ﬁlters.


Index time join
And nested documents.


Joining - Block join query
• Lucene block join queries:
• ToParentBlockJoinQuery
• ToChildBlockJoinQuery

• Lucene collector:
• ToParentBlockJoinCollector

• Index time join requires block indexing.


Joining - Block indexing
• Atomically adding documents.
• A block of documents.

• Each document gets sequentially assigned Lucene
document id.

• IndexWriter#addDocuments(docs);


Joining - Block indexing
• Index doesn't record blocks.
• App is responsible for identifying block documents.
• Segment merging doesn’t re-order documents in a
segment.

• Adding athe whole block.block requires you to
reindex
document to a


Joining - block join query

• Parent is the last document in a block.


Block join - ToChildBlockJoinQuery

Marking parent documents



Add block

Add block


• Parent ﬁlter marks the parent documents.

• Child query is executed in the parent space.


Block join & Elasticsearch
• In Elasticsearch exposed as nested objects.
• Documents are constructed as JSON.
• JSON’s nested structure works nicely with block
indexing.

• Elasticsearchoftakes nested documents. and also
keeps track the
care of block indexing


Elasticsearch’s nested support
• Support for a nested type in mappings.
• Nested query.
• Nested ﬁlter.
• Nested facets.


Nested type
• The nested types enables Lucene’s block indexing.
index

curl -XPUT 'localhost:9200/products' -d '{
type "mappings" : {
"product" : {
"properties" : {
Nested offers "offers" : { "type" : "nested" }
}
}
}
}'


Indexing nested objects
index type

curl -XPOST 'localhost:9200/products/product' -d '{
"name" : "Polo shirt",
"description" : "Made of 100% cotton",
nested objects "offers" : [
{
"color" : "red",
"size" : "s",
"price" : 999
},
{
"color" : "red",
"size" : "m",
"price" : 1099
},
{
"color" : "blue",
"size" : "s",
"price" : 999
}
]
}'


Nested query
The nested ﬁeld
curl -XPOST 'localhost:9200/products/product/_search' -d '{
"query" : {
path in mapping.
"nested" : {
"path" : "offers",
"score_mode" : "total", Sum the individual
"query" : { nested matches.
"bool" : {
"must" : [
{
"term" : {
"color" : "blue"
}
},
Color red would match
{ the previous document.
"term" : {
"size" : "m"
}
}
]
}
}
}
}
}'


Nested facets
curl -XPOST 'localhost:9200/products/product/_search' -d '{
"facets" : {
"color" : {
"terms_stats" : {
"key_field" : "size",
"value_field" : "price"
},
"nested" : "offers"
} "facets":{
} "color":{
}' "_type":"terms_stats",
"missing":0,
"terms":[
{
A facet for nested ﬁeld "term":"s",
"count":2,
offers. "total_count":2,
"min":999.0,
"max":999.0,
"total":1998.0,
"mean":999.0
},
...
Counts 2 nested documents ]
for term: s }
}


Query time join
and parent & child relations.


Query time joining
• Documents are joined during query time.
• More expensive, but more ﬂexible.

• Two types of query time joins:
• Parent child joining.
• Field based joining.


Lucene’s query time join
• Query time joining is executed in two phases.
• Field based joining:
• ‘from’ ﬁeld
• ‘to’ ﬁeld

• Doesn’t require block indexing.

Query time join - JoinUtil
• Firstthe documents thatthe terms in the fromField
for
phase collects all
match with the original
query.

• The second the collected termsdocumentsprevious
match with
phase returns the
from the
that
phase in the toField.

• One public method:
• JoinUtil#createJoinQuery(...)


Joining - JoinUtil

Referrer the product id.


Joining - JoinUtil


Joining - JoinUtil

Join utility

• Result will contain one products.
• Possible to do ‘join’ across indices.


Elasticsearch’s query time join
• A parent child solution.
• Not related to Lucene’s query time join.
• Support consists out of:
• The _parent ﬁeld.
• The top_children query.
• The has_parent & has_child ﬁlter & query.
• Scoped facets.


The _parent field
• Points to the parent type.
• Mapping attribute to be define on the child type.
curl -XPUT 'localhost:9200/products' -d '{
"mappings" : {
"offer" : {
"_parent" : {
"type" : "product"
}
}
}
}'

• Elasticsearch uses the _parent field to build an id
cache.
• Makes parent/child queries & filters fast.


Indexing parent & child documents
• Parent document:
curl -XPOST 'localhost:9200/products/product/1' -d '{
"name" : "Polo shirt",
"description" : "Made of 100% cotton"
}' The id of the parent
document. Also used for
• Child documents: routing.

curl -XPOST 'localhost:9200/products/offer?parent=1' -d '{
"color" : "red",
"size" : "s",
"price" : 999
}'

curl -XPOST 'localhost:9200/products/offer?parent=1' -d '{
"color" : "blue",
"red",
"size" : "s",
"m",
"price" : 999
1099
}'


The ‘top_children’ query
curl -XPOST 'localhost:9200/products/_search' -d '{
"query" : {
Child type
"top_children" : {
"type" : "offer",
"query" : {
"term" : { Child query
"size" : "m"
}
},
"score" : "sum" Score mode
}
}
}'

• Internally the in order to getpotentially executed
several times
child query is
enough parent hits.


The ‘has_child’ query
curl -XPOST 'localhost:9200/products/_search' -d '{ Child type
"query" : {
"has_child" : {
"type" : "offer",
"query" : {
"term" : {
"size" : "m"
Child query
}
}
}
}
}'

• Doesn’tdoc. Works as a ﬁlter. into the matching
parent
map the child scores

• The has_parent query matches child document
instead.


Scoped facets
curl -XPOST 'localhost:9200/products/_search' -d '{
"query" : {
"has_child" : {
"type" : "offer",
"query" : {
"term" : {
"size" : "m"
}
},
"_scope" : "my_scope"
} Execute facets inside
},
"facets" : {
a speciﬁc scope.
"color" : {
"terms_stats" : {
"key_field" : "size",
"value_field" : "price"
},
"scope" : "my_scope"
}
}
}'


Conclusion
• Block join & nested object are fast and efficient,
but lack flexibility.

• Query time and parent child join are flexible at the
cost of performance and memory.
• Field based query time joining is the most flexible.
• Parent child based joining is the fastest.

• Faceting in combination with document relations
gives a nice analytical view.


Any questions?


Document relations

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Document relations (20)

Document relations