Interactive Query and Search for your Big Data
1 like707 views
This document discusses building a web application for interactively querying and exploring big data with Solr. It describes the goals of quickly exploring data and making Solr/Hadoop easier to use. The architecture is presented as a user interface on top of the standard Solr API using REST. The history and improvements of the user experience are covered. Advanced features like analytic facets, nested facets, and operations on data buckets are introduced.
![ADDING A WIDGET
LIFECYCLE
Query
part
1
Query
Part
2
Augment
Solr
response
facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000&
f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10
q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000]
{
'facet_counts':{
'facet_ranges':{
'bytes':{
'start':10000,
'counts':[
'900000',
3423,
'1800000',
339,
...
]
}
}
}
{
...,
'normalized_facets':[
{
'extraSeries':[
],
'label':'bytes',
'field':'bytes',
'counts':[
{
'from’:'900000',
'to':'1800000',
'selected':True,
'value':3423,
'field’:'bytes',
'exclude':False
}
], ...
}
}
}](https://image.slidesharecdn.com/june11310pmclouderarigaux-150629222838-lva1-app6891/85/Interactive-Query-and-Search-for-your-Big-Data-14-320.jpg)
![JSON TO WIDGET
{
"field":"rate_code",
"counts":[
{
"count":97797,
"exclude":true,
"selected":false,
"value":"1",
"cat":"rate_code"
} ...
{
"field":"medallion",
"counts":[
{
"count":159,
"exclude":true,
"selected":false,
"value":"6CA28FC49A4C49A9A96",
"cat":"medallion"
} ….
{
"extraSeries":[
],
"label":"trip_time_in_secs",
"field":"trip_time_in_secs",
"counts":[
{
"from":"0",
"to":"10",
"selected":false,
"value":527,
"field":"trip_time_in_secs",
"exclude":true
} ...
{
"field":"passenger_count",
"counts":[
{
"count":74766,
"exclude":true,
"selected":false,
"value":"1",
"cat":"passenger_count"
} ...](https://image.slidesharecdn.com/june11310pmclouderarigaux-150629222838-lva1-app6891/85/Interactive-Query-and-Search-for-your-Big-Data-15-320.jpg)
Interactive Query and Search for your Big Data
- 1. INTERACTIVELY QUERY AND SEARCH YOUR BIG DATA Romain Rigaux
- 2. GOALS Build a Web app Quickly explore data … with Solr make Solr / Hadoop easier to use +
- 3. ARCHITECTURE “Just a view” on top of the standard Solr API REST
- 6. ARCHITECTURE NEXT! Lot of learning, UX Boost needed Simple, don’t know it is Solr
- 10. ARCHITECTURE /select /admin/collections /get /luke... /add_widget /zoom_in /select_facet /select_range... REST AJAX Templates + JS Model www….
- 11. ARCHITECTURE UI FOR FACETS Query Collection Layout All the 2D positioning (cell ids), visual, drag&drop Dashboard, fields, template, widgets (ids) Search terms, selected facets (q, fqs)
- 12. ADDING A WIDGET LIFECYCLE Load the initial page Edit mode and Drag&Drop /solr/zookeeper/clusterstate.json /solr/admin/luke… /get_collection
- 13. ADDING A WIDGET LIFECYCLE /solr/select?stats=true /new_facet Select the field Guess ranges (number or dates) Rounding (number or dates)
- 14. ADDING A WIDGET LIFECYCLE Query part 1 Query Part 2 Augment Solr response facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000& f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10 q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000] { 'facet_counts':{ 'facet_ranges':{ 'bytes':{ 'start':10000, 'counts':[ '900000', 3423, '1800000', 339, ... ] } } } { ..., 'normalized_facets':[ { 'extraSeries':[ ], 'label':'bytes', 'field':'bytes', 'counts':[ { 'from’:'900000', 'to':'1800000', 'selected':True, 'value':3423, 'field’:'bytes', 'exclude':False } ], ... } } }
- 15. JSON TO WIDGET { "field":"rate_code", "counts":[ { "count":97797, "exclude":true, "selected":false, "value":"1", "cat":"rate_code" } ... { "field":"medallion", "counts":[ { "count":159, "exclude":true, "selected":false, "value":"6CA28FC49A4C49A9A96", "cat":"medallion" } …. { "extraSeries":[ ], "label":"trip_time_in_secs", "field":"trip_time_in_secs", "counts":[ { "from":"0", "to":"10", "selected":false, "value":527, "field":"trip_time_in_secs", "exclude":true } ... { "field":"passenger_count", "counts":[ { "count":74766, "exclude":true, "selected":false, "value":"1", "cat":"passenger_count" } ...
- 16. REPEAT UNTIL…
- 17. GAME CHANGER! Possibilihes 5.1 / 5.2 Analyhc Facets
- 18. FACET FUNCTIONS Count Sum Avg Percentile Max ... Count(id) Sum(bytes) Avg(mul(price, quantity)) Percentile(salary, 50, 90) Max(temperature) ...
- 19. FACET FUNCTIONS
- 20. SUB “NESTED” FACETS top_os { type: term, field: os, limit: 5 } top_os { type: term, field: os, limit: 5, facet : { by_country: { type: term, field: country } } }
- 21. FUNCTION + NESTED = ANALYTICS states { type: term, field: state, facet : { by_month : { type: range, field: time, start: “TODAY-‐6MONTHS”, end: “TODAY”, gap: “MONTH”, facet : { avg_sal: “avg(salary)” } } } } states { type: term, field: state, facet : { avg_sal: “avg(salary)” } }
- 22. OPERATIONS ON BUCKETS OF DATA Counts → Functions
- 23. OPERATIONS ON BUCKETS OF DATA Nested → nD functions
- 24. ENTERPRISE FEATURES - Access to Search App configurable, LDAP/SAML auths - Share by link - Solr Cloud (or non Cloud) - Proxy user /solr/jobs_demo/select?user.name=hue&doAs=romain&q= - Security Kerberos - Sentry Collection level, Solr calls like /admin, /query, Solr UI, ZooKeeper
- 25. SEARCH AS ONLY APP IN HUE gethue.com/solr-‐search-‐ui-‐only/
- 26. • Spark in your browser • Notebooks • New REST Server SPARK INDEXING WHAT
- 27. • Open source REST for Spark Shell • Runs locally or inside YARN • Spark Scala, PySpark and jar/py submission SPARK INDEXING WHAT hsps://github.com/cloudera/hue/tree/master/apps/spark/java
- 28. SPARK STREAMING Real hme! Spark Solr
- 29. • Pytho • Scala • Charts NOTEBOOKS / SHELL WHAT
- 30. DEMO TIME • Analyze Bay area bike share • Visualize one year of data • Know your users, predict behavior
- 32. • Full Analyhcs • Easier indexing • Geo • Export/Share results • “More like this” • Solr Joins, Solr SQL • Spark, SQL... integrahon, Hue 4 WHAT’S NEXT NEW FEATURES