Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
1 like2,125 views
The document discusses the development of a web application designed to enhance the usability of Solr and Hadoop, focusing on features like user history management, a user-friendly interface, and the addition of analytic capabilities. It includes details on UI layout, widget management, and facet functions for data analysis, alongside enterprise features like security and authentication. The document also touches on integration with Spark for advanced analytics and indexing functionalities.
![ADDING A WIDGET
LIFECYCLE
Query
part
1
Query
Part
2
Augment
Solr
response
facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000&
f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10
q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000]
{
'facet_counts':{
'facet_ranges':{
'bytes':{
'start':10000,
'counts':[
'900000',
3423,
'1800000',
339,
...
]
}
}
}
{
...,
'normalized_facets':[
{
'extraSeries':[
],
'label':'bytes',
'field':'bytes',
'counts':[
{
'from’:'900000',
'to':'1800000',
'selected':True,
'value':3423,
'field’:'bytes',
'exclude':False
}
], ...
}
}
}](https://image.slidesharecdn.com/hadoop-summit-interactive-big-data-analysis-solr-hue-spark-150612180719-lva1-app6892/85/Hadoop-Summit-Interactive-Big-Data-Analysis-with-Solr-Spark-and-Hue-14-320.jpg)
![JSON TO WIDGET
{
"field":"rate_code",
"counts":[
{
"count":97797,
"exclude":true,
"selected":false,
"value":"1",
"cat":"rate_code"
} ...
{
"field":"medallion",
"counts":[
{
"count":159,
"exclude":true,
"selected":false,
"value":"6CA28FC49A4C49A9A96",
"cat":"medallion"
} ….
{
"extraSeries":[
],
"label":"trip_time_in_secs",
"field":"trip_time_in_secs",
"counts":[
{
"from":"0",
"to":"10",
"selected":false,
"value":527,
"field":"trip_time_in_secs",
"exclude":true
} ...
{
"field":"passenger_count",
"counts":[
{
"count":74766,
"exclude":true,
"selected":false,
"value":"1",
"cat":"passenger_count"
} ...](https://image.slidesharecdn.com/hadoop-summit-interactive-big-data-analysis-solr-hue-spark-150612180719-lva1-app6892/85/Hadoop-Summit-Interactive-Big-Data-Analysis-with-Solr-Spark-and-Hue-15-320.jpg)
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
- 1. INTERACTIVELY QUERY AND SEARCH YOUR BIG DATA Romain Rigaux
- 2. GOALS Build a Web app Quickly explore data … with Solr make Solr / Hadoop easier to use +
- 3. ARCHITECTURE “Just a view” on top of the standard Solr API REST
- 6. ARCHITECTURE NEXT! Lot of learning, UX Boost needed Simple, don’t know it is Solr
- 10. ARCHITECTURE /select /admin/collections /get /luke... /add_widget /zoom_in /select_facet /select_range... REST AJAX Templates + JS Model www….
- 11. ARCHITECTURE UI FOR FACETS Query Collection Layout All the 2D positioning (cell ids), visual, drag&drop Dashboard, fields, template, widgets (ids) Search terms, selected facets (q, fqs)
- 12. ADDING A WIDGET LIFECYCLE Load the initial page Edit mode and Drag&Drop /solr/zookeeper/clusterstate.json /solr/admin/luke… /get_collection
- 13. ADDING A WIDGET LIFECYCLE /solr/select?stats=true /new_facet Select the field Guess ranges (number or dates) Rounding (number or dates)
- 14. ADDING A WIDGET LIFECYCLE Query part 1 Query Part 2 Augment Solr response facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000& f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10 q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000] { 'facet_counts':{ 'facet_ranges':{ 'bytes':{ 'start':10000, 'counts':[ '900000', 3423, '1800000', 339, ... ] } } } { ..., 'normalized_facets':[ { 'extraSeries':[ ], 'label':'bytes', 'field':'bytes', 'counts':[ { 'from’:'900000', 'to':'1800000', 'selected':True, 'value':3423, 'field’:'bytes', 'exclude':False } ], ... } } }
- 15. JSON TO WIDGET { "field":"rate_code", "counts":[ { "count":97797, "exclude":true, "selected":false, "value":"1", "cat":"rate_code" } ... { "field":"medallion", "counts":[ { "count":159, "exclude":true, "selected":false, "value":"6CA28FC49A4C49A9A96", "cat":"medallion" } …. { "extraSeries":[ ], "label":"trip_time_in_secs", "field":"trip_time_in_secs", "counts":[ { "from":"0", "to":"10", "selected":false, "value":527, "field":"trip_time_in_secs", "exclude":true } ... { "field":"passenger_count", "counts":[ { "count":74766, "exclude":true, "selected":false, "value":"1", "cat":"passenger_count" } ...
- 16. REPEAT UNTIL…
- 17. GAME CHANGER! Possibilihes 5.1 / 5.2 Analyhc Facets
- 18. FACET FUNCTIONS Count Sum Avg Percentile Max ... Count(id) Sum(bytes) Avg(mul(price, quantity)) Percentile(salary, 50, 90) Max(temperature) ...
- 19. FACET FUNCTIONS
- 20. SUB “NESTED” FACETS top_os { type: term, field: os, limit: 5 } top_os { type: term, field: os, limit: 5, facet : { by_country: { type: term, field: country } } }
- 21. FUNCTION + NESTED = ANALYTICS states { type: term, field: state, facet : { by_month : { type: range, field: time, start: “TODAY-‐6MONTHS”, end: “TODAY”, gap: “MONTH”, facet : { avg_sal: “avg(salary)” } } } } states { type: term, field: state, facet : { avg_sal: “avg(salary)” } }
- 22. OPERATIONS ON BUCKETS OF DATA Counts → Functions
- 23. OPERATIONS ON BUCKETS OF DATA Nested → nD functions
- 24. ENTERPRISE FEATURES - Access to Search App configurable, LDAP/SAML auths - Share by link - Solr Cloud (or non Cloud) - Proxy user /solr/jobs_demo/select?user.name=hue&doAs=romain&q= - Security Kerberos - Sentry Collection level, Solr calls like /admin, /query, Solr UI, ZooKeeper
- 25. SEARCH AS ONLY APP IN HUE gethue.com/solr-‐search-‐ui-‐only/
- 26. • Spark in your browser • Notebooks • New REST Server SPARK INDEXING WHAT
- 27. • Open source REST for Spark Shell • Runs locally or inside YARN • Spark Scala, PySpark and jar/py submission SPARK INDEXING WHAT hsps://github.com/cloudera/hue/tree/master/apps/spark/java
- 28. SPARK STREAMING Real hme! Spark Solr
- 29. • Pytho • Scala • Charts NOTEBOOKS / SHELL WHAT
- 30. DEMO TIME • Analyze Bay area bike share • Visualize one year of data • Know your users, predict behavior
- 32. • Full Analyhcs • Easier indexing • Geo • Export/Share results • “More like this” • Solr Joins, Solr SQL • Spark, SQL... integrahon, Hue 4 WHAT’S NEXT NEW FEATURES