Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar
26 likes10,424 views
This document describes building a REST job server for interactive Spark as a service using Livy. It discusses the history and challenges of running Spark jobs in Hue, introduces Livy as a Spark submission server, and details its architecture for running Spark interactively in local or YARN cluster mode. The document also covers how Livy uses interpreters to run code in different languages and enable magics for functions like JSON output, tables, and plotting.
![SESSION CREATION AND EXECUTION
% curl -XPOST localhost:8998/sessions
-d '{"kind": "spark"}'
{
"id": 0,
"kind": "spark",
"log": [...],
"state": "idle"
}
% curl -XPOST localhost:8998/sessions/0/statements -d '{"code": "1+1"}'
{
"id": 0,
"output": {
"data": { "text/plain": "res0: Int = 2" },
"execution_count": 0,
"status": "ok"
},
"state": "available"
}](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-32-320.jpg)
![[('', 506610), ('the', 23407), ('I', 19540)... ]
JSON MAGIC
> counts
sparkIMain.valueOfTerm(“counts”)
.toJson()
Interpreter
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%json counts](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-51-320.jpg)
![JSON MAGIC
> counts
sparkIMain.valueOfTerm(“counts”)
.toJson()
Interpreter
{
"id": 0,
"output": {
"application/json": [
{ "count": 506610, "word": "" },
{ "count": 23407, "word": "the" },
{ "count": 19540, "word": "I" },
...
]
...
}
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%json counts](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-52-320.jpg)
![[('', 506610), ('the', 23407), ('I', 19540)... ]
TABLE MAGIC
> counts
Interpreter
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%table counts
sparkIMain.valueOfTerm(“counts”)
.guessHeaders().toList()](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-53-320.jpg)
![TABLE MAGIC
> counts
sparkIMain.valueOfTerm(“counts”)
.guessHeaders().toList()
Interpreter
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%table counts
"application/vnd.livy.table.v1+json": {
"headers": [
{ "name": "count", "type": "BIGINT_TYPE" },
{ "name": "name", "type": "STRING_TYPE" }
],
"data": [
[ 23407, "the" ],
[ 19540, "I" ],
[ 18358, "and" ],
...
]
}](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-54-320.jpg)
![PySpark shell
RDD
Shell
Python
Shell
r = sc.parallelize([])
srdd = ShareableRdd(r)](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-74-320.jpg)
![PySpark shell
RDD
{'ak': 'Alaska'}
{'ca': 'California'}
Shell
Python
Shell
r = sc.parallelize([])
srdd = ShareableRdd(r)](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-75-320.jpg)
![PySpark shell
RDD
{'ak': 'Alaska'}
{'ca': 'California'}
Shell
Python
Shell
curl -XPOST /sessions/0/statement {
'code': srdd.get('ak')
}
r = sc.parallelize([])
srdd = ShareableRdd(r)](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-76-320.jpg)
![PySpark shell
RDD
{'ak': 'Alaska'}
{'ca': 'California'}
Shell
Python
Shell
states = SharedRdd('host/sessions/0', 'srdd')
states.get('ak')
r = sc.parallelize([])
srdd = ShareableRdd(r)
curl -XPOST /sessions/0/statement {
'code': srdd.get('ak')
}](https://image.slidesharecdn.com/05romainrigaux-151104001627-lva1-app6891/85/Building-a-REST-Job-Server-for-interactive-Spark-as-a-service-by-Romain-Rigaux-and-Erick-Tryzelaar-77-320.jpg)
Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar
- 1. BUILDING A REST JOB SERVER FOR INTERACTIVE SPARK AS A SERVICE Romain Rigaux - Cloudera Erick Tryzelaar - Cloudera
- 2. WHY?
- 5. NOTEBOOKS EASY ACCESS FROM ANYWHERE SHARE SPARK CONTEXTS AND RDDs BUILD APPS SPARK MAGIC … WHY SPARK AS A SERVICE?
- 7. HISTORY V1: OOZIE • It works • Code snippet THE GOOD • Submit through Oozie • Shell ac:on • Very Slow • Batch THE BAD workflow.xml snippet.py stdout
- 8. HISTORY V2: SPARK IGNITER • It works beAer THE GOOD • Compiler Jar • Batch only, no shell • No Python, R • Security • Single point of failure THE BAD Compile Implement Upload json output Batch Scala jar Ooyala
- 9. HISTORY V3: NOTEBOOK • Like spark-submit / spark shells • Scala / Python / R shells • Jar / Python batch Jobs • Notebook UI • YARN THE GOOD • Beta? THE BAD Livy code snippet batch
- 14. ARCHITECTURE • Standard web service: wrapper around spark-submit / Spark shells • YARN mode, Spark drivers run inside the cluster (supports crashes) • No need to inherit any interface or compile code • Extended to work with additional backends
- 15. LIVY WEB SERVER ARCHITECTURE LOCAL “DEV” MODE YARN MODE
- 32. SESSION CREATION AND EXECUTION % curl -XPOST localhost:8998/sessions -d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" } % curl -XPOST localhost:8998/sessions/0/statements -d '{"code": "1+1"}' { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }
- 39. INTERPRETERS • Pipe stdin/stdout to a running shell • Execute the code / send to Spark workers • Perform magic opera:ons • One interpreter per language • “Swappable” with other kernels (python, spark..) Interpreter > println(1 + 1) 2 println(1 + 1) 2
- 49. INTERPRETER MAGIC • table • json • plotting • ...
- 60. • Pluggable Backends • Livy's Spark Backends – Scala – pyspark – R • IPython / Jupyter support coming soon PLUGGABLE INTERPRETERS
- 61. • Re-using it • Generic Framework for Interpreters • 51 Kernels JUPYTER BACKEND
- 62. SPARK AS A SERVICE
- 71. SHARING RDDS
- 79. • SSL Support • Persistent Sessions • Kerberos SECURITY