SparkSQL et Cassandra - Tool In Action Devoxx 2015
Download as PPT, PDF1 like1,395 views
This document discusses the integration and use of Spark SQL with Cassandra for data analysis, highlighting key concepts like Resilient Distributed Datasets (RDDs) and various stages of data processing. It includes practical code snippets for SQL queries, data transformations in Scala and Python, and the use of DataStax connectors. The overall goal is to analyze and manage conference talk data from Devoxx using efficient techniques that leverage both Spark and Cassandra capabilities.
![@alexanderDeja @maxospiquante#TiaSparkSQL
val rddTalk = cc.sql("select annee, titre, speakers, type_talk
from devoxx.talk")
// On sort de SparkSQL pour retravailler les données
val splitBySpeakersRdd =
rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]])
.map(m => (m,r) ))
case class Talk(titre: String, speaker: String, annee: Int, type_talk: String)
val talksSchemaRdd = splitBySpeakersRdd.map(
t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3)))
talksSchemaRdd.registerTempTable("talks_par_speaker")
Code Scala](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-15-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
val rddTalk = cc.sql("select annee, titre, speakers, type_talk
from devoxx.talk")
// On sort de SparkSQL pour retravailler les données
val splitBySpeakersRdd =
rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]])
.map(m => (m,r) ))
case class Talk(titre: String, speaker: String, annee: Int, type_talk: String)
val talksSchemaRdd = splitBySpeakersRdd.map(
t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3)))
talksSchemaRdd.registerTempTable("talks_par_speaker")
Code Scala](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-16-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
val rddTalk = cc.sql("select annee, titre, speakers, type_talk
from devoxx.talk")
// On sort de SparkSQL pour retravailler les données
val splitBySpeakersRdd =
rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]])
.map(m => (m,r) ))
case class Talk(titre: String, speaker: String, annee: Int, type_talk: String)
val talksSchemaRdd = splitBySpeakersRdd.map(
t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3)))
talksSchemaRdd.registerTempTable("talks_par_speaker")
Code Scala](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-17-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
val rddTalk = cc.sql("select annee, titre, speakers, type_talk
from devoxx.talk")
// On sort de SparkSQL pour retravailler les données
val splitBySpeakersRdd =
rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]])
.map(m => (m,r) ))
case class Talk(titre: String, speaker: String, annee: Int, type_talk: String)
val talksSchemaRdd = splitBySpeakersRdd.map(
t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3)))
talksSchemaRdd.registerTempTable("talks_par_speaker")
Code Scala](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-18-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
Code Scala : insertion Cassandra
val connector = CassandraConnector(sc.getConf)
talksSchemaRdd.foreachPartition(partition => {
connector.withSessionDo{ session =>
partition.foreach(r => session.execute(
"UDPATE devoxx.talk_par_speaker USING TTL ? " +
set type_talk=?, titre=?, annee=? " +
WHERE id_speaker = ?"),
86400, r.type_talk, r.titre,
r.annee.asInstanceOf[java.lang.Integer],r.speaker)
)}
})](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-20-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
def split_keywords(row):
## fonction splittant les titres par mot
rddTalk = sqlContext.sql("SELECT titre, speakers, annee, categorie, type_talk FROM devoxx.talk")
splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r))
splitByKeywordRdd_schema = sqlContext.inferSchema(
splitByKeywordRdd.filter(lambda word:len(word[0])>1)
.map(lambda x:Row(keyword=x[0],annee=x[1])))
splitByKeywordRdd_schema.registerTempTable("tmp_keywords")
keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb
FROM tmp_keywords
GROUP BY keyword, annee""")
keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...)))
keyword_count_schema.registerTempTable("tmp_keywords_count")
sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb
FROM tmp_keywords_count""")
Code Python](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-31-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
def split_keywords(row):
## fonction splittant les titres par mot
rddTalk = sqlContext.sql("select titre, speakers, annee, categorie, type_talk from devoxx.talk")
splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r))
splitByKeywordRdd_schema = sqlContext.inferSchema(
splitByKeywordRdd.filter(lambda word:len(word[0])>1)
.map(lambda x:Row(keyword=x[0],annee=x[1])))
splitByKeywordRdd_schema.registerTempTable("tmp_keywords")
keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb
FROM tmp_keywords
GROUP BY keyword, annee""")
keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...)))
keyword_count_schema.registerTempTable("tmp_keywords_count")
sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb
FROM tmp_keywords_count""")
Code Python](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-32-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
def split_keywords(row):
## fonction splittant les titres par mot
rddTalk = sqlContext.sql("select titre, speakers, annee, categorie, type_talk from devoxx.talk")
splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r))
splitByKeywordRdd_schema = sqlContext.inferSchema(
splitByKeywordRdd.filter(lambda word:len(word[0])>1)
.map(lambda x:Row(keyword=x[0],annee=x[1])))
splitByKeywordRdd_schema.registerTempTable("tmp_keywords")
keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb
FROM tmp_keywords
GROUP BY keyword, annee""")
keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...)))
keyword_count_schema.registerTempTable("tmp_keywords_count")
sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb
FROM tmp_keywords_count""")
Code Python](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-33-320.jpg)
![@alexanderDeja @maxospiquante#TiaSparkSQL
def split_keywords(row):
## fonction splittant les titres par mot
rddTalk = sqlContext.sql("select titre, speakers, annee, categorie, type_talk from devoxx.talk")
splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r))
splitByKeywordRdd_schema = sqlContext.inferSchema(
splitByKeywordRdd.filter(lambda word:len(word[0])>1)
.map(lambda x:Row(keyword=x[0],annee=x[1])))
splitByKeywordRdd_schema.registerTempTable("tmp_keywords")
keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb
FROM tmp_keywords
GROUP BY keyword, annee""")
keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...)))
keyword_count_schema.registerTempTable("tmp_keywords_count")
sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb
FROM tmp_keywords_count""")
Code Python](https://image.slidesharecdn.com/pres-tia-150410031126-conversion-gate01/85/SparkSQL-et-Cassandra-Tool-In-Action-Devoxx-2015-34-320.jpg)
SparkSQL et Cassandra - Tool In Action Devoxx 2015
- 1. @alexanderDeja @maxospiquante#TiaSparkSQL SparkSQL pour analyser vos données Cassandra
- 2. @alexanderDeja @maxospiquante#TiaSparkSQL Qui sommes-nous ? Alexander DEJANOVSKI @alexanderDeja Développeur Maxence LECOINTE @maxospiquante Développeur
- 3. @alexanderDeja @maxospiquante#TiaSparkSQL Cassandra • Base NoSQL distribuée • Langage de requête : CQL~=SQL • SELECT * FROM ze_table WHERE ze_key=1 • Pas de jointure, pas de group by, pas d’insert/select
- 4. @alexanderDeja @maxospiquante#TiaSparkSQL Spark • Map/Reduce en mémoire • 10x-100x plus rapide que Hadoop • Scala, Java ou Python • Modules : Spark Streaming, MLlib, GraphX, SparkSQL
- 5. @alexanderDeja @maxospiquante#TiaSparkSQL Objectif Cassandra << >> SparkSQL Création de tables d’index Calcul de statistiques (simples…) sur les confs Devoxx FR de 2012 à 2015
- 6. @alexanderDeja @maxospiquante#TiaSparkSQL Datastax Spark Cassandra Connector
- 7. @alexanderDeja @maxospiquante#TiaSparkSQL Setup • Spark 1.1 ou 1.2 pour Scala et Java • Connecteur Datastax : http://github.com/datastax/spark-cassandra-connector • Spark 1.1 pour Python • Connecteur Calliope de TupleJump : http://tuplejump.github.io/calliope/start-with-sql.html
- 8. @alexanderDeja @maxospiquante#TiaSparkSQL Pour vous éviter (certaines) galères… • Sources de ce TIA : https://github.com/adejanovski/devoxx2015 • Lisez le README
- 9. @alexanderDeja @maxospiquante#TiaSparkSQL C’est quoi un RDD ? • Resilient Distributed Dataset • Collection d’objets distribuée et résiliente • Permet le stockage de n’importe quel format de donnée
- 14. @alexanderDeja @maxospiquante#TiaSparkSQL Scala-Fu : split par speaker
- 15. @alexanderDeja @maxospiquante#TiaSparkSQL val rddTalk = cc.sql("select annee, titre, speakers, type_talk from devoxx.talk") // On sort de SparkSQL pour retravailler les données val splitBySpeakersRdd = rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]]) .map(m => (m,r) )) case class Talk(titre: String, speaker: String, annee: Int, type_talk: String) val talksSchemaRdd = splitBySpeakersRdd.map( t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3))) talksSchemaRdd.registerTempTable("talks_par_speaker") Code Scala
- 16. @alexanderDeja @maxospiquante#TiaSparkSQL val rddTalk = cc.sql("select annee, titre, speakers, type_talk from devoxx.talk") // On sort de SparkSQL pour retravailler les données val splitBySpeakersRdd = rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]]) .map(m => (m,r) )) case class Talk(titre: String, speaker: String, annee: Int, type_talk: String) val talksSchemaRdd = splitBySpeakersRdd.map( t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3))) talksSchemaRdd.registerTempTable("talks_par_speaker") Code Scala
- 17. @alexanderDeja @maxospiquante#TiaSparkSQL val rddTalk = cc.sql("select annee, titre, speakers, type_talk from devoxx.talk") // On sort de SparkSQL pour retravailler les données val splitBySpeakersRdd = rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]]) .map(m => (m,r) )) case class Talk(titre: String, speaker: String, annee: Int, type_talk: String) val talksSchemaRdd = splitBySpeakersRdd.map( t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3))) talksSchemaRdd.registerTempTable("talks_par_speaker") Code Scala
- 18. @alexanderDeja @maxospiquante#TiaSparkSQL val rddTalk = cc.sql("select annee, titre, speakers, type_talk from devoxx.talk") // On sort de SparkSQL pour retravailler les données val splitBySpeakersRdd = rddTalk.flatMap(r => (r(2).asInstanceOf[scala.collection.immutable.Set[String]]) .map(m => (m,r) )) case class Talk(titre: String, speaker: String, annee: Int, type_talk: String) val talksSchemaRdd = splitBySpeakersRdd.map( t =>Talk(t._2.getString(1),t._1,t._2.getInt(0),t._2.getString(1),t._2.getString(3))) talksSchemaRdd.registerTempTable("talks_par_speaker") Code Scala
- 19. @alexanderDeja @maxospiquante#TiaSparkSQL cc.sql("insert into devoxx.talk_par_speaker select speaker, type_talk, titre, annee from talks_par_speaker").collect() Code Scala : insertion Cassandra
- 20. @alexanderDeja @maxospiquante#TiaSparkSQL Code Scala : insertion Cassandra val connector = CassandraConnector(sc.getConf) talksSchemaRdd.foreachPartition(partition => { connector.withSessionDo{ session => partition.foreach(r => session.execute( "UDPATE devoxx.talk_par_speaker USING TTL ? " + set type_talk=?, titre=?, annee=? " + WHERE id_speaker = ?"), 86400, r.type_talk, r.titre, r.annee.asInstanceOf[java.lang.Integer],r.speaker) )} })
- 24. @alexanderDeja @maxospiquante#TiaSparkSQL SchemaRDD nbTalkParSpeaker = cassandraSQLContext.sql( “SELECT B.nom_speaker as nom_speaker, A.annee as annee, “A.id_speaker as id_speaker “ + “FROM devoxx.talk_par_speaker A JOIN devoxx.speakers B “+ “ON A.id_speaker = B.id_speaker "); nbTalkParSpeaker.registerTempTable(“tmp_talk_par_speaker"); cassandraSQLContext.sql( “INSERT INTO devoxx.speaker_par_annee “ + “SELECT nom_speaker, annee, count(*) as nb “+ “FROM tmp_talk_par_speaker group by nom_speaker, annee").collect(); Code Java
- 25. @alexanderDeja @maxospiquante#TiaSparkSQL SchemaRDD nbTalkParSpeaker = cassandraSQLContext.sql( “SELECT B.nom_speaker as nom_speaker, A.annee as annee, “A.id_speaker as id_speaker “ + “FROM devoxx.talk_par_speaker A JOIN devoxx.speakers B “+ “ON A.id_speaker = B.id_speaker "); nbTalkParSpeaker.registerTempTable(“tmp_talk_par_speaker"); cassandraSQLContext.sql( “INSERT INTO devoxx.speaker_par_annee “ + “SELECT nom_speaker, annee, count(*) as nb “+ “FROM tmp_talk_par_speaker group by nom_speaker, annee").collect(); Code Java
- 26. @alexanderDeja @maxospiquante#TiaSparkSQL SchemaRDD nbTalkParSpeaker = cassandraSQLContext.sql( “SELECT B.nom_speaker as nom_speaker, A.annee as annee, “A.id_speaker as id_speaker “ + “FROM devoxx.talk_par_speaker A JOIN devoxx.speakers B “+ “ON A.id_speaker = B.id_speaker "); nbTalkParSpeaker.registerTempTable(“tmp_talk_par_speaker"); cassandraSQLContext.sql( “INSERT INTO devoxx.speaker_par_annee “ + “SELECT nom_speaker, annee, count(*) as nb, id_speaker “+ “FROM tmp_talk_par_speaker “+ “GROUP BY nom_speaker, annee, id_speaker").collect(); Code Java
- 27. @alexanderDeja @maxospiquante#TiaSparkSQL ./spark-submit --class devoxx.Devoxx….. --master spark://127.0.0.1:7077 devoxxSparkSql.jar Submit Java
- 31. @alexanderDeja @maxospiquante#TiaSparkSQL def split_keywords(row): ## fonction splittant les titres par mot rddTalk = sqlContext.sql("SELECT titre, speakers, annee, categorie, type_talk FROM devoxx.talk") splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r)) splitByKeywordRdd_schema = sqlContext.inferSchema( splitByKeywordRdd.filter(lambda word:len(word[0])>1) .map(lambda x:Row(keyword=x[0],annee=x[1]))) splitByKeywordRdd_schema.registerTempTable("tmp_keywords") keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb FROM tmp_keywords GROUP BY keyword, annee""") keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...))) keyword_count_schema.registerTempTable("tmp_keywords_count") sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb FROM tmp_keywords_count""") Code Python
- 32. @alexanderDeja @maxospiquante#TiaSparkSQL def split_keywords(row): ## fonction splittant les titres par mot rddTalk = sqlContext.sql("select titre, speakers, annee, categorie, type_talk from devoxx.talk") splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r)) splitByKeywordRdd_schema = sqlContext.inferSchema( splitByKeywordRdd.filter(lambda word:len(word[0])>1) .map(lambda x:Row(keyword=x[0],annee=x[1]))) splitByKeywordRdd_schema.registerTempTable("tmp_keywords") keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb FROM tmp_keywords GROUP BY keyword, annee""") keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...))) keyword_count_schema.registerTempTable("tmp_keywords_count") sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb FROM tmp_keywords_count""") Code Python
- 33. @alexanderDeja @maxospiquante#TiaSparkSQL def split_keywords(row): ## fonction splittant les titres par mot rddTalk = sqlContext.sql("select titre, speakers, annee, categorie, type_talk from devoxx.talk") splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r)) splitByKeywordRdd_schema = sqlContext.inferSchema( splitByKeywordRdd.filter(lambda word:len(word[0])>1) .map(lambda x:Row(keyword=x[0],annee=x[1]))) splitByKeywordRdd_schema.registerTempTable("tmp_keywords") keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb FROM tmp_keywords GROUP BY keyword, annee""") keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...))) keyword_count_schema.registerTempTable("tmp_keywords_count") sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb FROM tmp_keywords_count""") Code Python
- 34. @alexanderDeja @maxospiquante#TiaSparkSQL def split_keywords(row): ## fonction splittant les titres par mot rddTalk = sqlContext.sql("select titre, speakers, annee, categorie, type_talk from devoxx.talk") splitByKeywordRdd = rddTalk.flatMap(lambda r:split_keywords(r)) splitByKeywordRdd_schema = sqlContext.inferSchema( splitByKeywordRdd.filter(lambda word:len(word[0])>1) .map(lambda x:Row(keyword=x[0],annee=x[1]))) splitByKeywordRdd_schema.registerTempTable("tmp_keywords") keyword_count = sqlContext.sql("""SELECT keyword, annee, count(*) as nb FROM tmp_keywords GROUP BY keyword, annee""") keyword_count_schema = sqlContext.inferSchema(keyword_count.map(lambda x:Row(...))) keyword_count_schema.registerTempTable("tmp_keywords_count") sqlContext.sql("""INSERT INTO devoxx.keyword_par_annee SELECT keyword, annee, nb FROM tmp_keywords_count""") Code Python
- 36. @YourTwitterHandle@YourTwitterHandle@alexanderDeja @maxospiquante#TiaSparkSQL Et voilà ! Des questions ?