![#4: not a broadcast join
scala> iotData.join(zipCodes).where($”zip" === $"zip").explain
== Physical Plan ==
*SortMergeJoin [zip2#17], [zip#9], Inner
:- *Sort [zip2#17 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(zip2#17, 200)](https://image.slidesharecdn.com/javaone2018-fire-fightingjavabigdataproblems-181024051931/85/Fire-fighting-java-big-data-problems-30-320.jpg)
![#4: broadcast join
scala> iotData.join(broadcast(zipCodes)).where($”zip” ===
$"zip").explain
== Physical Plan ==
*BroadcastHashJoin [zip2#17], [zip#9], Inner, BuildRight](https://image.slidesharecdn.com/javaone2018-fire-fightingjavabigdataproblems-181024051931/85/Fire-fighting-java-big-data-problems-31-320.jpg)
![Schema
{
"type": "record",
"name": "IotData",
"fields" : [
{"name": "zip", “type": "string"},
{"name": "status", "type": "long"},
{"name": "created", "type": “long"}
]
}](https://image.slidesharecdn.com/javaone2018-fire-fightingjavabigdataproblems-181024051931/85/Fire-fighting-java-big-data-problems-40-320.jpg)
![Schema update
{
"type": "record",
"name": “IotData",
"fields" : [
{"name": "status", "type": "long"},
{"name": “zip", "type": "string"},
{"name": "created", "type": "long"}
]
}
Swap fields](https://image.slidesharecdn.com/javaone2018-fire-fightingjavabigdataproblems-181024051931/85/Fire-fighting-java-big-data-problems-41-320.jpg)
![Fields are serialized in schema order
4
48
2
52
51
52
57
10
field=“zip”, contents = “94043”
field=“status”, contents = 1L
field=“created”, contents = 2L
bytes
{
"type": "record",
"name": "IotData",
"fields" : [
{"name": “zip", “type": "string"},
{"name": "status", "type": "long"},
{"name": "created", "type": “long"}
]
}](https://image.slidesharecdn.com/javaone2018-fire-fightingjavabigdataproblems-181024051931/85/Fire-fighting-java-big-data-problems-45-320.jpg)
![It could have been worse…
{
"type": "record",
"name": "IotData",
"fields" : [
{"name": "status", "type": "long"},
{"name": "created", "type": “long"}
]
}
{
"type": "record",
"name": "IotData",
"fields" : [
{"name": "created", "type": “long”},
{"name": “status", "type": "long"}
]
}
the following schema change wouldn’t have
resulted in any runtime errors …](https://image.slidesharecdn.com/javaone2018-fire-fightingjavabigdataproblems-181024051931/85/Fire-fighting-java-big-data-problems-55-320.jpg)
Fire-fighting java big data problems
- 1. Fire-fighting Java big data problems
- 3. WHOOPS
- 4. Format Look at a real-life fire Understand root cause Put out the fire Avoiding the fire
- 7. Sprint 14:18 100% Back The Boss More My query has been running for over 4 hours!!! Our CEO needs this data right now…
- 8. 10110101010010 10001001101111 10110101010010 10001001101111 10110101010010 10001001101111 10110101010010 10001001101111 zip code, car status Distributed FS
- 9. Spark Storage general-purpose distributed data processing distributed storage, e.g. HDFS, S3, Azure, …
- 15. Root cause
- 16. 94043,speeding 94043,stopped 94103,under_limit IoT data: zip code, car status 94043,Mountain View 94103,San Francisco Reference data: zip code, city 94043,under_limit 94043,stopped 94043,speeding IoT data: zip code, car status Shuffle (hash join)
- 18. Zip code Number of cars
- 19. Skew 94043,speeding 94043,stopped 94103,under_limit 94043,speeding,Mountain View 94043,stopped,Mountain View 94043,under_limit,Mountain View 94043,stopped,Mountain View 94043,speeding,Mountain View 94103,under_limit,San Francisco IoT data: zip code, car status 94043,Mountain View 94103,San Francisco Reference data: zip code, city 94043,under_limit 94043,stopped 94043,speeding IoT data: zip code, car status Shuffle (hash join)
- 22. The fix(es)
- 23. #1: understand your data
- 24. #2: filter-out the skew
- 25. #3: only include fields you need in the result before the join
- 26. #4: broadcast join JVM JVM 94043,Mountain View 94103,San Francisco 94043,Mountain View 94103,San Francisco 94043,Mountain View 94103,San Francisco 1) replicate the small dataset to each JVM, and cache in memory
- 27. #4: broadcast join JVM JVM 94043,Mountain View 94103,San Francisco 94043,Mountain View 94103,San Francisco 94043,Mountain View 94103,San Francisco 1) replicate the small dataset to each JVM, and cache in memory 94043,speeding 94043,stopped 94103,under_limit 94043,under_limit 94043,stopped 94043,speeding 2) process large data set and join with small in-memory data set
- 28. #4: broadcast join this explicitly tells Spark to do a broadcast join . . . Spark may automatically do this for you if one dataset is < 10MB (spark.sql.autoBroadcastJoinThreshold)
- 29. How do we know whether Spark is performing a hash or broadcast join?
- 30. #4: not a broadcast join scala> iotData.join(zipCodes).where($”zip" === $"zip").explain == Physical Plan == *SortMergeJoin [zip2#17], [zip#9], Inner :- *Sort [zip2#17 ASC NULLS FIRST], false, 0 : +- Exchange hashpartitioning(zip2#17, 200)
- 31. #4: broadcast join scala> iotData.join(broadcast(zipCodes)).where($”zip” === $"zip").explain == Physical Plan == *BroadcastHashJoin [zip2#17], [zip#9], Inner, BuildRight
- 32. #4: broadcast join By removing the shuffle, we remove the impact data skew has on our processing Only works if one data set fits entirely in memory
- 33. Other advanced options Filtered broadcast join Salting + filtered salting Bin packing
- 34. Takeaways Spark isn’t a database Check for skew - build join key histograms Heavily filter + project your data Avoid or minimize the network
- 35. Why can’t you just change?
- 36. Sprint 14:18 100% Back The Boss More Build me a dashboard to view the top zip codes in real-time.
- 37. Queue IoT front end Analytics DB Web app
- 38. Queue IoT front end Analytics DB Web app what data format do you use?
- 39. Queue IoT front end Analytics DB Web app Distributed FS
- 40. Schema { "type": "record", "name": "IotData", "fields" : [ {"name": "zip", “type": "string"}, {"name": "status", "type": "long"}, {"name": "created", "type": “long"} ] }
- 41. Schema update { "type": "record", "name": “IotData", "fields" : [ {"name": "status", "type": "long"}, {"name": “zip", "type": "string"}, {"name": "created", "type": "long"} ] } Swap fields
- 42. Sprint 14:18 100% Back Ops More The IoT front end deployment is complete.
- 43. Sprint 14:18 100% Back Ops More The analytics job is failing with an java.io.EOFException!
- 44. Root cause
- 45. Fields are serialized in schema order 4 48 2 52 51 52 57 10 field=“zip”, contents = “94043” field=“status”, contents = 1L field=“created”, contents = 2L bytes { "type": "record", "name": "IotData", "fields" : [ {"name": “zip", “type": "string"}, {"name": "status", "type": "long"}, {"name": "created", "type": “long"} ] }
- 46. Serialized fields are reordered if you reorder them in the schema 4 48 2 52 51 52 57 10 field=“zip”, contents = “94043” field=“status”, contents = 1L field=“created”, contents = 2L bytes 52 2 57 10 48 51 4 52 field=“status”, contents = 1L field=“zip”, contents = “94043” field=“created”, contents = 2L bytes “zip” and “status” reversed Original
- 47. Avro serialization Fields written in schema order Field names aren’t included Schema typically isn’t included
- 48. Avro deserialization Fields read in schema order Readers need the writers schema Writers schema isn’t typically in message
- 49. Queue IoT front end Analytics DB Web app Only the producer is updated Consumer doesn’t have the updated schema
- 50. The fix
- 51. Put out the fire … Update the consumer process with the new schema Skip over records you can’t process, or catch exception and attempt with old schema
- 52. Avoid the fire… Never change the schema Include the schema in each message Schema registry Use a different data format
- 53. Queue Application Analytics DB Confluent schema registry tag each message with a schema version push message schema to registry 1 fetch message schema from registry using tag 3 deserialize the message using the schema from the registry 4
- 54. Avro supported evolutions Deleting optional fields Re-ordering fields Changing field names https://avro.apache.org/docs/1.8.2/ spec.html#Schema+Resolution
- 55. It could have been worse… { "type": "record", "name": "IotData", "fields" : [ {"name": "status", "type": "long"}, {"name": "created", "type": “long"} ] } { "type": "record", "name": "IotData", "fields" : [ {"name": "created", "type": “long”}, {"name": “status", "type": "long"} ] } the following schema change wouldn’t have resulted in any runtime errors …
- 56. Takeaways Avro is brittle without a registry Understand how schema evolution works with your data format Have a strategy for schema changes Enforce schema evolution rules (Confluence schema registry does this) Test schema evolution in QA
- 58. Sprint 14:18 100% Back The Boss More Our customers are complaining that our portal isn’t showing current data…what’s going on?
- 59. Kafka Spark streaming Receiver Spark batches of data results pushed to external system DB
- 61. Root cause
- 62. Spike in data
- 63. By default, Spark streaming will perform an unbounded pull from “current” to “latest”
- 64. Which means that it may take a long time to catch up - if at all
- 65. The fix
- 66. Put out the fire … Skip data + restart from latest Give your application and database more resources*
- 67. Avoid the fire… Know your spike load and tune your cluster + DB to handle it Limit the amount of Kafka data pulled in each batch (spark.streaming.kafka.maxRatePerPartition) Automatically skip over spikes Over-provision or implement a lambda architecture
- 68. Takeaways Measure and alert on lag (wall clock - event time) Load-test beyond your expected max rate before going to production Have a strategy to handle unexpected spikes
- 69. I just wanted to take a look…
- 71. I wonder what the data looks like...
- 72. SELECT * FROM top_zips LIMIT 1;
- 74. Sprint 14:18 100% Back SRE More Our production database is down!!! *^?@!#
- 75. Root cause
- 76. Node Node Node perform a token scan Client
- 77. Flash back to my 2015 J1 talk… V VVV V VVVVV V KKKKKKKKKKK V VVV V VVVVV V KKKKKKKKKKK tombstone markers indicate that the column has been deleted deletes in Cassandra are soft; deleted columns are marked with tombstones these tombstoned columns slow-down reads
- 78. Node Read all the data, keeping deleted data in-memory until you find the first non-deleted record
- 79. Node Read all the data, keeping deleted data in-memory until you find the first record OOM!
- 80. The fix
- 81. Put out the fire … Bounce Cassandra Don’t run that query again
- 82. Avoid the fire … Don’t treat Cassandra as a OLAP database Don’t allow users to run arbitrary queries Instead use Spark, Splunk, or a relational/ OLAP database
- 83. Takeaways Cassandra isn’t a data warehouse You will bring it to its knees if you treat it like one Learn Cassandra schema design patterns to avoid tombstones impacting your reads when working with heavy delete workloads
- 84. What happened?
- 85. What’s worse than the issues we’ve seen?
- 86. Not knowing they happened
- 87. Or your customer discovering it before you
- 88. Root cause
- 89. We don’t think of failure as a first-class citizen
- 90. The fix
- 91. Update your SDLC to measure, test and mitigate failure
- 92. Todo In Progress Done Measure everything (latencies, errors, TPS) Soak tests Load testing beyond expected peaks Define runbook Review runbook + alerts with ops/SRE Verify metrics are live + correct Chaos monkey Have a plan for extended outages, or huge data spikes
- 93. Takeaways Measure failure Anticipate failure Try and break your systems (before production) Alert when you fail Mitigate for it Learn and improve from it
- 94. Thank you!