Bending Spark towards enterprise needs
2 likes560 views
The document discusses using Spark for enterprise applications. Some key advantages include easy scalability, high throughput for non-low latency apps, and separated concerns which is good for distributed teams. However, it notes that Spark may be over-engineering for small projects and to think twice for pure CRUD apps where a traditional approach may be better suited. It also cautions that with Spark it is easy to introduce bugs so careful design is important.
Bending Spark towards enterprise needs
- 2. Leading DWH at AOL. / Vidible division / Major expertise Big Data and Enterprise Cofounder of Odessa JUG Passionate follower of Scala Associate professor at ONPU ABOUT ME
- 7. P R E S E N TAT I O N L AY E R A P P L I C AT I O N S E R V I C E S D O M A I N L AY E R D A L INFRASTRUCTURE CLASSIC LAYERS
- 11. CQRS NAKED USER INTERFACE SERVICE LAYER WRITE STORE COMMAND BUS COMMAND HANDLER REPOSITORY Domain Model Domain Model EVENT BUS EVENT HANDLER Commands READ MODEL STORE QUERY FACADE Query DTO WRITE MODEL READ MODEL
- 12. USER INTERFACE MULTIPLE READ MODELS USER INTERFACE SERVICE LAYER WRITE STORE COMMAND BUS COMMAND HANDLER REPOSITORY Domain Model Domain Model EVENT BUS EVENT HANDLER Commands EVENT HANDLER READ MODEL STORE 1 READ MODEL STORE 2 QUERY FACADE QUERY FACADE WRITE MODEL READ MODEL
- 15. SERVICE LAYER SPARK IS COMING USER INTERFACE MONGO READ/WRITE STORE KAFKA COMMAND HANDLER REPOSITORY Domain Model Domain Model Commands QUERY FACADE Query DTO WRITE MODEL READ MODEL SPARK
- 16. SERVICE LAYER SPARK IS COMING USER INTERFACE MONGO READ/WRITE STORE KAFKA COMMAND HANDLER REPOSITORY Domain Model Domain Model Commands QUERY FACADE Query DTO WRITE MODEL READ MODEL SPARK SHARED STORE
- 17. TECHNOLOGY STACK FAÇADE • Java 8 • Spring Boot & MVC • Mongo COMMAND PROCESSOR • Kafka 0.9.0.1 • Scala 2.11 • Spark 2.0.0 Streaming
- 21. WAIT
- 22. WELCOME TO ASYNCHONOUS HELL AT LEAST FOUR SOLUTIONS • C'mon, it’s OK • Pulling service until entity is added • Add push channel and deliver events upstairs • Make REST methods blocked until command is finished
- 25. RETROSPECTIVE
- 27. dataStream.foreachRDD { rdd => val values = rdd.values values.map { cmdMessage => … } .foreach { cmdMessage => … } } DESERIALIZATION ERRORS MIGHT FAIL DURING LAUNCH IF PIPELINE HAS BEEN CHANGED RISK: AFFECTS CHECKPOINTS
- 28. DESERIALIZATION ERRORS • Hard to keep code tolerant against these errors and testable at the same time. • Spark with Kafka connector 0.10 allows to save Kafka offsets and it is possible to restore from them instead of checkpoints.
- 30. IDENTIFIER ASSIGN RESPONSIBILITY • Client generates unique identifier and passes it to service • Service generates unique identifier and passes it back to client
- 31. EVENTUAL VS STRONG CONSISTENCY • CQRS apps usually are eventually consistent • Choose wisely between CQRS and traditional approach • Consider optimistic or pessimistic locking to achieve strong consistency • In some cases defined reasonable timeouts or in-progress lists might help to solve race conditions (depends on command roundtrips)
- 32. FAULT TOLERANCE
- 34. SUMMARY P R O SC O N S
- 35. SUMMARY P R O SC O N S Best fit for highly throughput and non-low latency apps
- 36. SUMMARY P R O SC O N S Easy to shoot one's foot Best fit for highly throughput and non-low latency apps
- 37. SUMMARY P R O SC O N S Easy to shoot one's foot Easy to scale Best fit for highly throughput and non-low latency apps
- 38. SUMMARY P R O SC O N S Easy to shoot one's foot Easy to scale Best fit for highly throughput and non-low latency apps Think twice on pure CRUD apps
- 39. SUMMARY P R O SC O N S Easy to shoot one's foot Easy to scale Best fit for highly throughput and non-low latency apps Separated concerns Think twice on pure CRUD apps
- 40. SUMMARY P R O SC O N S Easy to shoot one's foot Easy to scale Best fit for highly throughput and non-low latency apps Might be an over-engineering for projects with small domain Separated concerns Think twice on pure CRUD apps
- 41. SUMMARY P R O SC O N S Easy to shoot one's foot Easy to scale Best fit for highly throughput and non-low latency apps Might be an over-engineering for projects with small domain Separated concerns Think twice on pure CRUD apps Good fit distributed teams with decomposed domains