Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Flink Jobs with GUI
0 likes521 views
TouK Nussknacker is a GUI for Apache Flink that was created to address client needs for a more user-friendly interface. It allows defining data pipelines and business rules through expressions that are accessible for semi-technical users. The architecture uses a model-driven approach where the data, sources, sinks, processors and rules are defined through JSON configuration. It has been used successfully in production for over a year at one of the largest Polish telcos for real-time marketing and fraud detection applications.
![Config creator
trait ProcessConfigCreator extends Serializable {
def services(config: Config) : Map[String, WithCategories[Service]]
def sourceFactories(config: Config): Map[String, WithCategories[SourceFactory[_]]]
def sinkFactories(config: Config): Map[String, WithCategories[SinkFactory]]
...
}](https://image.slidesharecdn.com/touknussknacker-flinkforward1-170914130917/85/Flink-Forward-Berlin-2017-Maciek-Prochniak-TouK-Nussknacker-creating-Flink-Jobs-with-GUI-16-320.jpg)
![Service/Enricher
@MethodToInvoke
def invoke(@ParamName("clientId") clientId: String)
(implicit ec: ExecutionContext): Future[Client] = {
...
}](https://image.slidesharecdn.com/touknussknacker-flinkforward1-170914130917/85/Flink-Forward-Berlin-2017-Maciek-Prochniak-TouK-Nussknacker-creating-Flink-Jobs-with-GUI-17-320.jpg)
![Custom stream transformer - POFO
@MethodToInvoke(returnType = classOf[EventCount])
def execute(@ParamName("key") key: LazyInterpreter[String],
@ParamName("length") length: String) =
FlinkCustomStreamTransformation((start: DataStream[InterpretationResult]) => {
start.keyBy(key.syncInterpretationFunction)
.transform("eventsCounter", new CounterFunction(lengthInMillis))
})](https://image.slidesharecdn.com/touknussknacker-flinkforward1-170914130917/85/Flink-Forward-Berlin-2017-Maciek-Prochniak-TouK-Nussknacker-creating-Flink-Jobs-with-GUI-27-320.jpg)
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Flink Jobs with GUI
- 1. TouK Nussknacker GUI for Flink
- 3. How it all began? POC with Apache Flink Great results :)
- 4. Configuration Client not (really) happy
- 5. Scala expression for filters Client still not very happy
- 6. Ok, we’ll build you a GUI Finally, approving nod...
- 8. Main assumptions ● Model & integration in “normal” code ● Expressions accessible for semi-technicals - like SQL ● Facilitate testing and experimentation
- 9. Basics
- 10. Architecture
- 11. Model + process JSON Config JAR JSON JSON
- 14. Model
- 15. Model ● Data - POJOs, case classes ● Source/Sink - Flink API + ε ● Processor/Enricher - Services
- 16. Config creator trait ProcessConfigCreator extends Serializable { def services(config: Config) : Map[String, WithCategories[Service]] def sourceFactories(config: Config): Map[String, WithCategories[SourceFactory[_]]] def sinkFactories(config: Config): Map[String, WithCategories[SinkFactory]] ... }
- 17. Service/Enricher @MethodToInvoke def invoke(@ParamName("clientId") clientId: String) (implicit ec: ExecutionContext): Future[Client] = { ... }
- 18. Business rules - expressions ● Filters ● Aggregation definitions ● Actions - mail subject, sms content How to define them...?
- 19. Expressions Spring Expression Language! ● Accessible enough ● More or less fast enough ● Ability to do basic validation/code-completion
- 22. Limitations of SPEL ● Speed ● Synchronous ● Type safety??
- 23. Advanced?
- 24. What about... ● Windows? ● State? ● Joins? ● Other Flink goodies?
- 25. Knowing where to stop Completness Difficulty
- 26. Knowing where to stop
- 27. Custom stream transformer - POFO @MethodToInvoke(returnType = classOf[EventCount]) def execute(@ParamName("key") key: LazyInterpreter[String], @ParamName("length") length: String) = FlinkCustomStreamTransformation((start: DataStream[InterpretationResult]) => { start.keyBy(key.syncInterpretationFunction) .transform("eventsCounter", new CounterFunction(lengthInMillis)) })
- 28. Tests/Ops
- 29. “Convincing” to test We’ll do it “our” way: first we run on production, then deploy to test
- 30. Tests - running
- 31. Tests - running
- 34. Taking control ● Do we process data? ● What is latency? ● Where do we filter out events? ● Are there errors?
- 35. Taking control
- 36. DEMO
- 37. Where can I use it? ● Telco ● Banking ● Media ● RTM ● Fraud detection
- 38. Adhoc jobs vs process Ad-hoc Query SQL RBEA CI CD Process Monitoring SAS Scala Testing
- 39. Does it work in production? ● 1 year in production ● One of largest Polish telco’s ● RTM + Fraud detection
- 40. Does it work in production?
- 42. Road ahead? ● Asynchronous services ● Pluggable security ● Define model via GUI/Schema registry
- 43. Road ahead? ● Integration with CEP and SQL ● DAGs instead of Trees in UI? ● Governance? ● Nussknacker as a Service?