Data Democratization at Nubank
1 like1,459 views
The document discusses the strategies and principles behind building a modern data platform at Nubank, emphasizing the need for alignment between business users and engineers to facilitate data-driven decision-making. It outlines key principles such as keeping systems simple, maintaining data integrity, and automating data ingestion, while also highlighting the importance of collaboration, minimizing complexity, and the use of functional programming. The document also presents metrics on datasets and performance, showcasing Nubank's engineering achievements and culture of innovation.
![[ 28 ':name' 'john' ]
29](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-29-320.jpg)
![entity
attribute value
[ 28 ':name' 'john' ]
30](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-30-320.jpg)
![[ 28 ':name' 'john' ]
'lennon'
entity
attribute value
31](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-31-320.jpg)
![[ 28 ':name' 'john' Tx₁ true]
[ 28 ':name' 'john' Tx₂ false]
[ 28 ':name' 'lennon' Tx₂ true]
entity
attribute value transaction op
32](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-32-320.jpg)
![(let [log (datomic.api/log datomic)
t1 10000
t2 50000]
(datomic.api/tx-range log t1 t2))
=> [Datom(:e 1234,
:a :transaction/value
:value 10.00
:t 10000)...]](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-33-320.jpg)
![Infrastructure as
code
{:name :diablo
:canary {:type :shard}
:datastores
{:datomic {:databases
[{:transactor “diablo”
:name “diablo”}]}
:kafka {:enabled? true}}
:environments
{… :prod #nu/prototypes-for [:prod :sharded]}
…
:pipelines
#{{:type :clojure-service
:prod {:promotion :automatic}
:cdc-test-frameworks #{:sachem}}}
{:name :diablo
:canary {:type :shard}
:datastores
{:datomic {:databases
[{:transactor “diablo”
:name “diablo”}]}
:kafka {:enabled? true}}
:environments
{… :prod #nu/prototypes-for [:prod :sharded]}
:pipelines
#{{:type :clojure-service
:prod {:promotion :automatic}
:cdc-test-frameworks #{:sachem}}}}](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-34-320.jpg)
![val report: (allTheData: Map[String,
DataFrame]) => DataFrame](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-38-320.jpg)
![trait SparkOperation {
val name: String
val inputs: Set[String]
val definition: (inputs: Map[String,
DataFrame]) => DataFrame
}](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-40-320.jpg)
![object OmbudsmanCalls extends SparkOp{
override val name: DataFrame = "dataset/ombudsman-calls"
override val inputs: Set[String] = Set(callsName)
override def definition(datasets: Map[String, DataFrame]):
DataFrame = {
val calls = datasets(callsName)
(calls
where ($"started_at".isNotNull
and $"our_number" === phoneNumber)
select ($"started_at", $"call__id", $"our_number"))
}
def phoneNumber: String = "999-9999"
def callsName: String = “contract-phone/calls”
}](https://image.slidesharecdn.com/andremidearodrigoney-191031205536/85/Data-Democratization-at-Nubank-41-320.jpg)
Data Democratization at Nubank
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. @Nubank Andre Midea Rodrigo Ney Data Democratization #UnifiedDataAnalytics #SparkAISummit
- 6. Credit card supported by a fully digital and branchless experience. 2014 2017 Our own version of a bank account, the simplest and most intelligent solution yet.
- 9. Growing Quickly A bank from scratch using Clojure
- 10. Roles making decisions Business Analyst Financial Analyst Legal Data-scientists Customer Support 1 2 3 4 5
- 11. We need to build a data platform
- 12. The Report Lifecycle Where good ideas perish
- 13. #UnifiedDataAnalytics #SparkAISummit Nooooo! Have you ever faced/seen this situation?
- 14. Business users engineers are too slow and don’t implement their ideas correctly and fatigue to implement new ideas because the process is too painful Engineers business users underestimate the challenges to implement something with a good level of quality Frustrations
- 15. Goal A data-platform where the incentives are aligned so that people are empowered to make more informed decisions, achieving that by reducing the friction for nontechnical people to create creative solutions using data.
- 16. the absence of hereditary or arbitrary class distinctions or privileges /democracy/
- 17. Agile DevOps DataOps1 Movements in the software community 2 3 Customer collaboration over contract negotiation. Relationships increases the sense of safety when we work together as partners. Decentralizing the maintenance of services, so every team is responsible for maintaining their services running, creating alerts, having SLA and SLOs Reduce heroism: As the pace and breadth of need for analytic insights ever increases, we believe analytic teams should strive to reduce heroism and create sustainable and scalable data analytic teams and processes.
- 18. a principled approach to data-engineering
- 19. Our Stack
- 21. • Accumulate-only/ Immutable • Git for your data • Transaction log as high level API
- 22. • Lazy • Declarative • FP inspired
- 23. 330+ 315 400 engineers Micro-Services deploys per week 80 13+ 2100 TB Data SHARDS datomic transactors In production Resources: Architecting a Modern Financial Institution (InfoQ), Challenges and Benefits of an Immutable Database, Nubank Talk Youtube Playlist
- 26. Principle 1 Having data coverage is important. I mean, ALL the data!
- 27. Love your logs
- 28. Datomic EAVT e Entity Long a Attribute String v Value Any t Transaction point in time Long tx Transaction entity id Long txInstant Transaction wall-clock time java.util.Date op Operation (assertion / retraction) Boolean 28
- 29. [ 28 ':name' 'john' ] 29
- 30. entity attribute value [ 28 ':name' 'john' ] 30
- 31. [ 28 ':name' 'john' ] 'lennon' entity attribute value 31
- 32. [ 28 ':name' 'john' Tx₁ true] [ 28 ':name' 'john' Tx₂ false] [ 28 ':name' 'lennon' Tx₂ true] entity attribute value transaction op 32
- 33. (let [log (datomic.api/log datomic) t1 10000 t2 50000] (datomic.api/tx-range log t1 t2)) => [Datom(:e 1234, :a :transaction/value :value 10.00 :t 10000)...]
- 34. Infrastructure as code {:name :diablo :canary {:type :shard} :datastores {:datomic {:databases [{:transactor “diablo” :name “diablo”}]} :kafka {:enabled? true}} :environments {… :prod #nu/prototypes-for [:prod :sharded]} … :pipelines #{{:type :clojure-service :prod {:promotion :automatic} :cdc-test-frameworks #{:sachem}}} {:name :diablo :canary {:type :shard} :datastores {:datomic {:databases [{:transactor “diablo” :name “diablo”}]} :kafka {:enabled? true}} :environments {… :prod #nu/prototypes-for [:prod :sharded]} :pipelines #{{:type :clojure-service :prod {:promotion :automatic} :cdc-test-frameworks #{:sachem}}}}
- 36. Principle 2 No need for excess of features Pave the right roads!
- 37. - Nathan Marz. “Big Data: Principles and best practices of scalable realtime data systems”. batch view = function(all data) “The portion of the Lambda Architecture that implements the: equation is called the batch layer. The batch layer stores the master copy of the dataset and precomputes batch views on that master dataset. The master dataset can be thought of as a very large list of records.”
- 38. val report: (allTheData: Map[String, DataFrame]) => DataFrame
- 39. Make datasets reusable by default Extend the same abstraction“ “ f(1,2) f(3,4) Dataset 1 Dataset 2 Dataset 4 Dataset 3 Dataset 5
- 40. trait SparkOperation { val name: String val inputs: Set[String] val definition: (inputs: Map[String, DataFrame]) => DataFrame }
- 41. object OmbudsmanCalls extends SparkOp{ override val name: DataFrame = "dataset/ombudsman-calls" override val inputs: Set[String] = Set(callsName) override def definition(datasets: Map[String, DataFrame]): DataFrame = { val calls = datasets(callsName) (calls where ($"started_at".isNotNull and $"our_number" === phoneNumber) select ($"started_at", $"call__id", $"our_number")) } def phoneNumber: String = "999-9999" def callsName: String = “contract-phone/calls” }
- 42. Yeah, but nobody will use this… It’s hard! Spark and Scala… We managed to have more than +300 business users working with our abstraction. More than 15 Pull Requests per day. ~500k LOC (only datasets)
- 43. Principle 3 KISS Keep it Simple Stupid! B Batch
- 44. Donald Knuth “… 97% of the time: premature optimisation is the root of all evil. Yet we should not pass up our opportunities in that critical 3%”
- 45. TYPE USER CAN USE? RAW CONTRACT DATASET SUPPORT Platform Maintained Platform Maintained & User Generated User Defined 45
- 47. user data User Land user data user data cache cache cache where t > t’' where t > t’' where t > t’' t’' t’' t’' raw/log raw/log raw/log contract contract contract
- 48. raw/log user dataset Immutable/Append only user dataset user dataset user dataset user dataset user dataset Daily Effects Introduce a Change The dataset breaks raw/log raw/log
- 50. Safety Nets Keep your DAG valid at all times! Principle 4
- 51. Complex
- 52. Spark Dataframe Raw layer Test DataFrame eavt schema We know the schema of all raw datasets!
- 53. Spark Dataframe Raw layer We can pipe the DataFrame from one operation downstream in the lineage R’ R’’ R’’’ R’’’’eavt schema Test DataFrame f(R’) f(R’’) f(R’’’) f(R’’’’) User Land It's lazy! u’ u’’ u’’’ u’’’’
- 54. Spark Dataframe Raw layer We can pipe the DataFrame from one operation downstream in the lineage R’ R’’ R’’’ R’’’’eavt schema Test DataFrame f(R’) f(R’’) f(R’’’) f(R’’’’) User Land It's lazy! u’ u’’ u’’’ u’’’’ f(U’,U’’) 2u’ 2u’’ f(U’’’,U’’’’)
- 55. Spark Dataframe Raw layer All Spark transformations are valid! R’ R’’ R’’’ R’’’’eavt schema Test DataFrame f(R’) f(R’’) f(R’’’) f(R’’’’) User Land u’ u’’ u’’’ u’’’’ f(U’,U’’) 2u’ 2u’’ f(U’’’,U’’’’) 3u’ 3u’’ 4u’ f(…) f(…) f(…)
- 56. • Integration Tests • Consumer Driven Contracts • Integrity Checks • Anomaly Detection based on a dataset's statics over time
- 58. REPL environment closer to production
- 59. new(service + db) logs pivoted contract user data dataset data warehousebi tool back to production… automatically! Meanwhile….
- 60. Results
- 61. 125 250 375 500 3000 6000 9000 12000 2016,Q3 2016,Q4 2017,Q1 2017,Q2 2017,Q3 2017,Q4 2018,Q1 2018,Q2 2018,Q3 2018,Q4 2019,Q1 2019,Q2 2019,Q3 2019,Q4 Datasets dataset per data engineer 11523 9246.283 7247.033 5686.479 2946.159 3804.225 2835.279 2289.881 1707.904 1034.355761.992 240.07759.7333.3333.333 59.733 240.077 761.992 1034.355 1707.904 2289.881 2835.279 3804.225 2946.159 5686.479 7247.033 9246.283 11523 Data Engineer vs Datasets That discrepancy is even more pronounced when we look at the number of datasets generated per data-engineer
- 62. “ “ “1. Think carefully in the roads you want to pave, and pave only a handful of them 2. Constraints are a good thing, if they hide complexity away users will be glad by them 3. Leverage the Transaction Log KEY TAKEAWAYS “ “ “4. Automate Data- Ingestion to the extreme 5. Have a Dev environment closer to production 6. Validate the DAG in test time, and create invariants for the datasets.
- 64. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT