Big data uservices
0 likes62 views
This document discusses trends in computing and data architectures. It notes that hardware is becoming more virtualized through containers and functions, bringing code closer to storage. Meanwhile, data structures are becoming more distributed through databases, distributed databases, and distributed ledgers. Computing scopes are decreasing while storage scopes are increasing. This is because data is inert but code can be more tightly controlled at smaller scopes. The document also discusses challenges of integrating microservices with data, including consistency, and notes the importance of people and processes ("metadata") alongside technology.
Big data uservices
- 2. whoami ● NETOPIA Payments ○ Surprisingly (or not) we do other things than payments ● mobilPay Wallet ● Elrond ● Blockchain Romania
- 3. The (physical) universe consists of a 3 x Space + 1 x Time manifold
- 4. The computational universe consists of 1 x Computing + 1 x Storage + 1 x Time manifold
- 5. In the computational universe Computing and Storage are bound together by Code
- 6. These days Code moves/runs further and further away* from Computing
- 7. Id est ● Hardware has morphed into virtual machines ● VMs have morphed into containers ● Containers are morphing into functions
- 8. At the same time ● When talking about “pure” data - the closer to the hardware the faster it can be accessed - and with smaller latency
- 9. Id est ● Files have morphed into databases ● Databases have morphed into distributed databases ● Distributed DBs have morphed into ○ distributed Ledgers and Blockchains ○ also into Data Lakes and Deltas (not sure yet about Seas and Oceans)
- 10. Aside: Big(ger) Data Structures OLAP OLTP Homogenous Data Lake, Delta, Sea Distributed Ledger* Heterogeneous Blockchain
- 11. Empirical observation Storage Increases in scope Computing Decreases in scope
- 12. Guesstimation ● This happens because data is “inert” but ● The smaller the unit of code that needs to run, the more controllable the output/behavior (we can even formally verify some - if not all - of it)
- 13. Microservices unify interaction (even if data is also involved)
- 15. Destructuring Monoliths ● 3 TA with distributed application layer and/or data layer ● Request driven (pull) vs Event driven (push) ● Messaging - Event Sourcing ● Sagas w/ or w/o orchestrators
- 16. Microservice Moving away from Monoliths Monolithic Application Computing Storage Storage Computing Microservice Storage Computing
- 17. Option 1 With this approach ● Big Data = Σ Small(ish) data ● Data consistency (say, transactions) is hard
- 18. Microservice Another option MicroserviceMonolithic Application Computing Storage Storage Computing Microservice Computing
- 19. Option 2 ● Storage = House of Big Data ● Does not work all the time. It might even be an anti-pattern.
- 20. Modus Operandi ● API Gateway / Service Mesh ● Service Registry These tend to handle only the computing side
- 21. Data side ● Additional concerns: “at rest” behavior vs “in flight” behavior ● Schemas => APIs ● Stateful serverless?
- 22. DataOps ● It’s not just DevOps for Data ● Intersection of Computing and Storage w/ Organization ○ Human Resource 4-dimensions ■ Wants ■ Knows ■ Time / Availability ■ Privileges
- 23. Storage and Computing are just as important as metadata (including processes, people ...)
- 24. Conway’s law Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure Melvin Conway, 1967
- 25. But even if we only talk about technology things are complicated
- 26. CNCF Landscape Img Src https://landscape.cncf.io/
- 30. YMMV: just because you can do it, it doesn’t mean that you should do it