Bringing Data to the Edge
1 like481 views
The document discusses the concept of edge computing, emphasizing its advantages over traditional cloud solutions, such as reduced latency and better data privacy. It outlines challenges related to data synchronization and proposes decentralized approaches like blockchain for efficient data management. It also presents ObjectBox, a lightweight database designed for edge devices, showcasing its potential for performance and simplicity in handling data.
Bringing Data to the Edge
- 1. Bringing Data to the Edge
- 2. Everything is connected Dirk Gently
- 3. Hello daho.am :) • Local event Decentralized Efficient information exchange Connects local people • Local data at the edge Decentralized Efficient information exchange Connects local devices
- 4. Mobile Data – My personal Journey • Previous life: enterprise stuff Java, expensive DBs, SQL, O/R Mappers (ORMs) • Apps, build two mobile ORMs on top of SQLite Object lover, never a fan of SQL, respect for SQLite • "We can do better than that" ObjectBox – founder and CTO
- 5. What is the "Edge" and Edge Computing? • Nothing really new • Edge: devices on the edge of the Internet E.g. mobile phones, desktops, IoT devices • Edge gateways Servers living on the edge • Locality is a good thing Goal: provide better solutions directly at the edge
- 6. And what's "Fog Computing"? • Sometimes used in the same context (EC) • "A hierarchic cloud" Bring computing and data closer to the edge • Locality levels E.g. building, city block, city, …
- 7. So, what's wrong with the Cloud? Nothing really. It's just a little overused.
- 10. But are we asking the right Question?
- 11. What happened to "It depends"? • Cloud == 42? • The universal answer of IT people: "It depends" Complex stuff, 1000 different solutions • Optimal solutions depend on the use case • When is the Cloud the best solution? It depends!
- 12. Cloud Advantages • Vs. servers: scale & redundancy Less to worry about, "managed" & less down time • Central processing is simple (Mostly) consistent state, easy to monitor, etc. • Implementing logic just once Clients just call an interface (e.g. Web/REST)
- 13. Cloud Disadvantages • Central everything Controlled by big companies (governments?) Hack-able: one breach – millions affected Who owns my data? Does private data exist? • Depends on network Clients are dead when offline, latency • Cloud costs
- 14. Cloud adds Latency • (Near) Real time requirements E.g. Industrial setting Cloud not an option
- 15. Cloud Costs for Apps • Medium app, 150k monthly users • Significant work to reduce cloud time 3 caching layers on the cloud side • US $400-500 per month • Wouldn't be affordable without caches Also on-device database with "conditional gets" • Big apps often spend > $100K per year on cloud
- 16. Cloud costs: IoT • Source: CTO of an IoT company • Single application, huge amount of sensor data • Local K/V cache to mitigate costs • Multiple € 100K per month • "Cloud costs are a problem" Very hard to make a project profitable
- 17. Is the Cloud the new Mainframe? • Once upon a time… • You could submit "jobs" to mainframes • "Jobs" moved to desktop computers • It happened before and it will happen again • Move "Jobs" to the edge
- 18. Utilizing the Edge • While you wait for the cloud… • What are client devices doing? • Nothing! Waiting == wasting resources • Edge computing is resourceful Using the powerful resources we already have
- 19. Data on the Edge – local Data is good Data • Upload of all data is often wasteful Example: IoT sensors Preprocessing & only upload what's needed • Data that doesn't leave the edge Private data stays "in the house" • Again: Edge computing is resourceful Decrease traffic & increase privacy
- 20. Moving to the Edge
- 21. Cloud: Compute & Data Cloud ClientClient Cloud: (Compute & Data) Edge Edge Device: Data & Compute Edge Device: Data & Compute Cloud: (Compute & Data) Edge* Edge Gateway: Data & Compute Client Edge Device: Data & Compute
- 22. Simple Example: In-App Search is already EC • App with some kind of catalog E.g. news, TV data, messages, … • Sync all (relevant) data to device Archive could stay in the cloud • Search data at the edge Functionality at the device, not in the cloud
- 23. Edge Computing & IoT
- 24. IoT devices • More devices than humans Likely to double in the next few years • Big range of technical capability From wearables to seriously powered devices • The typical "dumb" device Sends sensor data to the cloud
- 25. From IoT to EoT • Does an "Internet of Things" make sense? Again: it depends • Edge of Things "Internet exposure considered harmful" • Smart home, edge version Local area network Optional gateway to the cloud Decide what to share
- 26. Bringing Data to the Edge: Challenges • Distributed data Local data, some of it must be synchronized • Keeping data in sync is complex Concurrent edits, conflicts, … • More data stored on device Higher capacity needed (e.g. flash memory) Data storage needs to scale (performance)
- 27. Status quo for getting Data to the Edge • REST based APIs (or GraphQL) Typically return JSON data • No standard to store returned data E.g. custom logic to insert into a SQL database • Requests often fetch all data Often redundant and inefficient
- 28. Data Sync is a Key Technology for Edge Computing
- 29. Data Synchronization • Generic approach to keep data up-to-date Pushing updates two way • Offline support When not connected: queue updates for later • Delta synchronization Sending only deltas is much more efficient • Conflict resolution
- 30. git: Good Example of Distributed Data • Decentralized • History is not linear Branches introduce another dimension • Merge operations Automatic, or manual conflict handling • git concepts data synchronization? Makes a pretty good starting point
- 34. git & Crypto: more Inspiration • Chain of cryptographic hashes Content is hashed Previous commit hash is part of the hash • Tampering data would break the hash chain Considered a different branch, easy to track down • Optional: signing E.g. prove that a commit comes from you
- 35. Data Synchronization Implementation • Transaction based databases A transaction is a list of actions • A transaction is a state transition Actions & data can be stored in a log • Each state should be identifiable (ID) Enable clients to pick up from previous state (aka delta synchronization)
- 36. Data Synchronization Implementation • State ID can be a hash Yep, just like git… • State transition A B with transaction log TXL hash(B) = hash(hash(A) + hash(TXL)) • Chain of hashes • Clients can pick up from their last hash ID Delta sync unlocked
- 38. Full Decentralization – Though Challenges • Anarchy and Chaos? No central instance may/can control/interfere • Data bubbles Data gets separated and inconsistent • Trust issues Which peers to trust?
- 39. Blockchain! The new silver bullet…
- 40. Blockchain is a DB Decentralized P2P, public, crypto-based
- 41. Blockchain Achievements • Alternative investments & a new mining industry Eventually also new currencies • Smart contracts "If this then that" - combining data and code • Crypto is cool again • Working well enough Showed the world that there are alternatives
- 42. Blockchain Tech • Block of transactions Merkle tree to produce a hash • Diverging data/history One branch is selected using a scoring model (e.g. longest chain) • Proof-of-work Calc nonce to find special hash(content + nonce)
- 43. Blockchain Tech – the Dark Side • Decentralized consensus: ridiculously inefficient Compared to central instance approval • Proof of work: "useless" expensive computation Redundantly done by thousands of miner nodes • Nonce lottery is like "wasting" ~99.99% of energy • Effects on the environment? • (Today's) blockchain is not the end of the journey
- 44. Data Sync vs. Blockchain • Implementations can share concepts Transactions, chained hashes • Key difference: consensus and trust Centralized vs. decentralized • Centralized consensus is preferred for apps Producing companies "own" their apps (Efficiency restored… )
- 45. What we are working on
- 46. ObjectBox – a Database for the Edge • "A DropBox for data objects" Allows to work offline • We start at the edge: embedded database Based on objects and relations Sync: stored bytes can be sent and applied 1:1 • Simplicity and efficiency Guiding principles
- 47. ObjectBox • Runs on mobile, desktop, IoT devices Android, Linux, Windows, i/MacOS, Raspberry, … • Low footprint: less than 1 MB • 10x faster CRUD operations Outperforms SQLite, fastest embedded DB • ACID compliant
- 48. ObjectBox • Objects all the way through No transformations required (no SQL, no REST/JSON parsing) • Convenience at K/V store speed Indexing, queries, relations • ObjectBox 2.0 Release 25th July 2018
- 49. Wrapping up
- 50. Edge Computing - the Golden Hybrid? Centralized Decentralized Server Cloud Peer-to-Peer Edge "Super Peers"
- 51. Data Dimensions • Data localilty Cloud, edge-only, shared cloud-edge • Online vs. offline data Online-only cloud, offline: synced or edge-only • Consensus: which data prevails? Central authority, decentralized heuristics
- 52. Questions to ask • Who owns "my data"? Can data be private at all? • How much (de)centralization do you need? What works best for your use case? • How can we make efficiency a virtue again? Can efficiency reduces your costs too?
- 53. Thanks! 🐬🐬 Contact: markus @ objectbox.io Follow me: @greenrobot_de