Data encoding and Metadata for Streams
7 likes3,600 views
The document discusses data encoding and metadata for streams, emphasizing the importance of efficient data practices in data pipelines, especially in the healthcare software industry. It explains the concept of streams as an abstract data structure, outlines various encoding formats, and highlights the significance of identity and metadata for stream elements. The conclusion advocates for the adoption of stream technology over traditional databases for improved performance and scalability.
![Metadata
> Metadata
• Metadata uses range from the very useful (like
http headers) to the very meta meta[1].
• Metadata on Stream elements is most of the
time implicit, like for example the Content-Type :
• “It’s a stream of JSONs” then every element
of the stream has “content-type=application/json”.
[1] I am looking at you RDF !](https://image.slidesharecdn.com/stream-141031061225-conversion-gate01/85/Data-encoding-and-Metadata-for-Streams-21-320.jpg)
![> Metadata
A quick note on provenance
The provenance is practical in distributed systems we
want to know :
• from which node do a element comes.
• on the behalf of which agent this element is created.
• from which environment[1] a element comes.
[1] with new architecture and Data Labs, environments are sometimes
shared on the same infrastructure (eg : no Pre-Production platform).
It’s then very useful to safeguard against the pollution of data.](https://image.slidesharecdn.com/stream-141031061225-conversion-gate01/85/Data-encoding-and-Metadata-for-Streams-23-320.jpg)
![> Metadata
{
"content-type":"application/json",
"profile":"domain.model.MessageSent",
"provenance":{
"application":{
"name":"webapp",
"version":"68546ca6e963981a8279aa327cc1e1362d15554e"
},
"node":{
"environement":"test",
"network":{
"interface":{
"en0":{
"addresses":{
"192.168.0.13":{
"family":"inet",
"netmask":"255.255.255.0",
"broadcast":"192.168.0.255"
}
}
}
}
},
"hostname":["Blaze"],
"platform_family":"mac_os_x"
}
}
}
• The metadata of an element can
represent a significant piece of data.
Sometimes more than the data itself.
• !! The same piece of metadata can be
shared across many elements. !!](https://image.slidesharecdn.com/stream-141031061225-conversion-gate01/85/Data-encoding-and-Metadata-for-Streams-24-320.jpg)
![> Datagram
Anatomy of an element
:ID :HEADERS
:BODY
DB7D919B-248F-4676-
8494-2698B48C69C3
57158663-5933-4CE6-
A54E-8179ECFBFCCA [“ich”,“bin”,“ein”,“JSON”]
e.g.](https://image.slidesharecdn.com/stream-141031061225-conversion-gate01/85/Data-encoding-and-Metadata-for-Streams-25-320.jpg)
![> Datagram
1. Create and register your headers
(in a distributed Key/Store for example) .
4813EDF2-B04E-4B70-
{
AB04-0F9EA456E032
"content-type":"application/json",
"profile":"domain.model.MessageSent",
"provenance":{
"application":{
"name":"webapp",
"version":"68546ca6e963981a8279aa327cc1e1362d15554e"
},
"node":{
"environement":"test",
"network":{
"interface":{
"en0":{
"addresses":{
"192.168.0.13":{
"family":"inet",
"netmask":"255.255.255.0",
"broadcast":"192.168.0.255"
}
}
}
}
},
"hostname":["Blaze"],
"platform_family":"mac_os_x"
}
}
}](https://image.slidesharecdn.com/stream-141031061225-conversion-gate01/85/Data-encoding-and-Metadata-for-Streams-26-320.jpg)
Data encoding and Metadata for Streams
- 1. Data encoding and metadata for streams
- 3. > Introduction Me at a glance • My name is Jonathan Winandy (@ahoy_jon). • I am a Data pipeline engineer : • I worked on a “DataLake” ! • I use tools in the larger Java ecosystem like Java, Scala, Clojure, Hadoop … • And I am an “entrepreneur”.
- 4. > Introduction I cofounded two companies and they use streams as their data backbone. Health care oriented software engineering. Provide : Coordination for health care professionals.
- 5. I cofounded two companies and they use streams as their data backbone. @PrimaticeData “Good dataviz, surreal backends.” > Introduction Provide : Tools and methods for Data capitalisation.
- 6. > Introduction What are Streams ? It’s an abstract data structure with the following : operations : • append(bytes) -> void? • readAt(int) -> null | bytes rule 1 : ∀p ∈ ℕ, for some definition of ‘==‘ x := readAt(p) y := readAt(p) ! x != null => x == y Rule 1 implies : Infinite cacheability once the data is available at a position.
- 7. Streams are the simplest way to manage data. 0 1 2 3 4 5 6 And they are naturally compatible with the perception of information from a singular observer … > Introduction
- 8. But be careful, streams are definitely not like queues, ESB, EAI, or what ever messaging solution comes to mind …
- 9. > Introduction There is a lot to tell on Streams • Sub events : Events are pre-projected into … • Quantum of action : A ‘user’ action generates zero or one event (no more). • Structural sharing for large payload (cf. Content Addressable Storage). • Garbage collection for append only data structures. this presentation ! • Causality enforcement in asynchronous contexts : On important request, causality is enforced. • Binary encoding and Metadata.
- 10. > Introduction A quick note on Causality If you don’t ensure causality for web apps, some strange comportements may arise : Sometimes, as a user, I cannot see my own “edits”. Sometimes, as a client, I cannot buy on the website after I checkout my basket. APP APP “Who is the fastest between the Data bus and the client ?” You don’t want to bet, especially under load.
- 11. Data encoding and metadata for streams
- 12. > Content Content : • Data encoding • Identity • Metadata • Datagram • Conclusion
- 13. > Data encoding State of data encodings in the industry • As always worse is considered better. • Most of streams have data encoded in : • CSV/TSV • JSON • Platform specific serialisations (eg: Java serialisation, Kryo)
- 14. > Data encoding Why this is important ? • Some streams may contains very large amount of Data, the chosen encoding must be cpu and space efficient. • Streams are processed by many programs, and many intermediaries, for many years, the chosen encoding must be processable in a generic way.
- 15. JSON is the lower denominator Plus : • It reaches the browser, you can produce and consume data from inside a web page. A lot of Cons : • Inefficient, • No dates, no proper numerics, • Very basic data structures, • Very error Prone. We all need JSON, but we should use it only when we can't avoid it. > Data encoding Eg : In our databases, we can avoid JSONs ;)
- 16. > Data encoding How bad JSON is ? {“name”:"Bob", “age":11, "gender":"Male"} 39 Bytes for 10 Bytes of data :02:06:62:6f:62:02:16:02:da:01
- 17. > Data encoding relevant ones popular binaries low tech cognitect “papa ?!” Avro Thrift Proto Buf JSON CSV Fressian Transit EDN XML RDF binary YES NO YES OK NO ?? generic YES ?? NO YES YES YES schema based YES NO YES NO ?? meta specific encoding S” YES OK Literal YES NO “STRING s reach the browser YES NO +++++ OK NO YES OK easy ? NO I PASS “true” YEP HUM ? … safe ? YES HUM? NO NO MISM ATCH <! YES has dates? Soon NO NO YES YES
- 18. Identity > Identity • Most mechanism around stream assure an “at most once delivery”. • An identity definition is necessary to ensure idempotency.
- 19. > Identity There are 2 ways to refer to a message : • with a fingerprint calculated from the message (digest). • with an external identifier (like UUIDs).
- 20. > Identity UUIDs allow : F0991FD1-D58A-4A5F-8D13-903F368882D1 8AA5C612-B365-4F8F-AF3F-DF623E1F6B22 93A87D37-0658-47C9-84F6-801E83A5821C • to manage things that are not encoded yet. • to avoid the hashing and the parsing of payloads. Recommandation : add an UUID (128bits) to every elements of the stream.
- 21. Metadata > Metadata • Metadata uses range from the very useful (like http headers) to the very meta meta[1]. • Metadata on Stream elements is most of the time implicit, like for example the Content-Type : • “It’s a stream of JSONs” then every element of the stream has “content-type=application/json”. [1] I am looking at you RDF !
- 22. What kind of metadata there are for streams element ? • Content-type or data-encoding : e.g. : application/json • Type or Profile : indicate that the given element is an instance of a given type. e.g. : domain.model.MessageSent • Provenance information : e.g. : {“env”:”test”, “application”:{“name”:”webapp”, “version”:{“commit”:”68546ca…”}}} > Metadata
- 23. > Metadata A quick note on provenance The provenance is practical in distributed systems we want to know : • from which node do a element comes. • on the behalf of which agent this element is created. • from which environment[1] a element comes. [1] with new architecture and Data Labs, environments are sometimes shared on the same infrastructure (eg : no Pre-Production platform). It’s then very useful to safeguard against the pollution of data.
- 24. > Metadata { "content-type":"application/json", "profile":"domain.model.MessageSent", "provenance":{ "application":{ "name":"webapp", "version":"68546ca6e963981a8279aa327cc1e1362d15554e" }, "node":{ "environement":"test", "network":{ "interface":{ "en0":{ "addresses":{ "192.168.0.13":{ "family":"inet", "netmask":"255.255.255.0", "broadcast":"192.168.0.255" } } } } }, "hostname":["Blaze"], "platform_family":"mac_os_x" } } } • The metadata of an element can represent a significant piece of data. Sometimes more than the data itself. • !! The same piece of metadata can be shared across many elements. !!
- 25. > Datagram Anatomy of an element :ID :HEADERS :BODY DB7D919B-248F-4676- 8494-2698B48C69C3 57158663-5933-4CE6- A54E-8179ECFBFCCA [“ich”,“bin”,“ein”,“JSON”] e.g.
- 26. > Datagram 1. Create and register your headers (in a distributed Key/Store for example) . 4813EDF2-B04E-4B70- { AB04-0F9EA456E032 "content-type":"application/json", "profile":"domain.model.MessageSent", "provenance":{ "application":{ "name":"webapp", "version":"68546ca6e963981a8279aa327cc1e1362d15554e" }, "node":{ "environement":"test", "network":{ "interface":{ "en0":{ "addresses":{ "192.168.0.13":{ "family":"inet", "netmask":"255.255.255.0", "broadcast":"192.168.0.255" } } } } }, "hostname":["Blaze"], "platform_family":"mac_os_x" } } }
- 27. > Datagram 2. use it in your stream ! 5462E738-ABAA-452F- 87E0-FD38AEB9DF81 4813EDF2-B04E-4B70- AB04-0F9EA456E032 {"cid": {"idStr": "498683D2-1192-4794-8C23-5BE49EEEC763"}, "userId": {"idStr": "BC3D8614-AF1F-48C8-B91F-0D907FD0FAF3"}, "content": " Contenu de message de test"} 81C76676-7B19-428E-8 56D-984BB67287D1 4813EDF2-B04E-4B70- AB04-0F9EA456E032 {"cid": {"idStr": "498683D2-1192-4794-8C23-5BE49EEEC763"}, "userId": {"idStr": "BC3D8614-AF1F-48C8-B91F-0D907FD0FAF3"}, "content": " Contenu de message de test”}
- 28. > Datagram Ho : You can have also have a stream of headers … 4813EDF2-B04E-4B70- AB04-0F9EA456E032 :HEADERS 4813EDF2-B04E-4B70- AB04-0F9EA456E032 5462E738-ABAA-452F- 87E0-FD38AEB9DF81 4813EDF2-B04E-4B70- AB04-0F9EA456E032 81C76676-7B19-428E-85 6D-984BB67287D1 4813EDF2-B04E-4B70- AB04-0F9EA456E032 69DFC711-9D21-4DD6- A51D-C04A7A6E20A9 0 1 2
- 29. > Conclusion If you don’t yet use streams instead of databases, start to use one next Monday (even with JSON and no headers…). If you do already use streams … Well, you know what to do ! ;)
- 33. Bonus :What is a CAS ? A Content Adressable Storage is a specific “key value store” : operations : • store(bytes) -> key • get(key) -> null | bytes rule 1 : key = h(data) h being a cryptographic hash function like md5 or sha1. rule 2 : ∀data get(store(data)) = data Rule 1 and 2 imply : Infinite cacheability and scalability.
- 34. Exemple of architectures CLASSICAL APP APP DB WITH STREAMS APP APP append broadcast
- 35. The broadcast mechanism is equivalent to Exemple of architectures a db replication mechanism. CLASSICAL APP REPLICATION (BIN/LOG) APP APP DB DB WITH STREAMS APP APP APP append broadcast