![Schema
Avro data is always serialized with its schema.
Writer
{
"type": "record",
"name": "Tweet",
"namespace": "com.avro.serialization.gen",
"fields": [
{
"name": "userId",
"type": "int"
},
{
"name": "userName",
"type": "string"
},
{
"name": "text",
"type": "string"
}
]
}
Reader
{
"type": "record",
"name": "Tweet",
"namespace": "com.avro.serialization.gen",
"fields": [
{
"name": "userIcon",
"type": "string"
},
{
"name": "userName",
"type": "string"
},
{
"name": "text",
"type": "string"
}
]
}
Name=Nathan,
userName: ran_than,
text: Hello
userName: ran_than,
text: Hello](https://image.slidesharecdn.com/apachebigdata2016speakingthelanguageofbigdata-160530015232/85/Apache-big-data-2016-Speaking-the-language-of-Big-Data-38-320.jpg)
Apache big data 2016 - Speaking the language of Big Data
- 1. Speaking the language of Big Data Ranganathan Balashanmugam, ThoughtWorks @ran_than the data interchange protocols
- 2. About Me ● Graduated as Civil engineer ● Technology Lead at ThoughtWorks, India ● Organizer of Hyderabad Scalability Meetup with ~3500 members.
- 3. “BigData because every byte has a story.”
- 4. “BigData because every byte has a story.” ➔ Write everything (no write schema) ➔ Read required content
- 5. Why in big data?
- 9. Expected language of Big Data
- 10. language
- 11. schema
- 12. Text Files “Written once; Processed many times”
- 13. Text Files ➔ Good at writes. ➔ Reads needs processing, every time.
- 14. Serialized Deserialized ● Locked to language ● Heavy ● Not flexible Java serialization, Ruby’s marshal, Python’s pickle, etc
- 15. Demo
- 16. CSV, XML, JSON, etc “Schema defined”
- 17. CSV, XML, JSON, etc ➔ CSV - schema not present with data ➔ Parsing efficiency ➔ communication ➔ flexibility ➔ data types ➔ rewrite Schema change wars
- 18. “Language and platform neutral way to store and exchange structured data in distributed systems.”
- 19. Expectations
- 20. Expectations ❏ Simple, readable schema ❏ Rich data structures, easy design to description ❏ Support multiple languages, with simple integration, code generation ❏ Efficiency (time/space) ❏ Easy integrations (availability of libraries) ❏ Tooling ❏ Versioning ❏ Old version, new server ❏ New version, old server ❏ Reduce boilerplate code ❏ Supports compression
- 21. Schema Python Java Other languages Less memory More Data; Less memory; faster Manual Design and Describe Auto Generated boilerplate code Auto Serialize Deserialize Compress Agile Flexible schema No code rewrite Stay calm and focusTooling Simple commands
- 22. “Language and platform neutral way to store and exchange structured data in distributed systems.”
- 23. Tada
- 24. Data Interchange Protocols ➔ Protocol Buffers ➔ Apache Thrift ➔ Apache Avro ➔ Cap’n Proto ➔ Flatbuffers ➔ Kryo (java only) ➔ MessagePack ➔ BERT ➔ Apache Etch ➔ Internet Communications Engine
- 25. Data Interchange Protocols ➔ Protocol Buffers ➔ Apache Thrift ➔ Apache Avro ➔ Cap’n Proto ➔ Flatbuffers ➔ Kryo (java only) ➔ MessagePack ➔ BERT ➔ Apache Etch ➔ Internet Communications Engine
- 26. Protocol Buffers A flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler.
- 27. Google Protobuf ● Google's own serializer ● Cross-language ● Schema evolution ● Compact ● Strong typed ● Battle tested
- 28. “All the data is just a sequence of bytes, which makes no sense without the encoding.” 0 1 1 0 1 0 0 1 1 0 1 0 0 0 0 1 1 1 1 0 1
- 29. Base 128 varints “Varints are a method of serializing integers using one or more bytes.” ➔ Each byte is a varint ➔ Last byte - most significant bit (msb) - indicated further bytes ➔ Lower 7 bits - two's complement 1 0000 0001 300 1010 1100 0000 0010 010 1100 000 0010 000 0010 010 1100 000 0010 ++ 010 1100 = 100101100
- 30. Non-varints Double, fixed64 - fixed 64bit lump of data
- 31. Message Structure “A protocol buffer message is a series of key-value pairs.” ➔ Key is varint ➔ Key = (field_number << 3) | wire_type Type Meaning 0 Varint 1 64-bit 2 length- delimited …. …. Key = 000 1000 Wire_type = Varint Field_number = right_shift (000 1000, 3) = 1 message Message { required int a = 1; }
- 32. Strings message String { required string a = 2; } a = “testing”; 12 07 74 65 73 74 69 6e 67 Key = HexToBinary(12) = 0001 0010 => Tag = 2, type = 2 Length varint = 7 bytes follows
- 34. Demo
- 35. Apache Avro Avro is a remote procedure call and data serialization framework.
- 36. Avro ➔ Originally developed by Doug Cutting for use with Hadoop. ➔ RPC framework. ➔ Part of Hadoop. ➔ Avro data is always serialized with its schema. ➔ Supports binary and JSON encoding.
- 37. Src: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html Compact of encodings
- 38. Schema Avro data is always serialized with its schema. Writer { "type": "record", "name": "Tweet", "namespace": "com.avro.serialization.gen", "fields": [ { "name": "userId", "type": "int" }, { "name": "userName", "type": "string" }, { "name": "text", "type": "string" } ] } Reader { "type": "record", "name": "Tweet", "namespace": "com.avro.serialization.gen", "fields": [ { "name": "userIcon", "type": "string" }, { "name": "userName", "type": "string" }, { "name": "text", "type": "string" } ] } Name=Nathan, userName: ran_than, text: Hello userName: ran_than, text: Hello
- 39. Avro Encoding ➔ Length followed by UTF-8 bytes. ➔ The parser matches fields in the reader and writer schema by name. ➔ Hadoop has millions of records with same schema. Object Container Files handle this -- schema in the beginning. ➔ Easy to load schema into Pig.
- 40. Avro - Object Container File Header Block 1 Block 2 Block ... Block N 16 bit Sync marker File Metadata: Includes avro.schema and avro.codec 4 bytes: ASCII ‘O’, ‘b’, ‘j’ Count of objects in block Size (in bytes) of serialized objects Serialized objects compressed by specified codec 16 bit Sync marker
- 41. Demo
- 42. Apache Thrift Thrift is a remote procedure call and data serialization framework. and was developed at Facebook for "scalable cross- language services development".
- 43. Thrift ➔ Designed by x-googler in 2007 ➔ Developed internally at Facebook ➔ RPC stack ➔ Many encoding options
- 47. Thrift Protocols ➔ TBinaryProtocol ➔ TCompactProtocol ➔ TDenseProtocol: Similar to TCompactProtocol but strips off the meta information from what is transmitted, and adds it back in at the receiver. ➔ TJSONProtocol ➔ TSimpleJSONProtocol : A write-only protocol using JSON. Suitable for parsing by scripting languages ➔ TDebugProtocol : Uses a human-readable text format to aid in debugging.
- 48. Demo
- 49. “It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change.” - Charles Darwin
- 50. THANK YOU For questions or suggestions: Ranganathan Balashanmugam @ran_than