MessagePack(msgpack): Compact and Fast Serialization Library
3 likes5,489 views
MessagePack is a compact and efficient serialization library that offers advantages over JSON, including smaller size, binary coding, and better handling of binary data. It allows easy interexchange between different programming languages and supports a variety of types, including composite data structures. The document also discusses specifics about packing, unpacking, and zero-copy deserialization methods within MessagePack.
MessagePack(msgpack): Compact and Fast Serialization Library
- 1. MessagePack(msgpack): A Compact and Fast Serialization Library Takatoshi Kondo 2015/5/14 1Copyright OGIS-RI 2015
- 2. • Taka (Takatoshi Kondo) – from TOKYO, JAPAN • OGIS-RI Co., Ltd. – Developing Pub/Sub IoT Platform using MessagePack • A committer on the msgpack-c OSS project • Other experience: Wrote the “Boost.MSM Guide” – http://redboltz.wikidot.com/boost-msm-guide 2015/5/14 Copyright OGIS-RI 2015 2 About me
- 3. What is MessagePack? 2015/5/14 Copyright OGIS-RI 2015 3
- 4. • Same as JSON – Portable – Contain basic type information • For example, map, array, string, boolean... – Composite data structure • Different from JSON – Smaller size – Binary coded – Can handle binary data without text encoding such as Base64 – Easier to parse, requires less computing power 2015/5/14 Copyright OGIS-RI 2015 4 MessagePack vs. JSON
- 5. • Same as JSON – Portable – Contain basic type information • For example, map, array, string, boolean... – Composite data structure • Different from JSON – Smaller size – Binary coded – Can handle binary data without text encoding such as Base64 – Easier to parse, requires less computing power 2015/5/14 Copyright OGIS-RI 2015 5 MessagePack vs. JSON msgpack::object nil boolean u64 i64 f64 str bin ext array map * * double, float object_kv key val
- 6. MessagePack/Boost.Serialization 2015/5/14 6Copyright OGIS-RI 2015 • The goals of the msgpack and the Boost.Serialization are different. – msgpack: Interexchange the data with different programming languages. – Boost.Serialization: Serialize every data and relations. e.g.) shared_ptr • msgpack can be used as a portable binary archive of the Boost.Serialization. – https://github.com/redboltz/rui/tree/support_boost_1_57_0 • Forked from Norihisa Fujita’s implementation C++ msgpack ruby python shared_ptr shared_ptr shared_ptr 42 42 Boost.Serialization format
- 7. Supported Programming Languages 2015/5/14 Copyright OGIS-RI 2015 7
- 8. Pack/Unpack 2015/5/14 8 #include <tuple> #include <string> #include <sstream> #include <msgpack.hpp> int main() { auto t1 = std::make_tuple("hello", true, 42, 12.3); std::stringstream ss; msgpack::pack(ss, t1); msgpack::unpacked unpacked = msgpack::unpack(ss.str().data(), ss.str().size()); msgpack::object obj = unpacked.get(); auto t2 = obj.as<std::tuple<std::string, bool, int, double>>(); assert(t1 == t2); } You can use any types that have the member function write(const char*, std::size_t); Copyright OGIS-RI 2015 packing unpacking convering
- 9. Stream Deserialization 2015/5/14 9 class unpacker { public: // Constructor is omitted in this presentation void reserve_buffer(std::size_t size); char* buffer(); void buffer_consumed(std::size_t size); bool next(unpacked& result); }; Copyright OGIS-RI 2015
- 10. Stream Deserialization 2015/5/14 10Copyright OGIS-RI 2015 std::size_t const try_read_size = 100; msgpack::unpacker unp; while (/* block until input becomes readable */) { unp.reserve_buffer(try_read_size); std::size_t actual_read_size = input.readsome( unp.buffer(), try_read_size); unp.buffer_consumed(actual_read_size); msgpack::unpacked result; // MessagePack data loop while(unp.next(result)) { msgpack::object obj(result.get()); } }
- 11. Stream Deserialization 2015/5/14 11Copyright OGIS-RI 2015 std::size_t const try_read_size = 100; msgpack::unpacker unp; while (/* block until input becomes readable */) { unp.reserve_buffer(try_read_size); std::size_t actual_read_size = input.readsome( unp.buffer(), try_read_size); unp.buffer_consumed(actual_read_size); msgpack::unpacked result; // MessagePack data loop while(unp.next(result)) { msgpack::object obj(result.get()); } } The size may decided by receive performance, transmit layer's protocol and so on.
- 12. Stream Deserialization 2015/5/14 12Copyright OGIS-RI 2015 std::size_t const try_read_size = 100; msgpack::unpacker unp; while (/* block until input becomes readable */) { unp.reserve_buffer(try_read_size); std::size_t actual_read_size = input.readsome( unp.buffer(), try_read_size); unp.buffer_consumed(actual_read_size); msgpack::unpacked result; // MessagePack data loop while(unp.next(result)) { msgpack::object obj(result.get()); } } The size may decided by receive performance, transmit layer's protocol and so on. unp has at least try_read_size buffer on this point.
- 13. Stream Deserialization 2015/5/14 13Copyright OGIS-RI 2015 std::size_t const try_read_size = 100; msgpack::unpacker unp; while (/* block until input becomes readable */) { unp.reserve_buffer(try_read_size); std::size_t actual_read_size = input.readsome( unp.buffer(), try_read_size); unp.buffer_consumed(actual_read_size); msgpack::unpacked result; // MessagePack data loop while(unp.next(result)) { msgpack::object obj(result.get()); } } The size may decided by receive performance, transmit layer's protocol and so on. unp has at least try_read_size buffer on this point. input is a kind of I/O library object. read message to msgpack::unpacker's internal buffer directly.
- 14. Stream Deserialization 2015/5/14 14Copyright OGIS-RI 2015 std::size_t const try_read_size = 100; msgpack::unpacker unp; while (/* block until input becomes readable */) { unp.reserve_buffer(try_read_size); std::size_t actual_read_size = input.readsome( unp.buffer(), try_read_size); unp.buffer_consumed(actual_read_size); msgpack::unpacked result; // MessagePack data loop while(unp.next(result)) { msgpack::object obj(result.get()); } } The size may decided by receive performance, transmit layer's protocol and so on. unp has at least try_read_size buffer on this point. input is a kind of I/O library object. read message to msgpack::unpacker's internal buffer directly. notify msgpack::unpacker actual consumed size.
- 15. Stream Deserialization 2015/5/14 15Copyright OGIS-RI 2015 std::size_t const try_read_size = 100; msgpack::unpacker unp; while (/* block until input becomes readable */) { unp.reserve_buffer(try_read_size); std::size_t actual_read_size = input.readsome( unp.buffer(), try_read_size); unp.buffer_consumed(actual_read_size); msgpack::unpacked result; // MessagePack data loop while(unp.next(result)) { msgpack::object obj(result.get()); } } The size may decided by receive performance, transmit layer's protocol and so on. unp has at least try_read_size buffer on this point. input is a kind of I/O library object. read message to msgpack::unpacker's internal buffer directly. notify msgpack::unpacker actual consumed size. Use obj. convert to C++ types All complete msgpack message is processed at this point, then continue to read addtional message.
- 16. Zero-Copy Deserialization 2015/5/14 16Copyright OGIS-RI 2015 unpacker buffer int string bin header payload header payload array client memory msgpack memory
- 17. Zero copy deserialization 2015/5/14 17Copyright OGIS-RI 2015 zone chunk_list finalizer_array object (array) object (int) object (string) object (bin) via.array.ptr chunk unpacker buffer int string bin header payload header payload via.str.ptr via.bin.ptr array (msgpack format) client memory msgpack memory managed using reference counter unpacked copy or reference is selectable
- 18. Zero copy deserialization 2015/5/14 18Copyright OGIS-RI 2015 zone chunk_list finalizer_array object (array) object (int) object (string) object (bin) via.array.ptr chunk unpacker buffer int string bin header payload header payload via.str.ptr via.bin.ptr array (msgpack format) client memory msgpack memory managed using reference counter unpacked convert Convert to any types that adapts MessagePack std::tuple<int, boost::string_ref, std::vector<char>> copyref copy or reference is selectable
- 19. MessagePack Adaptors 2015/5/14 19Copyright OGIS-RI 2015 https://github.com/msgpack/msgpack-c/wiki/v1_1_cpp_adaptor C++ type msgpack::object type bool bool char* str std::deque array char positive/negative integer signed ints *1 positive/negative integer unsigned ints *2 positive integer std::list array std::map array std::pair array std::set array std::string str std::vector array std::vector<char> bin *1 signed ints signed char, signed short, signed int, signed long, signed long long *2 unsigned ints unsigned char, unsigned short, unsigned int, signed long, signed long long C++11 type msgpack:: object type std::array array std::array<char> bin std::forward_list array std::tuple array std::array array std::unordered_map array std::unordered_set array boost type msgpack:: object type boost::optional<T> T boost::string_ref str
- 20. MessagePack Adaptors 2015/5/14 20Copyright OGIS-RI 2015 https://github.com/msgpack/msgpack-c/wiki/v1_1_cpp_adaptor #include <msgpack.hpp> struct your_class : base1, base2 { int a; std::string b; // You can choose any order. // It is represented to the msgpack array elements order. MSGPACK_DEFINE(a, b, MSGPACK_BASE(base1), MSGPACK_BASE(base2)); };
- 21. • If you have questions, feel free to contact me :) • Takatoshi Kondo – redboltz@gmail.com – twitter: redboltz • Resources – MessagePack • http://msgpack.org/ – msgpack-c • https://github.com/msgpack/msgpack-c – msgpack-c Documents • https://github.com/msgpack/msgpack-c/wiki – msgpack-c stream unpack algorithm • A little old but concept is the same • http://www.slideshare.net/taka111/msgpackc 2015/5/14 Copyright OGIS-RI 2015 21 Thank you
- 22. Extra Slides 2015/5/14 Copyright OGIS-RI 2015 22
- 23. Format name first byte (in binary) first byte (in hex) positive fixint 0xxxxxxx 0x00 - 0x7f fixmap 1000xxxx 0x80 - 0x8f fixarray 1001xxxx 0x90 - 0x9f fixstr 101xxxxx 0xa0 - 0xbf nil 11000000 0xc0 (never used) 11000001 0xc1 false 11000010 0xc2 true 11000011 0xc3 bin 8 11000100 0xc4 bin 16 11000101 0xc5 bin 32 11000110 0xc6 ext 8 11000111 0xc7 ext 16 11001000 0xc8 ext 32 11001001 0xc9 float 32 11001010 0xca float 64 11001011 0xcb uint 8 11001100 0xcc uint 16 11001101 0xcd uint 32 11001110 0xce uint 64 11001111 0xcf2015/5/14 Copyright OGIS-RI 2015 23 MessagePack Formats https://github.com/msgpack/msgpack/blob/master/spec.md Format name first byte (in binary) first byte (in hex) int 8 11010000 0xd0 int 16 11010001 0xd1 int 32 11010010 0xd2 int 64 11010011 0xd3 fixext 1 11010100 0xd4 fixext 2 11010101 0xd5 fixext 4 11010110 0xd6 fixext 8 11010111 0xd7 fixext 16 11011000 0xd8 str 8 11011001 0xd9 str 16 11011010 0xda str 32 11011011 0xdb array 16 11011100 0xdc array 32 11011101 0xdd map 16 11011110 0xde map 32 11011111 0xdf negative fixint 111xxxxx 0xe0 - 0xff
- 24. Format name first byte (in binary) first byte (in hex) positive fixint 0xxxxxxx 0x00 - 0x7f fixmap 1000xxxx 0x80 - 0x8f fixarray 1001xxxx 0x90 - 0x9f fixstr 101xxxxx 0xa0 - 0xbf nil 11000000 0xc0 (never used) 11000001 0xc1 false 11000010 0xc2 true 11000011 0xc3 bin 8 11000100 0xc4 bin 16 11000101 0xc5 bin 32 11000110 0xc6 2015/5/14 Copyright OGIS-RI 2015 24 MessagePack Formats https://github.com/msgpack/msgpack/blob/master/spec.md
- 25. 2015/5/14 Copyright OGIS-RI 2015 25 MessagePack Format https://github.com/msgpack/msgpack/blob/master/spec.md#int-format-family
- 26. 2015/5/14 Copyright OGIS-RI 2015 26 MessagepPack format
- 27. What is MessagePack? 2015/5/14 Copyright OGIS-RI 2015 27