Building a maintainable bi-directional cross platform protocol

Building a maintainable
bi-directional cross
platform protocol
Pavel Dovbush
William Lewis
@

Background
Solution and Implementation
Examples

● Application Programming Interface
● Operations
● Inputs
● Outputs
● Type definitions
● Independent of implementation
API
API

Requirements
Perfect Client Server API:
● Just works – Magic! :)
● Flexible
● Extensible
● Testable
● Maintainable
● Platform and language neutral
● Focused on features, not bytes over the wire
Requirements

Overview
● Encoding
● Message exchange
● Data access
data
access
Server
Client
exchange data
exchange access
serialise
deserialise
Overview

REST+JSON
PHP
array
Server
Client
URI JS
Ajax Object
JSON
URL
REST + JSON

REST+JSON problems
Message exchange:
● Client: HTTP request/response model
● Server: URI-based, config on web- or app- server
Data access:
● No canonical definition
● No versioning
● Duplicate implementations and configuration
REST + JSON
Encoding:
● Server: JSON
● Client: URLEncode

Badoo APIs
Badoo APIs
Core API
Mobile
API
???

Protocol implementations
SOAP
Protocol Values

Protocol values
● Protocol description
● Encoding
● Data access
● Message exchange
● Versioning
Protocol

Protobuf + Own RPC
Generated
class
Server
Client
Events Generated
Events class
Protobuf
Protobuf
Description, versioning
Usage

Google Protocol Buffers
● Interface description language
● Internal representation
● Language support
○ v2.3 plugin support
● Encoding and network efficiency
Protcol values

Interface description language
Protobuf is self-describing - descriptor.proto
● enum
● message
● field
● service
● option

Label Type Name Number
optional
required
repeated
bool
string
message
enum
float
int32
+ more numeric
field_name = 1;
required string user_name = 1;
optional uint32 age = 2;

enum Role {
ADMIN = 1;
USER = 2;
}
message User {
required string name = 1;
repeated string nickname = 2;
optional uint32 age = 3;
required Role role = 4;
}

Protobuf usage
Encoding:
● Only binary
Message exchange:
● Simple RPC
Data access:
● No support for PHP
● No support for JS
Protobuf compiler plugins

Protobuf v2.3.0 compiler plugins
● Brilliant internal architecture
● Very simple plugin system
● Can generate any code in any language
● IDL is completely separated from serialization part
● gist with example
Protobuf complier plugins

Encoding
Encoding
Core API
Mobile
Native binary API
serialization
Custom JSON
serialization

Code auto-generation
● WebSite DEVEL - On any request if ‘.proto’ file is
newer than generated code - regenerate
● Mobile DEVEL - Grunt task - regenerate on file
change
● PRODUCTION - generate before deploy
Code auto-generation

Message exchange
service SearchService {
rpc Search (SearchRequest)
returns (SearchResponse);
}
● Too simple for a complex application
● We need a wrapper for every Request/Response
● Anytime responses
Message exchange

Message exchange (two-way RPC)
new RPC(request_type, parameter)
.on(response_type, callback)
.on([type1, type2], callback)
.request();
RPC.any.on(type3, callback);
function callback(err, /** Type1 */ response1) {}
request_type & response_type are values of enum
MessageType
parameter & response are Protobuf messages
Message exchange

Versioning
Old clients ignore:
● new fields
● unsupported commands
Migrating to new protocol version:
● cause compilation error on field removal
Versioning

Take aways
● Protocol defined in one place
● Code uses data-access classes
● Validation
● Encoding can vary
● Flexible message exchange
● Versioning
● Any part can be changed without affecting anything
else
Summary

Building a service
● Location search
○ Client query: city name
○ Server response: list of cities
(ID, name, lat, long)
● Client notification
○ Anytime server response: user message
Example

Protobuf definition
enum MessageType {
// body: CityQuery
SERVER_SEARCH_CITIES = 1;
// body: Cities
CLIENT_FOUND_CITIES = 2;
// body : ClientNotification
CLIENT_NOTIFICATION = 3;
}
message ClientNotification {
required string id = 1;
optional string title = 2;
optional string message = 3;
}
message CityQuery {
optional string name = 1;
}
message Cities {
repeated City cities = 1;
}
message City {
required int32 id = 1;
required string name = 2;
optional double longitude = 3;
optional double latitude = 4;
}
Example protobuf

Protobuf definition
message RPCMessage {
required int32 version = 1;
optional int32 message_id = 2;
repeated MessageBody body = 3;
}
message MessageBody {
required MessageType message_type = 1;
optional CityQuery city_query = 2;
optional Cities cities = 3;
optional ClientNotification client_notification = 4;
}
Example protobuf

Generated classes examples
define(['GPB/gpb2'], function(/** $gpb */$gpb) {
var Protocol = $gpb.namespace('Demo');
/**
* CityQuery
* @class {Protocol.CityQuery}
* @extends {$gpb.Message}
*/
var CityQuery = Protocol.CityQuery = function() {
$gpb.Message.apply(this, arguments);
};
$gpb.extend(CityQuery, CityQuery, $gpb.$gpb.Message);
Message);
CityQuery.prototype.$gpb = 'Demo.CityQuery';
CityQuery.prototype._prototype._descriptor descriptor = {"= fields": {"fields": {"name": {"name": {"type": {"type": 9, "number": 9, "number":
1,
1, "label": 1}}};
"label": 1}}};
return CityQuery;
});
Generated class

Generated classes examples
<?php
namespace GPBJSDemo;
class CityQuery GPBJSBaseMessage
{
protected static $name = 'Demo.CityQuery';
extends GPBJSBaseMessage
protected static $fields = array(
'name' => array('type' => 'string', 'optional' => true, 'repeatable' => false,
'hash' => false, 'raw' => False, 'is_enum' => false, 'is_message' => false),
);
static $fields = array(
=> array('type' => 'string', 'optional' => true, 'repeatable' => false, 'hash' =>
false, 'raw' => False, 'is_enum' => false, 'is_message' => false),
public function setName($value)
{
$this->_setFieldValue('name', $value);
return $this;
}
public function getName()
{
return $this->_getFieldValue('name');
}
}
Generated class

RPC city query example
new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) {
if (err) { /* error handling */ return; }
for (var city in cities.getCities()) {
cityListView.update(city.getId(), city.getName());
}
}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {
alert(notification.getTitle() + ‘n’ + notification.getMessage());
}
Example usage

RPC city query example
var cityQuery = new Protocol.CityQuery().setName(‘london’);
new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) {
if (err) { /* error handling */ return; }
for (var city in cities.getCities()) {
cityListView.update(city.getId(), city.getName());
}
}
}
Example usage

RPC anytime response example
RPC.any.on(Protocol.MessageType.CLIENT_NOTIFICATION,
onNotification);
}
Example usage

Building a maintainable bi-directional cross
platform protocol
● REST + JSON
● Protocol values
○ Interface description language
○ Encoding and performance
○ Protobuf compiler plugins
○ Code auto-generation
○ Message exchange (two-way RPC)
○ Versioning
● Examples
Summary

Thanks! Questions?
Pavel Dovbush <dpp@corp.badoo.com>
William Lewis <william.lewis@corp.badoo.com>
@netproteus
Slides: techblog.badoo.com
Thanks:
Google team for Protobuf itself
Andrey Nigmatulin <anight@corp.badoo.com> for bringing GPB in Badoo in 2009

Building a maintainable bi-directional cross platform protocol

More Related Content

What's hot (18)

Similar to Building a maintainable bi-directional cross platform protocol (20)

Recently uploaded (20)

Building a maintainable bi-directional cross platform protocol