End-to-end Streaming Between gRPC Services Via Kafka with John Fallows
1 like4,493 views
This document discusses integrating gRPC services with Kafka to enable end-to-end streaming between gRPC services. It describes how the open source Zilla library can translate between gRPC and Kafka protocols to allow gRPC requests and responses to be sent over Kafka topics. This integration enables features like load balancing, edge caching, and elastic scalability for gRPC services. Examples of using Zilla to implement request-response streaming and server streaming patterns between gRPC and Kafka are provided.
End-to-end Streaming Between gRPC Services Via Kafka with John Fallows
- 1. End-to-end Streaming Between gRPC Services via Kafka John Fallows
- 2. 2 From Northern Ireland, UK California, USA Now Experience John Fallows CTO @ Aklivity Serial Entrepreneur HTML5 Contributor WebSocket Pioneer 25+ years Java Author Pro JSF & Ajax JMS, Kafka, gRPC, MQTT, … grpc-kafka who am i?
- 3. grpc-kafka what is grpc? ❖ open source RPC (Remote Procedure Call) framework ❖ protocol buffers for service definitions package example; service EchoService { rpc EchoUnary(EchoMessage) returns (EchoMessage); } message EchoMessage { string message = 1; }
- 4. grpc-kafka why grpc? ❖ strong contract for service definitions ❖ code generation for many languages ❖ protobuf over http/2 optimizes communication ❖ unified http/2 connection per client
- 5. grpc-kafka grpc challenges ❖ load balancing, client vs proxy, connection vs rpc ❖ limited browser support, requires proxy ❖ binary message format is not human readable ❖ lack of edge caching
- 6. grpc-kafka compare grpc vs kafka service service service ⚙ service ⚙ service ⚙ service ❖ contract ❖ coupling ❖ communication ❖ codec
- 7. grpc-kafka integrate grpc + kafka (open source zilla) Every network and application data flow is a stream. Streams are composed to create efficient protocol translation pipelines. shared memory tcpin tlsin httpin grpcin kafkaout tcpout ● Garbage free ● Wait free ● Elastic scalability ● Sliding window flow control ● Multi-core CPU friendly ● Extensible by design ⚙
- 8. grpc-kafka unary rpc package example; service EchoService { rpc EchoUnary(EchoMessage) returns (EchoMessage); rpc EchoBidiStream(stream EchoMessage) returns (stream EchoMessage); } message EchoMessage { string message = 1; }
- 9. server (unary) grpc-kafka unary rpc grpc kafka kafka grpc zilla zilla kafka client (unary) POST POST (correlated)
- 10. grpc-kafka bidi streaming rpc package example; service EchoService { rpc EchoUnary(EchoMessage) returns (EchoMessage); rpc EchoBidiStream(stream EchoMessage) returns (stream EchoMessage); } message EchoMessage { string message = 1; }
- 11. server (bidi) grpc-kafka bidi streaming rpc grpc kafka kafka grpc zilla zilla kafka client (bidi) POST POST
- 12. demo grpc end-to-end streaming via kafka
- 13. grpc-kafka integration impact (observe) HEADERS :method POST :path /example.EchoService/EchoBidiStream content-type: application/grpc 1: {"Hello, world"} topic: echo-requests zilla:service example.EchoService zilla:method EchoBidiStream zilla:reply-to echo-responses zilla:correlation-id <UUID> 1: {"Hello, world"} null 1: {"Hello, again"} 1: {"Hello, again"} EOS HEADERS :status 200 content-type: application/grpc 1: {"Hello, world"} 1: {"Hello, again"} topic: echo-responses zilla:service example.EchoService zilla:method EchoBidiStream zilla:correlation-id <UUID> 1: {"Hello, world"} null 1: {"Hello, again"} TRAILERS :grpc-status 0 HEADERS :method POST :path /example.EchoService/EchoBidiStream content-type: application/grpc 1: {"Hello, world"} 1: {"Hello, again"} EOS HEADERS :status 200 content-type: application/grp 1: {"Hello, world"} 1: {"Hello, again"} TRAILERS :grpc-status 0
- 14. grpc-kafka integration impact (reach) topic: echo-requests zilla:service example.EchoService zilla:method EchoBidiStream zilla:reply-to echo-responses zilla:correlation-id <UUID> 1: {"Hello, world"} null 1: {"Hello, again"} HEADERS :status 200 content-type: application/grpc 1: {"Hello, world"} 1: {"Hello, again"} topic: echo-responses zilla:service example.EchoService zilla:method EchoBidiStream zilla:correlation-id <UUID> 1: {"Hello, world"} null 1: {"Hello, again"} TRAILERS :grpc-status 0 HEADERS :method POST :path /example.EchoService/EchoBidiStream content-type: application/grpc 1: {"Hello, world"} 1: {"Hello, again"} EOS event driven micro service
- 15. grpc-kafka integration impact (migrate) HEADERS :method POST :path /example.EchoService/EchoBidiStream content-type: application/grpc 1: {"Hello, world"} topic: echo-requests zilla:service example.EchoService zilla:method EchoBidiStream zilla:reply-to echo-responses zilla:correlation-id <UUID> 1: {"Hello, world"} null 1: {"Hello, again"} 1: {"Hello, again"} EOS topic: echo-responses zilla:service example.EchoService zilla:method EchoBidiStream zilla:correlation-id <UUID> 1: {"Hello, world"} null 1: {"Hello, again"} HEADERS :status 200 content-type: application/grp 1: {"Hello, world"} 1: {"Hello, again"} TRAILERS :grpc-status 0 event driven micro service
- 16. grpc-kafka beyond request-response ❖ grpc server streaming from kafka (fanout) ❖ grpc server streaming from kafka (fanout, reliable) ❖ grpc client streaming to kafka (oneway) ❖ grpc client streaming to kafka (oneway, reliable)
- 17. grpc-kafka server streaming (fanout) package example; import "google/protobuf/empty.proto"; service FanoutService { rpc FanoutServerStream(google.protobuf.Empty) returns (stream FanoutMessage); } message FanoutMessage { string message = 1; }
- 18. grpc-kafka server streaming (fanout) grpc kafka zilla kafka client (server streaming) POST event driven micro service
- 19. grpc-kafka server streaming (fanout, reliable) HEADERS :method POST :path /example.FanoutService/… content-type: application/grpc EOS HEADERS :status 200 content-type: application/grpc 1: {"Hello, world"} 32767: 0xfda8c… grpc kafka zilla kafka client 1: {"Hello, again"} 32767: 0xe1ac… ✖
- 20. grpc-kafka server streaming (fanout, recovery) HEADERS :method POST :path /example.FanoutService/… content-type: application/grpc last-message-id-bin: base64(0xfda8c…) EOS HEADERS :status 200 content-type: application/grp grpc kafka zilla kafka client 1: {"Hello, again"} 32767: 0xe1ac… ✔
- 21. demo grpc reliable server streaming via kafka
- 22. grpc-kafka zilla examples ❖ grpc.kafka.echo Echoes messages sent to a Kafka topic via gRPC from a gRPC client ❖ grpc.kafka.proxy Correlates gRPC requests and responses over separate Kafka topics ❖ grpc.kafka.fanout Streams messages published to a Kafka topic, applying conflation based on log compaction ❖ … github.com/aklivity/zilla-examples
- 24. www.aklivity.io github.com/aklivity/zilla ★ Q & A pssst…we have 😎 socks at Booth S11 Stream Data Anywhere.