Oleksandr Tarasenko "Using Kafka in your python applications"

Using Kafka in your
python application
Oleksandr Tarasenko
EVO Pay / EVO.company

Agenda
● intro to kafka
● kafka and other solutions
● kafka + python
● EVO solutions with kafka
4

What is Apache Kafka
A Distributed Streaming Platform
● Publish & Subscribe
● Process streams
● Store data
6

What is Apache Kafka
A Distributed Streaming Platform
● Publish & Subscribe
● Process streams
● Store data
● Queues ?
8

Kafka vs ???
● kafka vs rabbitmq
● kafka vs nsq
● kafka vs jms
● kafka vs nuts
● kafka vs pulsar
● kafka vs rest ?
● kafka vs akka ??
12

Apache Kafka good in
● Event sourcing
● Stream processing
● Messaging
● Log aggregation
● Activity tracking
● ...
14

Kafka Ecosystem
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

Kafka Ecosystem
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
Stream processing
Packaging
Distributions
Kafka connect
Hadoop integration
Database integration
Metrics
Management consoles
Logging
AWS integration
Kafka Camel Integration
Packing and Deployment

Kafka in python world
● kafka-python
● confluent-kafka-python
● pykafka
● aiokafka
● faust*
18

Kafka in python world
● kafka-python (sync)
● confluent-kafka-python (sync)
● pykafka (dead)
● aiokafka (async)
● faust* (async)
19

kafka-python
https://github.com/dpkp/kafka-python
https://kafka-python.readthedocs.io/
● pythonic
● work with old kafka versions

kafka-python
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:1234')
...
for product_data in some_data_source:
producer.send('product_topic', product_data)
future = producer.send('product_topic', b'product_data')
result = future.get(timeout=60)
producer.flush()
…
producer.send('product_topic', key=b'product_category_id', value=b'product_data')
21

kafka-python
from kafka import KafkaConsumer
consumer = KafkaConsumer('product_topic')
for msg in consumer:
print(msg)
...
consumer = KafkaConsumer('product_topic', group_id='discount_product_group')
for msg in consumer:
print(msg)
22

confluent-kafka-python
https://github.com/confluentinc/confluent-kafka-python
https://docs.confluent.io/current/clients/confluent-kafka-python/index.html
● librdkafka
● faster
● thread safe

from confluent_kafka import Producer
p = Producer({'bootstrap.servers': 'localhost:1234'})
def delivery_report(err, msg):
if err is not None:
print('Message delivery failed: {}'.format(err))
else:
print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))
for data in some_data_source:
p.poll(0)
p.produce('product_topic', data.encode('utf-8'), callback=delivery_report)
p.flush()
24

from confluent_kafka import Consumer
c = Consumer({
'bootstrap.servers': 'localhost:1234',
'group.id': 'discount_product_group',
'auto.offset.reset': 'earliest'
})
c.subscribe(['product_topic'])
while True:
msg = c.poll(1.0)
if msg is None:
continue
if msg.error():
print("Consumer error: {}".format(msg.error()))
continue
print('Received message: {}'.format(msg.value().decode('utf-8')))
c.close() 25

aiokafka
https://github.com/aio-libs/aiokafka
https://aiokafka.readthedocs.io/en/stable/
● aio-libs
● async/await model
● based on kafka-python
● ukrainian developers

aiokafka
from aiokafka import AIOKafkaProducer
import asyncio
loop = asyncio.get_event_loop ()
async def send_one():
producer = AIOKafkaProducer (
loop=loop, bootstrap_servers ='localhost:1234' )
await producer.start()
try:
await producer.send_and_wait("product_topic" , b"product_data" )
finally:
await producer.stop()
loop.run_until_complete (send_one())
27

aiokafka
from aiokafka import AIOKafkaConsumer
import asyncio
async def consume():
consumer = AIOKafkaConsumer (
'product_topic' , 'other_topic',
loop=loop, bootstrap_servers ='localhost:1234' ,
group_id ="discount_product_group" )
await consumer.start()
try:
async for msg in consumer:
print("consumed: ", msg.topic, msg.partition, msg.offset,
msg.key, msg.value, msg.timestamp)
finally:
await consumer.stop()
loop.run_until_complete (consume())
28

faust
https://github.com/robinhood/faust
https://faust.readthedocs.io/en/latest/
● python 3.6+, async/await
● like framework
● standalone
● kafka streams
● celery*

faust
import faust
app = faust.App('myapp', broker='kafka://localhost:1234')
class Product(faust.Record):
product_id: str
price: int
@app.agent(value_type=Product)
async def product(products):
async for product in products:
print(f'Product for {product.product_id}: {product.price}')
30

faust
import faust
class Product(faust.Record):
product_id: str
price: int
app = faust.App('hello-app', broker='kafka://localhost:1234')
topic = app.topic('hello-topic', value_type=Product)
@app.agent(topic)
async def example_source(products):
async for product in products:
print(f'Hello from {product.product_id} to {product.price}')
@app.timer(value_type=1.0)
async def example_producer(app):
await example_source.send(
value=Product(product_id='123zxc', price=100),
)
if __name__ == '__main__':
app.main()
31

Example 1 - Products indexation

Example scheme with python-kafka

Example 2 - Delivery data stream

Example scheme with aiokafka + faust

Oleksandr Tarasenko "Using Kafka in your python applications"

More Related Content

What's hot (19)

Similar to Oleksandr Tarasenko "Using Kafka in your python applications" (20)

More from Fwdays (20)

Recently uploaded (20)

Oleksandr Tarasenko "Using Kafka in your python applications"