Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

Stream it together
3 realities of modern programming
Neha Narkhede

3
3 realities of
modern programming

4
managed servicesmicroservicesdata explosion
1 1 1 10 0 0 10 1 10
1 1 1 11 0 0 10 1 01
0 1 1 00 0 0 10 1 00
0 0 1 11 1 0 11 1 10
1 1 0 1
1
1 0 0 10 1 01
1 1 1 00 0
0
0 1 1
0 0 11
0
0
0
0
0
01
1
0 11 1 00
0 1 1
1
1
1 1
1
10 0
0
01
0 10
0 1
1
1
1
0
0
1 1 0 0
1
0
1 11
1 0 0
11
1
0
00
0
0
1
1 0 10
0
0
0
0
0
0
0
1
1
1
1
1
0
1
0
0
0
1 10
1 0
0
11
1 0
0
0
0
0
1 0
0
1 11 11
0 1 1
11
1
1
1
0
0
01
1
1
1
1
0 0 0
0
1
1
11 1
1
1
1
0
0 0
0 1
1
0
0
0
0
0 0
0
1
0
0
11
11
1
1
11
1
1
1
1
1
0 0
0
1
01
1
1
1
0
0
1
0 0
1
0
1
0
0
11
1
0
00
0
1
1
0
0
0
0
0 0
0
1
1
1
1
0
1
0
0
0
10
0
1
1
1
0
0
0 0
0
0
0
1
1
11
1
1
1
1
1
0
0
01
1
1
1
1
0
0
0
111
1
00
1 1 0 11 0 0 10 1 11
1 0 1 10 0 1 01 1 10
0 1 0 10 1 0 10 1 11
1 0 0 11 0 1 10 0 10
1
1 1 0
0
0
0
1 10
0
0
0
0
0
01 0
0
11
1
1
1
1
1
1
1
1
0
0 01 0
1
1
1
1
0
0
1
0 0 1
0
1
0
0
11
1
0
0
0
0
1
1 0
0
0
0
1
1
1
1
0
1
0
0
0
1
0
0
1
11
00
0
0 0
0
0
00
11 1
1
11
1
1
1
1
0
0
0
1
1
1
1
1
1
0 0
0
111
1
1 1 1 10 0 0 10 1 10
1 1 1 11 0 0 10 1 01
0 1 1 00 0 0 10 1 00
0 0 1 11 1 0 11 1 10
1 1 0 11 0 0 10 1 01
1 1 1 00 0 0 10 0 11
0 0 0 01 1 0 11 1 00
0 1 1 10 0 0 01 0 10
0 1 1 01 1 0 10 1 11
1 0 1 00 0 0 11 0 10
0 0 0 11 1 0 10 1 10
1 0 1 01 0 0 11 1 11
0 1 1 10 0 0 11 1 00
1 1 0 11 0 0 10 1 11
1 0 1 10 0 1 01 1 10
0 1 0 10 1 0 10 1 11
1 0 0 11 0 1 10 0 10

7
all your data as a stream
Imagine…

12
But communicating between services can get messy

13
A stream ﬁxes the madness

15
Over 1m lines of code in a single repo
Single monolithic app: yelp-main
2011
Move to micro services; Kafka is the backbone
70+ production services
2013
Saved $10m in R&D from 2013-today
2017 Reduced code complexity and downtime

16
The rise of
Managed services

17
of Kafka users are in the cloud
52%

Running distributed systems is really, really tough
18

19
Coding
Can be pretty tricky

Twice as hard as writing the program
Debugging
20

21
Where do you even start?
Operations

23
Change is inevitable; systems evolve

24
High throughput messaging broker

25
Highly available replicated log

26
Full streaming platform with Connect + Streams

27
Exactly once streaming platform

28

29

30

31

32

33
You can’t ﬁx what you can’t see

34
Observability is a dark art perfected
only with time and experience!

35
Sh!t happens
Figure out observability
Keep up with change

37
Focus on building your ideas - not running Kafka

39
Launch
Signup in
Throughput
Retention 30 days
100 MB/s
3 seconds

40
Not just Kafka, but its entire OSS ecosystem
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
...
Producer<String, String> producer = new KafkaProducer<>(props);
for(int i = 0; i < 100; i++)
producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));
producer.close();
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s", record.offset(), record.key(), record.value());
}

41
No lock-in; freedom to deploy Kafka in
any cloud

Kafka everywhere
On prem, in the cloud
42

43
Have an amazing Kafka Summit!
http://conﬂuent.io/conﬂuent-cloud

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

More Related Content

What's hot (20)

Similar to Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming (20)

More from confluent (20)

Recently uploaded (20)

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming