Lessons learned form Kafka in production (Tim Berglund, Confluent)

Here is the link.

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support. We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more int his overview of common Kafka production issues.

Outline
1. A reminder about streams
2. Apache kafka review
3. Strange happenings with partitions
4. Automated liveness check
5. Adding a broker hurts!

Streaming platform

Event-centric thinking

Mobile    ---------->
Web app   --------->    streaming platform  ->  Rec engine, security, hadoop, real-time analytics
API          ---------->

Forward compatible

Outline

Producers -> Karfka cluster -> consumers

Logs
0 1 2 3 4 5 6 7 8 9  <-------------  next write
            |___  reader

KAFKA Topic = Partitioned Log

Order is only in each partition. But ......

API level

30:44

Partition?
. String consistency
. Leader
. Follower(s)
. Controller

three replications, 4 partitions


34:56
In-sync replicas
.Replica can fall behind
.ISR list grows, shrinks
.What if it gets too small?

Moral:
Waht your ISR list!

41:23
. > partitions, > throughput
. Plan ahead
. Clean leader shutdown
. Leader failure

Moral
Automation is a sharp knife

Auto data balancing

Actionable Items

I really enjoy the learning from the presentation. I like to watch a few more videos so that I can learn the rough idea how to manage my preparation.