Kafka Is originally from Linkedin Companies to develop , It's a distribution 、 The partition 、 More copies of the 、 More subscribers , be based on zookeeper Coordinated distributed messaging systems ,Linkedin On 2010 Years have been devoted to Apache The foundation has also become a top open source project .
2 Introduction to message system
A messaging system is responsible for passing data from one application to another , Applications only need to focus on data , There's no need to focus on how data is passed between two or more applications . Distributed messaging is based on reliable message queues , Deliver messages asynchronously between client applications and message systems . There are two main messaging patterns ： Point to point transfer mode 、 Release - A subscription model . Most of the information systems choose to publish - A subscription model .Kafka It's a kind of release - A subscription model .
2.1 Point to point transfer mode
The producer sends a message to queue, Only one consumer can receive , The schematic diagram of this architecture is as follows ：
2.2 Release - Subscribe to messaging mode
Publisher sent to topic The news of , Only subscription topic That's the message to subscribers of , The schematic diagram of this architecture is as follows ：
3 Kafka HA Design analysis
3.1 How to put all Replica Evenly distributed throughout the cluster
For better load balancing ,Kafka Try to Partition Evenly distributed across the cluster . A typical deployment is a Topic Of Partition The number is larger than Broker The number of . At the same time, in order to improve Kafka The ability of fault tolerance , Also need to put the same Partition Of Replica Try to spread it to different machines . actually , If all Replica All in the same Broker On , Once it's time to Broker Downtime , The Partition All of the Replica Can't work , It can't reach HA The effect of . meanwhile , If a Broker It's down. , You need to make sure that the load on it can be evenly distributed to all the others that survive Broker On .
Kafka Distribute Replica The algorithm is as follows ：
1. Will all Broker（ Assume that a total of n individual Broker） And to be assigned Partition Sort
2. Will be the first i individual Partition Assign to （i mod n） individual Broker On
3. Will be the first i individual Partition Of the j individual Replica Assign to （(i + j) mode n） individual Broker On
3.2 Message delivery synchronization strategy
Producer When publishing a message to a Partition when , Through the first ZooKeeper To find the Partition Of Leader, And then whatever Topic Of Replication What's the quantity ,Producer Only send this message to this Partition Of Leader.Leader The message is written to its local Log. Every Follower From the Leader pull data . In this way ,Follower The order of stored data and Leader bring into correspondence with .Follower After receiving the message and writing it Log after , towards Leader send out ACK. once Leader received ISR All in Replica Of ACK, The news is thought to have commit 了 ,Leader towards Producer send out ACK（ Premise message producer send The message is settings ack=all or -1）.
To improve performance , Every Follower After receiving the data, immediately send it to Leader send out ACK, Instead of waiting for data to be written Log in . therefore , For already commit The news of ,Kafka It can only be guaranteed that it is stored in multiple Replica The memory of the , There is no guarantee that they will be persisted to disk , There is no guarantee that the message will be able to be Consumer consumption .
Consumer Reading news is also from Leader Read , Only by commit It's not until you've heard the news that you've heard Consumer.
Kafka Replication The data flow of is shown in the figure below ：
4 High Availability Mechanism
4.1 kafka copy
All replicas in a partition are collectively referred to as AR（Assigned Repllicas） Stored in zk in , All and leader A replica that maintains a certain degree of synchronization （ Include Leader） form ISR（In-Sync Replicas）,ISR A set is AR A subset of a set
1、leader copy ： Respond to the client's read / write request
2、follow copy ： Backup leader The data of , No read and write operations
3、ISR Set table ：leader Copies and all that can be associated with leader The replica is basically synchronized follow copy , Such as fruit follow Copies and leader Replica data synchronization is too slow ( The message difference exceeds replica.lag.max.messages threshold ) Or stuck nodes ( Heart rate lost more than replica.lag.time.max.ms threshold ), The follow Will be T Out ISR Set table
ISR The condition that a copy in a set table must satisfy
The node where the replica is located is the same as zk Connected to a
The last message of the copy and leader The difference between the last message of the replica cannot exceed the threshold replica.lag.time.max.ms If it's time to follower Did not catch up in this interval leader, Then follower Will be T Out ISR
4.2 kafka Fault tolerance by two means
The data backup ： With partition Backup for units , The number of copies can be set . When the number of copies is N when , representative 1 individual leader,N-1 individual followers,followers Can be regarded as leader Of consumer, Pull leader The news of ,append Into your own system
When leader be in Out of sync when , System from followers Middle election new leader( kakfa In a lightweight way ： from broker Choose one of the clusters as controller, This controller The monitor hangs up broker, For the sake of broker above leader Select the primary partition again )
When a follower The status changes to Out of sync when ,leader Will follower To eliminate ISR, When this follower Resume and complete data synchronization and then enter again ISR
5 kafka High throughput mechanism
5.1、 Zero copy
Through this kind of “ Zero copy ” The mechanism of ,Page Cache combination sendfile Method ,Kafka The performance of the consumer side has also been greatly improved . This is why sometimes the consumer side is constantly consuming data , We don't see disks io Relatively high , At this point, it's the operating system cache that provides data
kafka Medium topic The content in can be divided into multiple points partition There is , Every partition Are subject to segment Sequential storage can improve the efficiency of reading and writing , So each operation is aimed at a small part of the operation , It's light , And add Parallel operation The ability of
5.3、 Bulk delivery
kafka Allow to send messages in bulk ,producter When sending a message , Messages can be cached locally , Wait until the fixed conditions are sent to kafka
Wait for the number of messages to reach a fixed number
Once over a period of time
5.4、 data compression
Kafka It also supports compression of message collections ,Producer Can pass GZIP or Snappy The format compresses the message set The advantage of compression is to reduce the amount of data transferred , Reduce the pressure on network transmission
6 kafka Prevent message loss and duplication
6.1 ack brief introduction
Kafka Of ack Mechanism , refer to producer Message sending confirmation mechanism of , This directly affects Kafka Throughput and message reliability of the cluster .
ack Yes 3 Optional values , Namely 1,0,-1,ack The default value of is 1
ack=1, In a nutshell ,producer Just receive one leader If the notification is successfully written, the push message is considered successful .
ack=0, In a nutshell ,producer Send once and never send again , Whether or not the transmission is successful
ack=-1, In a nutshell ,producer The push message is considered successful only after receiving the notification of successful writing of all copies in the partition .
6.1 Producers lose and repeat
6.2 Consumers lose and repeat
7 kafka How can messages be sent to Partition Partition
7.1 The producer specifies a partition
7.1 kafka The content of the message contains key-value Key value pair ,key One of its functions is to determine where the message partition is located . By default , kafka It's using hash The partition algorithm of modulus is hash(key) % partitions.size
7.2 It's not specified partition either key be ”metadata.max.age.ms” The polling algorithm selects a
8 kafka How the message is determined Partition Partition to Consumer
If all consumer instances are in the same consumer group , Message logging is load balanced to every consumer instance .
If all consumer instances are in different consumer groups , Each message record is broadcast to all consumer processes in the consumer group .
Pictured , This Kafka There are two in the cluster server Of , Four zones (p0-p3) And two consumer groups . Consumer groups A There are two consumers , Consumer groups B There are four consumers .
c1、c2 In a group A The consumption of different P, Again group B In the same way , Ensure that a node lost in consumption can be consumed normally
9 kafka Strong dependence on ZooKeeper How it works