编程知识 cdmana.com

The two sides of the headline Kafka face to face summary: understand the problem of Kafka, you are not far from the big factory!! (recommended Collection)

Hello everyone , I'm glacier ~~

lately , Some readers go to the front page , I was asked by the interviewer about Kafka The problem of : Multiple Kafka How consumers consume the same... At the same time Topic It's the same under Partition The data of ? It seems like a simple question , Even confused the reader !

today , Let's talk about this interview question , Okay , Start today's topic .

Topic analysis

First , Be clear about the interviewer's questions : Multiple Kafka How consumers consume the same... At the same time Topic It's the same under Partition The data of ? The question is already obvious , We just have to answer how to make multiple Kafka Consumers consume the same... At the same time Topic It's the same under Partition That's all .

In order to answer this question , We need to understand Kafka A concept in , Namely Consumer group (Consumer Group). The consumer group is Kafka The basis and means of realizing unicast and broadcast message models . For the same Topic( The theme ) Come on , Every consumer group can get this Topic All data in . All consumers in the consumer group work together to subscribe and consume Kafka Topic All sections in . here , Each partition can only be consumed by one consumer in the same consumer group .

here , For better understanding , Let's simply draw one Kafka Schematic diagram of consumption message , As shown below .

 Insert picture description here

In this picture , A theme can be configured with several partitions , Messages sent by producers are distributed to different partitions , When consumers receive data, they receive data according to consumer groups ,Kafka Ensure that messages in each partition can only be consumed by the same consumer in the same consumer group , If you want to repeat consumption , Then you need other consumer groups to consume .Zookeerper Save the consumption of each partition under each topic in each consumer group offset.

new edition kafka Put this offset Saved to a __consumer_offsert Under the theme of . This __consumer_offsert Yes 50 Zones , By linking the consumer group's id Hash value %50 To determine which partition to save to . This is also to take into account Zookeeper The reason why I'm not good at reading and writing a lot of data .

therefore , If you want a consumer group to consume several consumers at the same time Kafka In the news , Multithreading is required to read , A thread is equivalent to a consumer instance . When the number of consumers is greater than the number of partitions , Some consumer threads can't read data .

Expanding knowledge

here , Let's give you an example : Suppose a topic test By groupA The consumption , Now start another new groupB To consume test, Default test-groupB Of Offset No 0, But still Kafka There is no such a Offset, Except when test When the subject has data ,groupB Will receive this data , This data is also the first data , here ,groupB Of Offset Also just initialized Offset, Unless you use explicit –from-beginning To get 0 Starting data .

Question answer

Multiple Kafka If consumers want to consume the same at the same time Topic It's the same under Partition The data of , You need to put these Kafka Consumers are placed in different consumer groups .

Receive benefits

CSDN Private letter glacier :

  • reply 【 Concurrent programming 】 Get the original glacier and download it all over the network 35W+ Of 《 Deep understanding of high concurrency programming 》 e-book .
  • reply 【 Infiltrate notes 】 Get the latest release of Glacier 《 Ice penetration notes 》 e-book .
  • reply 【PDF】 Get other original Glacier 8 Ben superhard core PDF e-book , Massive interview materials and resume templates .

At the end

If you want to enter a large factory , I want a promotion and a raise , Or I'm confused about my current job , You can communicate with me by private letter , I hope some of my experiences can help you ~~

Recommended reading :

Okay , That's all for today , Like it, guys 、 Collection 、 Comment on , Get up with one button three times in a row , I'm glacier , See you next time ~~

版权声明
本文为[glacier]所创,转载请带上原文链接,感谢
https://cdmana.com/2021/10/20211001234630547D.html

Scroll to Top