编程人 cdmana.com

Rocketmq principle

Basic concepts

Message model (Message Model)

RocketMq It mainly consists of three parts :Producer( Send a message )、Broker( Store messages )、Consumer( News consumption ).Broker The actual deployment corresponds to a server , Every Broker Can store multiple Topic news , Every Topic Messages can also be stored in different pieces Broker in ,Message Queue The physical address used to store messages , Every Topic Message address stores multiple Message Queue in .

producer (Producer)

Responsible for production information , Generally, the business system is responsible for the production messages . A message producer sends messages generated in a business application system to broker The server .RocketMQ There are many ways to send , The synchronous 、 Send asynchronously 、 Send in sequence 、 Send one way . Both synchronous and asynchronous modes require Broker Return confirmation , One way delivery doesn't need .

Among producers, the same kind of Producer Make up a set called producer group , If the producer goes down after sending a transaction message , be Broker The service contacts other instances of the same producer group to submit or backtrack consumers .

consumer (Consumer)

Responsible for consumer information , The background system is responsible for asynchronous consumption , Consumers from Broker Pull the message and provide it to the application , Provide two forms of consumption : Pull consumption 、 Promoting consumption .

  • Pull consumption usually takes the initiative to call Consumer Pull message method from Broker Pull the news , The initiative is controlled by the application .
  • Promote consumption by Broker After receiving the data, actively push it to consumers , This mode has good real-time performance .

Consumers will also put a kind of Consumer Make a collection : Consumer group , It's very easy to achieve load balancing and fault tolerance , Consumer groups must subscribe to the same Topic,RocketMQ Support two consumption patterns : Cluster consumption Clustering, Radio consumption Broadcasting.

  • Cluster consumption is the same Consumer Group Every Consumer Instance average allocation message .
  • Broadcast consumption is the same Consumer Group Every Consumer Instance receives full messages .

The theme (Topic)

Represents a set of messages , Each topic contains several messages , Each message belongs to only one topic .
Suggest a App To build a Topic adopt Tag To represent different types of messages .

news (Message)

The smallest unit of production and consumption , Every consumption has a unique Message ID, Can carry business key, The system provides Message ID and key Query function . We can also fight for consumption tag label , Filtering different business messages .

Message store

When to store messages

Distributed queues are reliable , All data should be persistent .

  • MQ After receiving the message, it returns a ACK Respond to , And store the message .
  • MQ Push A message to the consumer and wait for the consumer ACK Respond to , Mark the message as consumed , If not marked as consumed MQ Constantly trying to push messages to consumers ( Default 16 Time ).
  • MQ Delete some expired messages regularly , Keep the service available at all times .

storage medium

MQ and Kafka Also through the file storage mechanism , Save messages directly from disk files , There is no need to resort to MySQL This kind of indexing tool .

  • Take the order IO And zero copy technology to speed up file reading and writing .
  • Asynchronous brush disk adopts MMAP Machine brush , You can only map at a time 1.5G~2G.

Storage structure

All messages are stored in CommitLog In file .
ConsumerQueue Message index , Every topic Create an index file ,CommitLog Offset and Message Tag Index .
IndexFile: adopt key,CommitLog Offset、TimeStamp、NextIndexOffset Indexing makes it easier for users to locate messages more accurately .

 Insert picture description here

Master slave copy

Broker Cluster mode deployment , There is one master Nodes and multiple slave node , The message needs to come from Master Copied to the Slave On , Message replication is divided into synchronous and asynchronous .

  • Synchronous replication :Master and Slave After the message is successfully written, it will be fed back to the client to write the successful status . If Master Node failure ,Slave All data is backed up , Easy to recover , But big data write latency , Reduce system throughput .

  • Asynchronous replication : Only Master If the write is successful, the success status will be returned , And then asynchronously replicate to Slave node . Asynchronous replication system has lower latency and higher throughput , If Master If the data is not copied completely, it will cause data loss .

 Insert picture description here

Load balancing

Producer Polling mode is adopted by default , Send to different MessageQueue in , If it's a sequential message sent to the specified MessageQueue in .

 Insert picture description here

Consumer If it's radio (BROADCASTING) Pattern, then every consumer receives a message .
Consumer If it's a cluster (CLUSTERING) Mode, then only one consumer can receive a message .

  • AllocateMessageQueueAveragely( Default ): Average distribution . When the number of message queues is less than the number of consumable clients , The correspondence between message queue and client is shown in the figure on the left ; When the number of message queues is greater than the number of consumable clients , The correspondence between message queue and client is shown in the figure on the right .
     Insert picture description here

  • AllocateMessageQueueAveragelyByCircle: Ring distribution . Evenly distributed in a ring .
     Insert picture description here

  • AllocateMachineRoomNearby: The same as the computer room . First of all, count consumers and broker In the computer room , Guarantee broker The messages in the server room are first consumed by consumers in the same computer room , If there is no consumer in the computer room , There are other computer room consumers . The actual queue allocation ( Same machine room or cross machine room ) You can specify other algorithms .
     Insert picture description here

  • AllocateMessageQueueByMachineRoom: Specify the computer room assignment . The first picture shows that the number of consumers is less than the number of queues , The second picture shows the number of redundant queues of consumers . Suppose there are three computer rooms , Configuration room 3 is not within the scope of consumer service , The corresponding relationship of actual consumption is shown in the following two figures .
     Insert picture description here

 Insert picture description here

  • AllocateMessageQueueByConfig: Manually assign . Load using consistent hashing algorithm , Each time the load recreates consistency hash Routing table , Get all the queue information that the local client is responsible for .RocketMQ default hash Algorithm for MD5. Suppose there is 4 A client's clientId And two message queues mq1,mq2,, adopt hash After the distribution in hash The different positions of the rings , In accordance with consistency hash The clockwise search principle of ,mq1 By client2 consumption ,mq2 By client3 consumption .

Message retry

  • Broadcast news : There is no message mechanism , That is, the message will not be sent again after consumption failure , And just continue to consume new news .
  • General news : When consumers fail , Message retrial can be achieved by setting the return status . return Action.ReconsumerLater( recommend ), return null, Throw an exception .

The retrying message needs to enter a %RETRY%+ConsumeGroup In line . By default, each message is allowed to retry at most 16 Time , The time interval for each retry is as follows :

Retry count The interval between the last retry and Retry count The interval between the last retry and
1 10 second 2 30 second
3 1 minute 4 2 minute
5 3 minute 6 4 minute
7 5 minute 8 6 minute
9 7 minute 10 8 minute
11 9 minute 12 10 minute
13 20 minute 14 30 minute
15 1 Hours 16 2 Hours

If the number of retries exceeds 16 The next time will be converted to dead letter queue .
Be careful : No matter how many retries Message ID It's all the same , Except for transaction messages .

Dead letter queue

When a message fails to consume ,RocketMQ The message will be retried automatically , And if the message exceeds the maximum number of times ,RocketMQ You think there's something wrong with the news , Put the message on the dead letter queue .
Dead letter queue name :%DLQ%+ConsumeGroup

features :

  • One dead letter queue corresponds to one ConsumeGroup, Instead of corresponding to a consumer instance .
  • If one ConsumeGroup There was no dead letter message ,RocketMQ The corresponding dead letter queue will not be created .
  • A dead letter queue contains this ConsumeGroup All the information in the letter , Indistinguishes Topic.
  • Messages in the dead letter queue will no longer be consumed by consumers .
  • The validity period of dead letter queue is the same as that of normal message . Default 3 God , It can be done by broker.conf Corresponding fileReservedTime Attribute configuration , It will be deleted after this time , Whether the news has been consumed or not .
  • The default permission of column message queue :2(2: No reading ,4: No writing ,6: Can read but write ), Need to change to 6 Maybe normal consumption .

Message idempotent

stay MQ There are three implementation semantics for message idempotent in the system :

  • at most once At most once : Each message can only be consumed once at most .
  • at least once At least once : Each message is consumed at least once .
  • exactly once Just once : Each message will be definitely consumed only once .

at most once Best guarantee , Using asynchronous transmission 、sendOneWay And so on .
at least once The synchronous 、 Transaction messages and other methods can also guarantee .
exactly once The hardest thing to guarantee ,RocketMQ Only guarantee at least once, There's no guarantee exactly once, It needs to be guaranteed through business . It is recommended to bring service when sending messages key, Such as the order id, Judge whether the order has been processed before consumption .

Scroll to Top