RocketMq It mainly consists of three parts ：Producer（ Send a message ）、Broker（ Store messages ）、Consumer（ News consumption ）.Broker The actual deployment corresponds to a server , Every Broker Can store multiple Topic news , Every Topic Messages can also be stored in different pieces Broker in ,Message Queue The physical address used to store messages , Every Topic Message address stores multiple Message Queue in .
Responsible for production information , Generally, the business system is responsible for the production messages . A message producer sends messages generated in a business application system to broker The server .RocketMQ There are many ways to send , The synchronous 、 Send asynchronously 、 Send in sequence 、 Send one way . Both synchronous and asynchronous modes require Broker Return confirmation , One way delivery doesn't need .
Among producers, the same kind of Producer Make up a set called producer group , If the producer goes down after sending a transaction message , be Broker The service contacts other instances of the same producer group to submit or backtrack consumers .
Responsible for consumer information , The background system is responsible for asynchronous consumption , Consumers from Broker Pull the message and provide it to the application , Provide two forms of consumption ： Pull consumption 、 Promoting consumption .
- Pull consumption usually takes the initiative to call Consumer Pull message method from Broker Pull the news , The initiative is controlled by the application .
- Promote consumption by Broker After receiving the data, actively push it to consumers , This mode has good real-time performance .
Consumers will also put a kind of Consumer Make a collection ： Consumer group , It's very easy to achieve load balancing and fault tolerance , Consumer groups must subscribe to the same Topic,RocketMQ Support two consumption patterns ： Cluster consumption Clustering, Radio consumption Broadcasting.
- Cluster consumption is the same Consumer Group Every Consumer Instance average allocation message .
- Broadcast consumption is the same Consumer Group Every Consumer Instance receives full messages .
Represents a set of messages , Each topic contains several messages , Each message belongs to only one topic .
Suggest a App To build a Topic adopt Tag To represent different types of messages .
The smallest unit of production and consumption , Every consumption has a unique Message ID, Can carry business key, The system provides Message ID and key Query function . We can also fight for consumption tag label , Filtering different business messages .
Distributed queues are reliable , All data should be persistent .
- MQ After receiving the message, it returns a ACK Respond to , And store the message .
- MQ Push A message to the consumer and wait for the consumer ACK Respond to , Mark the message as consumed , If not marked as consumed MQ Constantly trying to push messages to consumers （ Default 16 Time ）.
- MQ Delete some expired messages regularly , Keep the service available at all times .
MQ and Kafka Also through the file storage mechanism , Save messages directly from disk files , There is no need to resort to MySQL This kind of indexing tool .
- Take the order IO And zero copy technology to speed up file reading and writing .
- Asynchronous brush disk adopts MMAP Machine brush , You can only map at a time 1.5G~2G.
All messages are stored in CommitLog In file .
ConsumerQueue Message index , Every topic Create an index file ,CommitLog Offset and Message Tag Index .
IndexFile： adopt key,CommitLog Offset、TimeStamp、NextIndexOffset Indexing makes it easier for users to locate messages more accurately .
Broker Cluster mode deployment , There is one master Nodes and multiple slave node , The message needs to come from Master Copied to the Slave On , Message replication is divided into synchronous and asynchronous .
Synchronous replication ：Master and Slave After the message is successfully written, it will be fed back to the client to write the successful status . If Master Node failure ,Slave All data is backed up , Easy to recover , But big data write latency , Reduce system throughput .
Asynchronous replication ： Only Master If the write is successful, the success status will be returned , And then asynchronously replicate to Slave node . Asynchronous replication system has lower latency and higher throughput , If Master If the data is not copied completely, it will cause data loss .
Producer Polling mode is adopted by default , Send to different MessageQueue in , If it's a sequential message sent to the specified MessageQueue in .
Consumer If it's radio （BROADCASTING） Pattern, then every consumer receives a message .
Consumer If it's a cluster （CLUSTERING） Mode, then only one consumer can receive a message .
AllocateMessageQueueAveragely（ Default ）： Average distribution . When the number of message queues is less than the number of consumable clients , The correspondence between message queue and client is shown in the figure on the left ; When the number of message queues is greater than the number of consumable clients , The correspondence between message queue and client is shown in the figure on the right .
AllocateMessageQueueAveragelyByCircle： Ring distribution . Evenly distributed in a ring .
AllocateMachineRoomNearby： The same as the computer room . First of all, count consumers and broker In the computer room , Guarantee broker The messages in the server room are first consumed by consumers in the same computer room , If there is no consumer in the computer room , There are other computer room consumers . The actual queue allocation （ Same machine room or cross machine room ） You can specify other algorithms .
AllocateMessageQueueByMachineRoom： Specify the computer room assignment . The first picture shows that the number of consumers is less than the number of queues , The second picture shows the number of redundant queues of consumers . Suppose there are three computer rooms , Configuration room 3 is not within the scope of consumer service , The corresponding relationship of actual consumption is shown in the following two figures .
- AllocateMessageQueueByConfig： Manually assign . Load using consistent hashing algorithm , Each time the load recreates consistency hash Routing table , Get all the queue information that the local client is responsible for .RocketMQ default hash Algorithm for MD5. Suppose there is 4 A client's clientId And two message queues mq1,mq2,, adopt hash After the distribution in hash The different positions of the rings , In accordance with consistency hash The clockwise search principle of ,mq1 By client2 consumption ,mq2 By client3 consumption .
- Broadcast news ： There is no message mechanism , That is, the message will not be sent again after consumption failure , And just continue to consume new news .
- General news ： When consumers fail , Message retrial can be achieved by setting the return status . return Action.ReconsumerLater（ recommend ）, return null, Throw an exception .
The retrying message needs to enter a %RETRY%+ConsumeGroup In line . By default, each message is allowed to retry at most 16 Time , The time interval for each retry is as follows ：
|Retry count||The interval between the last retry and||Retry count||The interval between the last retry and|
|1||10 second||2||30 second|
|3||1 minute||4||2 minute|
|5||3 minute||6||4 minute|
|7||5 minute||8||6 minute|
|9||7 minute||10||8 minute|
|11||9 minute||12||10 minute|
|13||20 minute||14||30 minute|
|15||1 Hours||16||2 Hours|
If the number of retries exceeds 16 The next time will be converted to dead letter queue .
Be careful ： No matter how many retries Message ID It's all the same , Except for transaction messages .
When a message fails to consume ,RocketMQ The message will be retried automatically , And if the message exceeds the maximum number of times ,RocketMQ You think there's something wrong with the news , Put the message on the dead letter queue .
Dead letter queue name ：%DLQ%+ConsumeGroup
- One dead letter queue corresponds to one ConsumeGroup, Instead of corresponding to a consumer instance .
- If one ConsumeGroup There was no dead letter message ,RocketMQ The corresponding dead letter queue will not be created .
- A dead letter queue contains this ConsumeGroup All the information in the letter , Indistinguishes Topic.
- Messages in the dead letter queue will no longer be consumed by consumers .
- The validity period of dead letter queue is the same as that of normal message . Default 3 God , It can be done by broker.conf Corresponding fileReservedTime Attribute configuration , It will be deleted after this time , Whether the news has been consumed or not .
- The default permission of column message queue ：2（2： No reading ,4： No writing ,6： Can read but write ）, Need to change to 6 Maybe normal consumption .
stay MQ There are three implementation semantics for message idempotent in the system ：
- at most once At most once ： Each message can only be consumed once at most .
- at least once At least once ： Each message is consumed at least once .
- exactly once Just once ： Each message will be definitely consumed only once .
at most once Best guarantee , Using asynchronous transmission 、sendOneWay And so on .
at least once The synchronous 、 Transaction messages and other methods can also guarantee .
exactly once The hardest thing to guarantee ,RocketMQ Only guarantee at least once, There's no guarantee exactly once, It needs to be guaranteed through business . It is recommended to bring service when sending messages key, Such as the order id, Judge whether the order has been processed before consumption .