编程知识 cdmana.com

Research on real time data consistency between redis and DB in MQ data synchronization

Catalog

Preset scene

Problem analysis

Process design

Practical verification

The conclusion of the question


Real time data , General business management data are basically unchanged , And the data that changes at any time is called real-time data . Real time data is difficult to guarantee consistency under different data sources , Unless you're looking at a snapshot of the data at some point ( Historical snapshot ), Otherwise, it's hard to keep data from different sources unchanged . therefore , The conclusion drawn from practice is that : It is a pseudo proposition that heterogeneous multi data sources ensure real-time data consistency , The use of cache is irreversible , In most cases, the cache cannot be used as a real data source .

Preset scene

To move App Personnel location management , requirement :

1、 Show the total number of users and the number of online ,MQ Data interaction to REDIS.

2、 Keep users online ,MQ Data synchronization to DB Keep online 、 Offline and location information .

3、 Verify that the number of lines on both sides is consistent .

Problem analysis

1、 from MQ Synchronize data to REDIS Manage user online status data , When the network state is unknown , Users go online frequently 、 offline .

  • a. user REDIS Cache location data efficiently
  • b. There is network jitter ( Uncertainty factors )

2、 from MQ Synchronize data to DB.

  • a.MQ Second consumption , Whether you put it first redis Let's go first DB Will cause data delay
  • b.DB and REDIS It can't be managed in the same transaction
  • c.DB Reading and writing and REDIS There is a gap in efficiency between reading and writing

3、 The conclusion is that :MQ Second consumption to different data sources cannot guarantee the real-time consistency of data , For both real-time and accuracy, only a single data source can be used .

Process design

Here's a picture of App The location of the heartbeat upload is distributed to RabbitMQ Location management services , The location is saved in REDIS in ,DB yes PG(Postgresql).

  • MQ Distribution logic : stay OnlineMQ、KeepLiveMQ and OfflineMQ A secondary queue is added to process the consumption ,RabbitMQ It's asynchronous execution consumption .
  • In the secondary queue :Java Program concurrent queue batch processing user status location data , Thread batch processing , It's to promote MQ The efficiency of their own consumption .
  • Data maintenance :Quality Service And grid aggregation is to maintain the user online time expired ancillary services .
  • user 30 No location and heartbeat in minutes, i.e. offline .
  • Live on a system with heartbeat but no location .
  • Users with heartbeat and location are online .
  • Real batch processing thread consumption must have a sleep time , Prevent idle scheduling CPU, The setting here is 50ms( No matter how small it is, it's a time difference ).
  • Under normal circumstances , offline 、 go online 、 Live queue data should be mutually exclusive .

Practical verification

redis The statistical results

pg The statistical results

 

The conclusion of the question

  • The real-time performance of asynchronous operation is uncontrollable .
  • If multiple data sources are not master-slave and cluster mode, it is not credible to synchronize real-time data .
  • The main function of caching is to assist database data services, not vice versa .
  • Heterogeneous database data synchronization, real-time data consistency can not be guaranteed .

Another problem that can not be ignored is to take redis When real-time location data, try to put scores As a filter , In this way, the user's valid data is more accurate , Please refer to the following code :

  /**
     *  According to the organization ID Inquire about REDIS【 Total number of users online summary 】
     *
     * @param orgId
     * @return
     */
    public Long getOrgOnlineUserFromRedis(String orgId) {
        long now = System.currentTimeMillis();
        long minScores = now - Constants.MOBILE_POSITION_TIME_OUT.longValue() * 1000;
        long maxScores = now + Constants.MOBILE_POSITION_TIME_OUT.longValue() * 1000;
        if (("" + Constants.DB_ORG_ROOT_ID).equals(orgId)) {
            String key = new StringBuffer(Constants.MOBILE_POSITION_ZSET_QG_KEY).toString();
            //return redisTemplate.opsForZSet().zCard(key);//  total 
            return redisTemplate.opsForZSet().count(key, minScores, maxScores); //  Summary of scoring conditions 
        } else {
            String key = new StringBuffer(Constants.MOBILE_POSITION_ZSET_DW_INCLUDE_CHILDREN_ORG_KEY).append(orgId).toString();
            // return redisTemplate.opsForZSet().zCard(key);//  total 
            return redisTemplate.opsForZSet().count(key, minScores, maxScores);//  Summary of scoring conditions 
        }
    }

 

版权声明
本文为[boonya]所创,转载请带上原文链接,感谢

Scroll to Top