编程知识 cdmana.com

Using redis technology to realize efficient data processing in batch

Catalog

Redis Pipeline technology

Redis Pipeline examples

Advantages of Pipeline Technology

Redis Business  

Transaction operation instance

Redis Business order

 Java Implementation pipeline + Transaction commit batch processing

RedisCallback

Pipeline call API


Redis It's a client based - Server model and request / In response to the protocol TCP service . This means that normally a request follows these steps :

  • The client sends a query request to the server , And monitor Socket return , Usually in blocking mode , Waiting for the server to respond .
  • The server handles commands , And return the result to the client .

Redis Pipeline technology

Redis Pipeline technology can be used when the server is not responding , The client can continue to send requests to the server , And finally read all the server's responses at once .

Redis Pipeline examples

see redis The Conduit , Just start redis Instance and enter the following command :

$(echo -en "PING\r\n SET w3ckey redis\r\nGET w3ckey\r\nINCR visitor\r\nINCR visitor\r\nINCR visitor\r\n"; sleep 10) | nc localhost 6379
 
+PONG
+OK
redis
:1
:2
:3

In the above example, we use  PING  Command view redis Is the service available , And then we set up w3ckey The value of is redis, Then we get w3ckey And make visitor Self increasing 3 Time .

In the returned results, we can see that these commands are directed to redis Service submission , And finally read all the server's responses at once


Advantages of Pipeline Technology

The most significant advantage of pipeline technology is the improvement of redis Service performance .

In the following test , We will use Redis Of Ruby client , Support pipeline technical characteristics , Test the effect of pipeline technology on speed . Example :

require 'rubygems' 
require 'redis'
def bench(descr) 
start = Time.now 
yield 
puts "#{descr} #{Time.now-start} seconds" 
end
def without_pipelining 
r = Redis.new 
10000.times { 
        r.ping 
} 
end
def with_pipelining 
r = Redis.new 
r.pipelined { 
        10000.times { 
                r.ping 
        } 
} 
end
bench("without pipelining") { 
        without_pipelining 
} 
bench("with pipelining") { 
        with_pipelining 
}

From... In the LAN Mac OS X The data of executing the above simple script on the system shows that , After opening the pipeline operation , The round-trip delay has been improved considerably .

without pipelining 1.185238 seconds 
with pipelining 0.250783 seconds

As you can see , After opening the pipe , Our speed and efficiency have improved 5 times .

Redis Business  

Redis Transactions can execute more than one command at a time , And with the following two important guarantees :

  • A transaction is a separate isolation operation : All commands in the transaction are serialized 、 To execute in order . Transaction is in the process of execution , Will not be interrupted by command requests from other clients .
  • A transaction is an atomic operation : The commands in the transaction are either all executed , Or none of it .

A transaction goes through the following three stages from the beginning to the execution :

  • Start business .
  • Order to join the team .
  • Perform transactions .

Transaction operation instance

Here is an example of a transaction , It starts with  MULTI  Start a transaction , Then queue multiple commands into the transaction , Finally by  EXEC  Command triggers transaction , Execute all the commands in the transaction :

redis 127.0.0.1:6379> MULTI
OK
 
redis 127.0.0.1:6379> SET book-name "Mastering C++ in 21 days"
QUEUED
 
redis 127.0.0.1:6379> GET book-name
QUEUED
 
redis 127.0.0.1:6379> SADD tag "C++" "Programming" "Mastering Series"
QUEUED
 
redis 127.0.0.1:6379> SMEMBERS tag
QUEUED
 
redis 127.0.0.1:6379> EXEC
1) OK
2) "Mastering C++ in 21 days"
3) (integer) 3
4) 1) "Mastering Series"
   2) "C++"
   3) "Programming"

Redis Business order

The following table lists them redis Orders related to the transaction :

Serial number Command and description
1 DISCARD  Cancel the business , Give up executing all commands within the transaction block .
2 EXEC  Execute commands within all transaction blocks .
3 MULTI  Mark the beginning of a transaction block .
4 UNWATCH  Cancel WATCH Command to all key Surveillance .
5 WATCH key [key ...]  Watch one ( Or more ) key , If before the transaction is executed ( Or these ) key Altered by other orders , Then the business will be interrupted .

 Java Implementation pipeline + Transaction commit batch processing

RedisCallback

redis Callback method for pipeline execution , It's kind of like Java Multithreading Callback Interface , But we don't have to deal with it ourselves :

/**
 * @Copyright: 2019-2021
 * @FileName: MessageStatusRedisCallback .java
 * @Author: PJL
 * @Date: 2020/6/5 14:25
 * @Description:  Notification status redis Instruction callback class 
 */
public class MessageStatusRedisCallback implements RedisCallback<Object> {

    Logger logger = LoggerUtils.getLogger(LoggerUtils.PatrolLoggerType.REDIS);

    List<MessageStatus> messageStatusList;

    /**
     *  Construction method 
     *
     * @param messageStatusList
     */
    public MessageStatusRedisCallback(List<MessageStatus> messageStatusList) {
        this.messageStatusList = messageStatusList;
    }

    @Override
    public Object doInRedis(RedisConnection redisConnection) throws DataAccessException {
        //  Verification instruction execution mode 
        if (ObjectUtils.isNotEmpty(messageStatusList)) {
            long s = System.currentTimeMillis();
            String json = null;
            try {
                //  Turn on REDIS Business 
                redisConnection.multi();
                for (MessageStatus messageStatus : messageStatusList) {
                    json = JSONObject.toJSONString(messageStatus);
                    /* redisTemplate.opsForHash().put(Constants.MOBILE_HASH_MESSAGE_KEY, userId, Boolean.toString(hasMessage));*/
                    redisConnection.hSet(Constants.MOBILE_HASH_MESSAGE_KEY.getBytes(), messageStatus.getUserId().getBytes(), Boolean.toString(messageStatus.isHasMessage()).getBytes());
                }
                //  Commit the transaction to batch execution 
                redisConnection.exec();
            } catch (Exception e) {
                //  The rollback transaction is not executed 
                redisConnection.discard();
                logger.error("【 The rollback transaction is not executed 】 Problem user notification status JSON:{}", json);
            }
            long e = System.currentTimeMillis();
            logger.info("pipeline  Batch execution of user notification announcements  status Command complete .... Time consuming :{} {}", (e - s), "ms");
        }
        return null;
    }
}

Pipeline call API

Use RedisTemplate Method provided to call :

/**
 * @Copyright: 2019-2021
 * @FileName: MessageStatusRedisPipelineService.java
 * @Author: PJL
 * @Date: 2020/6/5 14:19
 * @Description:  Notice notice redis pipeline Execute command service 
 */
@Service
public class MessageStatusRedisPipelineService {

    @Qualifier("redisTemplateByLettuce")
    @Autowired
    RedisTemplate redisTemplate;

    /**
     *  Set the user notification notification status record to cache to REDIS
     *
     * @param messageStatusList
     */
    public void saveMessageStatusToRedis(List<MessageStatus> messageStatusList) {
        redisTemplate.executePipelined(new MessageStatusRedisCallback(messageStatusList));
    }
}

thus , We realized an opportunity Pipeline Pipeline technology and batch operation of transactions .

Reference reading :

版权声明
本文为[boonya]所创,转载请带上原文链接,感谢

Scroll to Top