编程知识 cdmana.com

Spring cloud gateway avalanche, I am stupid

This series is I TM People are stupid The sixth issue of the series [ Over your face ], Past highlights :

Spring Cloud Gateway  The avalanche , I  TM  People are stupid

Hello everyone , I'm stupid again . This experience tells us , Lazy to write code and steal , Sooner or later .

Problem phenomenon and background

Our gateway avalanched for some time last night , Phenomenon is :

1. There are constantly various micro service exceptions : Writing HTTP In response , Connection closed

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response 

2. At the same time, there are requests that have not been read , The connection has been closed

org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is java.io.IOException: UT000128: Remote peer closed connection before all data could be read 

3. There are constant requests for timeout alarms at the front end ,504 Gateway Time-out

4. The gateway process is restarted due to continuous health check failure

5. Restart the gateway process , The number of requests immediately surged , Peak value of each instance 2000 qps, Every instance in idle time 500 qps, During busy hours, due to capacity expansion, each instance can be kept in 1000 qps within , Then the health check interface does not respond for a long time , This causes the instance to restart continuously

among ,1 and 2 The problem should be that the gateway should be constantly restarted , And the graceful shutdown fails for some reasons, resulting in forced shutdown , Forced closure causes the connection to be forcibly disconnected, resulting in 1 and 2 Related anomalies .

Our gateway is based on Spring Cloud Gateway Realized , And there is an automatic basis CPU Load expansion mechanism . It's strange when the number of requests increases ,CPU Utilization has not improved much , Stay in 60% about , because CPU The load does not reach the limit of expansion , So there has been no automatic capacity expansion . In order to solve the problem quickly , We manually expanded several gateway instances , The single instance load of the gateway is controlled to 1000 within , Solved the problem for the time being .

Problem analysis

In order to solve the problem completely , We use JFR analysis . First, analyze according to the known clues :

  1. Spring Cloud Gateway Is based on Spring-WebFlux Asynchronous responsive gateway ,http Business threads are limited ( The default is 2 * serviceable CPU Number , Here we are 4).
  2. The gateway process continuously failed the health check , The health check calls /actuator/health Interface , This interface has been timed out .

There are generally two reasons for health check interface timeout :

  1. When the health check interface checks a component , Blocked up . For example, if the database is stuck , Then maybe the database health check will never return .
  2. http The thread pool did not have time to process the health check request , The request timed out .

We can go and see JFR Timing stack in , See if there is any http The thread is stuck on the health check . Check the thread stack after the problem , Focus on that 4 individual http Threads , It turns out that 4 The stack of two threads is basically the same , It's all about execution Redis command :

"reactor-http-nio-1" #68 daemon prio=5 os_prio=0 cpu=70832.99ms elapsed=199.98s tid=0x0000ffffb2f8a740 nid=0x69 waiting on condition  [0x0000fffe8adfc000]   java.lang.Thread.State: TIMED_WAITING (parking)    at jdk.internal.misc.Unsafe.park([email protected]/Native Method)    - parking to wait for  <0x00000007d50eddf8> (a java.util.concurrent.CompletableFuture$Signaller)    at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)    at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1798)    at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3128)    at java.util.concurrent.CompletableFuture.timedGet([email protected]/CompletableFuture.java:1868)    at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:2021)    at io.lettuce.core.protocol.AsyncCommand.await(AsyncCommand.java:83)    at io.lettuce.core.internal.Futures.awaitOrCancel(Futures.java:244)    at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:75)    at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)    at com.sun.proxy.$Proxy245.get(Unknown Source)    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:68)    at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:267)    at org.springframework.data.redis.connection.DefaultStringRedisConnection.get(DefaultStringRedisConnection.java:406)    at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:57)    at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:60)    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:222)    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:189)    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96)    at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:53)    at com.jojotech.apigateway.filter.AccessCheckFilter.traced(AccessCheckFilter.java:196)    at com.jojotech.apigateway.filter.AbstractTracedFilter.filter(AbstractTracedFilter.java:39)    at org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:137)    at org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:44)    at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:117)    at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain$$Lambda$1478/0x0000000800b84c40.get(Unknown Source)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at com.jojotech.apigateway.common.TracedMono.subscribe(TracedMono.java:24)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at com.jojotech.apigateway.common.TracedMono.subscribe(TracedMono.java:24)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at com.jojotech.apigateway.common.TracedMono.subscribe(TracedMono.java:24)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at com.jojotech.apigateway.common.TracedMono.subscribe(TracedMono.java:24)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at com.jojotech.apigateway.common.TracedMono.subscribe(TracedMono.java:24)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at com.jojotech.apigateway.common.TracedMono.subscribe(TracedMono.java:24)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)    at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:73)    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:281)    at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:860)    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:73)    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:281)    at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:860)    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)    at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onNext(MonoFilterWhen.java:149)    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2397)    at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onSubscribe(MonoFilterWhen.java:112)    at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:448)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250)    at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onNext(FluxDematerialize.java:98)    at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onNext(FluxDematerialize.java:44)    at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:270)    at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:228)    at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.request(FluxDematerialize.java:127)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:235)    at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onSubscribe(FluxDematerialize.java:77)    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:164)    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)    at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)    at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:448)    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:218)    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:164)    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at org.springframework.cloud.sleuth.instrument.web.TraceWebFilter$MonoWebFilterTrace.subscribe(TraceWebFilter.java:184)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)    at reactor.core.publisher.Mono.subscribe(Mono.java:4150)    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)    at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)    at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:915)    at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:654)    at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:478)    at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:526)    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)    at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:209)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)    at reactor.netty.http.server.logging.AccessLogHandlerH1.channelRead(AccessLogHandlerH1.java:59)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)    at java.lang.Thread.run([email protected]/Thread.java:834)

Find out http The thread is not stuck in the health check , At the same time, other threads do not have any stack related to health check ( Asynchronous environment , Health checks are also asynchronous , Some of these processes may be handed over to other threads ). therefore , The health check request should have timed out and cancelled before being executed .

So why is that ? At the same time , I also found that it uses RedisTemplate, yes spring-data-redis Synchronization of Redis API. I suddenly remembered when I wrote the code here before , Because it's just verifying one key Existence and modification key The expiration time of , It's no use being lazy API. This is not because of the use of synchronization API blocked http Thread induced avalanche ?

Let's test this conjecture : In our project redis The operation is through spring-data-redis + Lettuce Connection pool , Enabled and added about Lettuce Ordered JFR monitor , Please refer to my article : This Redis The new monitoring method of connection pool is not poked ~ I'll add a little more seasoning , So far, my pull request Merged , This feature will be in 6.2.x Version release . Let's look around the time of the problem Redis Command acquisition , As shown in the figure below :

Spring Cloud Gateway  The avalanche , I  TM  People are stupid

Let's simply calculate and execute Redis Blocking time caused by command ( Our collection is 10s once ,count Is the number of commands , The unit of time is microseconds ): Use the number of commands here multiplied by 50% The median , Divide 10( the reason being that 10s), Get every second because Redis Blocking time caused by command :

32*152=48641*860=8605*163=81532*176=56321*178=17816959*168=2849112774*176=1362243144*166=52190417343*179=3104397702*166=116532 The sum of the  67405186740518 / 10 = 674051.8 us = 0.67s

This is just the blocking time calculated using the median , From the distribution on the graph, we can see that the real value should be larger than this , This is likely to take place every second Redis The blocking time on the synchronization interface exceeds 1s, Keep asking , The request is not reduced , This leads to more requests , Last avalanche .

And because it's blocking the interface , Threads spend a lot of time waiting io 了 , therefore CPU Couldn't get on , As a result, there is no automatic capacity expansion . During peak business hours , Due to the preset capacity expansion , As a result, the single instance of the gateway does not reach the problem pressure , So no problem .

solve the problem

Let's rewrite the original code , Use synchronization spring-data-redis Api The original code is ( In fact, that is spring-cloud-gateway Of Filter The core method of the interface public Mono<Void> traced(ServerWebExchange exchange, GatewayFilterChain chain) Method body of ):

if (StringUtils.isBlank(token)) {    // If  token  non-existent , Then decide whether to continue the request or return the status code that needs to be logged in according to the path     return continueOrUnauthorized(path, exchange, chain, headers);} else {    try {        String accessTokenValue = redisTemplate.opsForValue().get(token);        if (StringUtils.isNotBlank(accessTokenValue)) {            // If  accessTokenValue  Not empty , Then it will be renewed  4  Hours , Ensure that the logged in user will not let  token  Be overdue             Long expire = redisTemplate.getExpire(token);            log.info("accessTokenValue = {}, expire = {}", accessTokenValue, expire);            if (expire != null && expire < 4 * 60 * 60) {                redisTemplate.expire(token, 4, TimeUnit.HOURS);            }            // analysis , obtain  userId            JSONObject accessToken = JSON.parseObject(accessTokenValue);            String userId = accessToken.getString("userId");            // If  userId  It is only legal if it is not empty             if (StringUtils.isNotBlank(userId)) {                // analysis  Token                 HttpHeaders newHeaders = parse(accessToken);                // Continue request                 return FilterUtil.changeRequestHeader(exchange, chain, newHeaders);            }        }    } catch (Exception e) {        log.error("read accessToken error: {}", e.getMessage(), e);    }    // If  token  illegal , Then decide whether to continue the request or return the status code that needs to be logged in according to the path     return continueOrUnauthorized(path, exchange, chain, headers);}

Use asynchronous instead :

if (StringUtils.isBlank(token)) {    return continueOrUnauthorized(path, exchange, chain, headers);} else {    HttpHeaders finalHeaders = headers;    // You have to use  tracedPublisherFactory  The parcel , Otherwise, the link information will be lost , Here I refer to another article :Spring Cloud Gateway  No link information , I  TM  People are stupid     return tracedPublisherFactory.getTracedMono(            redisTemplate.opsForValue().get(token)                    // You must switch threads , Otherwise, subsequent threads still use  Redisson  The thread of , If it takes a long time, it will affect other uses  Redis  The business of , And this time also counts in  Redis  Connection command timeout                     .publishOn(Schedulers.parallel()),            exchange    ).doOnSuccess(accessTokenValue -> {        if (accessTokenValue != null) {            //accessToken Renewal ,4 Hours             tracedPublisherFactory.getTracedMono(redisTemplate.getExpire(token).publishOn(Schedulers.parallel()), exchange).doOnSuccess(expire -> {                log.info("accessTokenValue = {}, expire = {}", accessTokenValue, expire);                if (expire != null && expire.toHours() < 4) {                    redisTemplate.expire(token, Duration.ofHours(4)).subscribe();                }            }).subscribe();        }    })    // Must be converted to non  null, otherwise  flatmap  Not execute ; You can't use... At the end  switchIfEmpty, Because the whole returns  Mono<Void>  Originally, it was empty , It will cause each request to be sent twice .    .defaultIfEmpty("")    .flatMap(accessTokenValue -> {        try {            if (StringUtils.isNotBlank(accessTokenValue)) {                JSONObject accessToken = JSON.parseObject(accessTokenValue);                String userId = accessToken.getString("userId");                if (StringUtils.isNotBlank(userId)) {                    // analysis  Token                     HttpHeaders newHeaders = parse(accessToken);                    // Continue request                     return FilterUtil.changeRequestHeader(exchange, chain, newHeaders);                }            }            return continueOrUnauthorized(path, exchange, chain, finalHeaders);        } catch (Exception e) {            log.error("read accessToken error: {}", e.getMessage(), e);            return continueOrUnauthorized(path, exchange, chain, finalHeaders);        }    });}

Here are a few caveats :

  1. Spring-Cloud-Sleuth about Spring-WebFlux Link tracking in is preferred , If we were Filter Create a new Flux perhaps Mono, There is no link information , We need to join... Manually . You can refer to another article of mine :Spring Cloud Gateway No link information , I TM People are stupid
  2. spring-data-redis + Lettuce Combination of connection pools , For asynchronous interfaces , We'd better switch to another thread pool after getting the response , Otherwise, subsequent threads still use Redisson The thread of , If it takes a long time, it will affect other uses Redis The business of , And this time also counts in Redis Connection command timeout
  3. Project Reactor If the intermediate result is null value , Then the back flatmap、map Wait for the stream operation to stop . If it ends here , The response received by the front end is problematic . So we have to consider the intermediate results at every step null problem .
  4. spring-cloud-gateway At the heart of GatewayFilter Interface , The core method returns Mono<Void>.Mono Originally, it was empty , So we can't use the end of switchIfEmpty To simplify the middle steps null, If used, it will cause each request to be sent twice .

After this modification , The pressure was measured , Single instance 2w qps The request did not have this problem .

WeChat search “ My programming meow ” Official account , Once a day , Easy to upgrade technology , Capture all kinds of offer

Spring Cloud Gateway  The avalanche , I  TM  People are stupid

版权声明
本文为[Dry goods are full]所创,转载请带上原文链接,感谢
https://cdmana.com/2021/10/20211002145637042M.html

Scroll to Top