编程知识 cdmana.com

The evolution of the "Rainbow Bridge" middleware platform for the acquisition database based on shardingsphere

This article is reprinted in official account

Preface

With what you get  App  Users began to grow rapidly , Business lines are becoming richer and richer , It also brings great pressure to the underlying database . Each line of business is responsible for data fragmentation 、 Read / write separation 、 The demand for shadow database routing has become a rigid demand , Therefore, a unified middleware is needed to support these requirements , Get something “ Rainbow bridge ” emerge as the times require .

In Nordic mythology , Rainbow bridge connects Asgard ( Asgard )【1】 and   Midgart ( Stalls / Midgard ) The huge rainbow bridge . We can think of it as “ Between the nine boundaries ” The connection channel of , It is also the only stable entrance to Asgard . The rainbow bridge is a database middle layer processing middleware connecting services and databases , It can be said that every order for something is closely related to it .

1.  Technology selection

In the early stage, we investigated  Mycat、ShardingSphere、kingshard、Atlas  Open source middleware , Comprehensive applicability 、 Advantages and disadvantages 、 Product reputation 、 Community activity 、 Practical cases 、 Scalability and other aspects , In the end, we chose  ShardingSphere . Ready to  ShardingSphere  On the basis of the second development 、 Customize a set of database middleware suitable for the internal environment of dewu .

Apache ShardingSphere  It is an open source distributed database ecological project , from  JDBC、Proxy  and  Sidecar( Planning ) 3  Product composition . Its core adopts pluggable architecture , Extend functions through components . On top of the database protocol and  SQL  Way to provide many enhancements , Including data fragmentation 、 Access routing 、 Data security, etc ; Native support for  MySQL、PostgreSQL、SQL Server、Oracle  And other data storage engines .ShardingSphere  Has been in 2020 year 4 month 16 Day to day  Apache  Top projects at the software foundation , And there are teams in many countries all over the world .

At present, we are mainly  ShardingSphere Of Proxy  Patterns provide services , The following will be in  JDBC&Proxy  Hybrid architectures continue to be explored .

2. The current capacity of rainbow bridge



Among them, the white module refers to the current stage and capabilities , The green module is planning & Functions being done . Here are some key functions . Be careful , The following functions are based on  Proxy  Pattern .

2.1  Data fragmentation

Data fragmentation refers to storing the data stored in a single database in multiple databases or tables according to a certain dimension, so as to improve the performance bottleneck and availability . The effective method of data fragmentation is to divide databases and tables into relational databases . Both sub database and sub table can effectively avoid the query bottleneck caused by the amount of data exceeding the tolerable threshold . besides , The sub database can also be used to effectively disperse the single point of access to the database ; Although sub table can't relieve the pressure of database , But it can provide the possibility of transforming distributed transaction into local transaction as much as possible , When it comes to cross database update operations , Distributed transactions tend to complicate problems . Use the multi master and multi slave split mode , It can effectively avoid data single point , So as to improve the availability of data architecture .

The amount of data in each table is kept below the threshold by splitting the data into sub database and sub table , And traffic grooming to deal with high traffic , It is an effective way to deal with high concurrency and massive data system . Data fragmentation is divided into vertical fragmentation and horizontal fragmentation .

The way of business splitting is called vertical segmentation , Also known as vertical split , Its core idea is dedicated to special storage . Rainbow bridge mainly provides horizontal slicing capability , Horizontal segmentation is also called horizontal splitting . Relative to the vertical slice , It no longer classifies data according to business logic , But through a certain field ( Or some fields ), Spread data across multiple libraries or tables according to certain rules , Each slice contains only a part of the data . for example : Slice according to the primary key , Even primary key records are put into  0  library ( Or table ), The record of odd primary key is put into  1  library ( Or table ), As shown in the figure below .



In theory, horizontal slicing breaks through the bottleneck of single machine data processing , And expand relative freedom , Is a standard solution for data fragmentation .

Of course, the segmentation rules of the actual use scenario are very complex , We provide some built-in algorithms, such as modulo 、HASH  modulus 、 Automatic time segment segmentation algorithm 、Inline  Expression etc. . When the built-in algorithm cannot meet the requirements , It can also be based on  groovy  To customize the exclusive sharding logic .


2.2  Read / write separation

In the face of increasing system access , Database throughput is facing a huge bottleneck . For an application system with a large number of concurrent read operations and fewer write operations at the same time , Split the database into master and slave , The main database is responsible for the transaction of addition, deletion and modification , The slave library is responsible for handling query operations , It can effectively avoid row lock caused by data update , The query performance of the whole system has been greatly improved . Through the configuration of one master and multiple slaves , Query requests can be evenly distributed across multiple copies of data , It can further improve the processing capacity of the system .

It is different from the horizontal fragmentation of data to each data node according to the fragmentation key , Read write separation is based on  SQL  Semantic analysis , Route read and write operations to the master and slave libraries respectively .



The configuration here is simple , Bind one or more slave libraries to the target master library 、 Set the corresponding load balancing algorithm .



The way to achieve this is through  SQL  analysis , Route the query statement to the corresponding slave database , But in some scenarios that are sensitive to master-slave synchronization delay , It may be necessary to force the main database to be taken , Here we also provide a  API( The principle is  SQL Hint), The upstream can specify some modules to read and force the master , There are also related global configurations that can make all read requests in a transaction master .

2.3  Shadow reservoir pressure measurement

Under the distributed application architecture based on microservices , The business needs multiple services through a series of services 、 Call the middleware to complete , Therefore, the stress test of a single service can no longer represent the real scenario . In the test environment , If a complete set of pressure measuring environment similar to the production environment is rebuilt , High cost , And it is often unable to simulate the complexity and traffic of the online environment . therefore , In the industry, full link voltage measurement is usually used , I.e. pressure measurement in the production environment , The test results thus obtained can accurately reflect the real capacity and performance level of the system .

Full link voltage measurement is a complex and huge task . Various micro services are required 、 Coordination and adjustment between middleware , To cope with the transmission of different flow and pressure measurement marks . Usually, a complete set of pressure test platforms will be built to apply to different test plans . Data isolation is required at the database level , In order to ensure the reliability and integrity of production data , It is necessary to route the data generated by the pressure measurement to the pressure measurement environment database , Prevent the pressure measurement data from polluting the real data in the production database . This requires business applications to execute  SQL  front , It can be marked according to the pressure measurement through transmission , Do a good job in data classification , Corresponding  SQL  Route to the corresponding data source .

The configuration here is similar to read / write separation , It is also a shadow library bound to the target main library , When  SQL  Carrying a shadow tag will be routed to the shadow database .


2.4  Current limiting & Fuse

When  DB  When the pressure exceeds its own water level , It can lead to  DB  failure . When we estimate the water level of a certain dimension , You can configure current limiting rules for , To protect by rejecting requests beyond its own waterline  DB. Let the system run at the maximum throughput and ensure the overall stability of the system . In terms of dimensions, we support  DB、Table、SQL  as well as  DML  type .


Hundreds of are connected under the rainbow bridge  RDS  example , Every  RDS  All instances may have various faults , When a single instance fails, the entire logical library will be affected , It will quickly cause blockage and induce avalanche effect . So we need a quick failure mechanism to prevent avalanches , At present, we support  DB  Instance level , Based on getting connections & SQL  perform  2  Species behavior , And the implementation time and failure ratio to realize fusing , In order to achieve in  DB  The effect of rapid failure in case of failure .

null

2.5  Flow correction

Under the dual active architecture , Rainbow Bridge acts as the proxy layer of the database , It can ensure that the traffic in the process of traffic switching under the dual active architecture is intercepted , Ensure data consistency . The principle is based on  SQL Hint  Carry the  userId  Match the machine room rules , Intercept traffic that does not belong to the current computer room .

3.  be based on  ShardingSphere  What changes have we made

although  ShardingSphere Proxy  In fact, it is strong enough , But for the object's internal environment , There are still some defects and lack of functions , It is mainly divided into the following points :

  • Ease of use
  • Fragmentation 、 Rule configuration files such as read / write separation are too complex , Not friendly to business development
  • The dynamic change of rules completely depends on the configuration center , Lack of complete change process
  • The governance capability of connection pool is not perfect
  • Hint  The way is not friendly enough , Business needs to write  RAL  sentence
  • Custom sharding algorithms need to be published
  • SQL  Compatibility
  • stability
  • The multi cluster governance function is missing
  • Lack of isolation between logical Libraries
  • The current limiting fuse assembly is missing
  • data source 、 Dynamic rule changes are detrimental to
  • Lack of flow correction function under dual active architecture
  • Publishing is damaging
  • Observability
  • SQL Trace  Incomplete function
  • Monitoring indicators are not comprehensive enough
  • SQL  Lack of insight
  • performance
  • Due to one more network forwarding , Single  SQL  Of  RT  Than the direct link will rise  2~3ms

To solve the above problems , We have made the following transformation and optimization .

3.1  Ease of use

  • For data sources 、 Configuration of rules 、 change 、 Audit and other operations , Integrated into the control console for unified operation . The fragmentation is reduced in a graphical way 、 Complexity of rule configuration files such as read / write separation , And a series of verifications are added to avoid some low-level errors caused by configuration file errors . Secondly, the audit function is added , It ensures the security and controllability of dynamic configuration changes .
  • The control console has added connection pool management , be based on  RDS  The number of connections 、Proxy  Number of nodes 、 The number of attached data sources and other factors automatically calculate a safe and reasonable connection pool size .
  • newly added  Client, in the light of  Hint  Make a series of adaptations , For example, shadow marker transmission 、 Dual active architecture users  id  Pass on 、trace  Pass on 、 Forced routing, etc . The user only needs to call  Client  Medium  API, You can send out  SQL  The phase is automatically rewritten to  Proxy  Recognizable  Hint  Enhanced statement .
  • The cluster management function has been added to the control console : Because we have deployed multiple sets of  Proxy  colony , In order to guarantee the explosion radius in case of failure to the greatest extent , We compare... By business domain  Proxy  Clusters are divided , Try to ensure that the logical library traffic under the unified business domain enters the same cluster .
  • Added  Groovy  To support custom sharding algorithms , After the partition logic is configured in the background, it will take effect after it is approved , There is no need to  Proxy  release

3.2  Improved stability

3.2.1 Proxy  Multi cluster governance
(1) background

With  Proxy  More and more business domains are hosted , If all traffic is routed to... Through load balancing Proxy  node , When a library has problems , It may cause the entire cluster to be paralyzed , The explosion range is uncontrollable . And because of  DB  The number of connections is limited , This leads to  Proxy  The nodes of cannot scale out on a large scale .

(2) Solution

For isolation between different business domains , We have deployed several sets  Proxy  colony , And maintain the relationship between each logical library and the cluster through the control console . Ensure that the logical library traffic under the same business domain enters the same cluster . And when a cluster fails , The logic library in the fault cluster can be quickly 、 Lossless dynamic switchover to the standby cluster , Reduce... As much as possible  Proxy  The loss to the business caused by its own failure .

For connection number management ,Proxy  When initializing the connection pool, you will judge whether the current logical library is in the current cluster , If the minimum number of connections is not set to the minimum , If it is, it will be loaded according to the normal configuration , Before and after cluster switching, the target cluster shall be preheated and the original cluster shall be recycled . This can greatly alleviate  DB  Connection pool resource pair  Proxy  Horizontal expansion of nodes .

(3) Realization principle

Upstream application introduction  Rainbow( Self developed connection pool ) after , Before the connection pool is initialized, the cluster where the current library is located will be read according to the logical library , And dynamically put  Proxy  Replace the domain name of the cluster with the domain name of the cluster . At the same time, it will also add monitoring for cluster configuration , After the control console switches the cluster operation ,Rainbow  A new connection pool will be created according to the switched cluster domain name , Then replace the old connection pool , The old connection pool will also be gracefully closed with delay , The whole process is insensitive to upstream applications .

(4) Architecture diagram


3.2.2 Proxy  Worker pool isolation
(1) background

Open source version of  Proxy, All logical libraries share a thread pool , When a single logical library is blocked , Thread pool resources will be exhausted , As a result, other logical libraries are also affected .

(2) Solution

We have adopted the isolation scheme based on thread pool , Introduced exclusive & The concept of shared thread pool , Priority is given to the use of the logical library alone busy process pool , When the exclusive thread pool is queued, the shared thread pool can be used , After the shared thread pool reaches a certain load, it will be forcibly routed to the exclusive thread pool within a time window , On the premise of ensuring isolation , Maximize the use of thread resources .
3.2.3  Flow control and fusing
(1) background

Open source version of  Proxy Missing Library 、 surface 、SQL  Equal dimensional flow control , When the flow exceeds the system water level in a short time , It is likely to bring down  DB  It leads to online failure , And there is a lack of targeting  DB  The mechanism of rapid failure , When  DB  failure ( such as  CPU 100%) Can't fail quickly , It will quickly cause blockage and induce avalanche effect .

(2) Solution

New dimensions (DB、table、SQL、 Statement type ) Current limiting of , Each database can configure a reasonable threshold according to the estimated water level and business needs , Maximum protection  DB . At the same time, we also introduce  DB  Instance fusing strategy , When an instance fails, it can fail quickly . In the sub database scenario , It can minimize the impact on other segments , The explosion radius of the fault is further reduced , Improve the overall stability of the rainbow bridge .

(3) Realization principle

Both flow control and fusing are based on  sentinel  To achieve , Configure the corresponding rules on the control console .
3.2.4  Non destructive release
(1) background

In the early stage ,Proxy  Every time you publish or restart , Will receive upstream applications  SQL  Some alarms of execution failure , The main reason is that the upstream and  Proxy  There's a long connection between them , At this time, if a connection is being executed  SQL, Then it will be forcibly disconnected, resulting in  SQL  Execution failure , To some extent, it will cause losses to the upstream business .

(2) Solution

The publishing system cooperates with the self-developed connection pool  Rainbow, stay  Proxy  Before the node is released or restarted , Will inform  Rainbow  Connection pool graceful shutdown should be applied to restart & released  Proxy  Connections between nodes , stay  Proxy  Flow drop  0  Then restart & Release .

3.3  Observability

3.3.1  Runtime metrics
(1) background

Open source version for  Proxy  There are few monitoring indicators at runtime , Cause unobservable  Proxy  Detailed status of each library run above .

(2) Solution

stay  Proxy  Embedded points are added in each execution stage , The following indicators have been added 、 And mapped the corresponding monitoring index .

  • library & Table level  QPS、RT、error、 slow  SQL  indicators
  • Proxy-DB  Each connection number index of the connection pool
  • Flow control fusing index
  • Number of active threads in thread pool 、 Queue size indicator

(3) design sketch




3.3.2  Full link tracking
(1) background

Open source version of  trace  Only support  Proxy  Link tracking in the internal execution phase , Cannot be connected in series with upstream , This leads to low barrier removal efficiency .

(2) Solution

Mainly through  RAL、SQL  notes  2  Two ways of transmission  trace  Information , The upstream to  Proxy  Full link tracking for .

(3) Realization principle

Our first idea is to pass  SQL  Pass by annotation  trace  Information , However, some problems were found after it was put into production , Used in the upstream  prepare  Pattern (useServerPrepStmts=true&cachePrepStmts=true) when ,Proxy  Will cache statmentId  and  SQL , and  trace  It's different every time , This will cause the storage cache to grow indefinitely and eventually lead to  OOM .



So pass  SQL  Pass by annotation  trace  Information only applies to non  prepare  scene . So we added a new scheme, that is, every time  SQL  Send a message before execution  RAL  Statement to pass  trace  Information , And in  Proxy  Medium cache  channelId  And  trace  Correspondence of information , Because of a single channel be-all SQL It's all serial execution , add  channelId  The quantity is controllable , It's not going to expand indefinitely . But this scheme is relative to each time  SQ L There is one time before the implementation  RAL  Execution of statements , The impact on performance is still relatively large . From the monitoring, every time  SQL  Executive  RT  Will float up  2-3ms( Network transmission ), It is fatal for some interfaces with long links .

(4) summary

Put it all together  2  In fact, both schemes have obvious disadvantages , Before addressing this issue  ShardingSphere  Our little partner came to our store for an in-depth communication , Brother Liang gives a feasible scheme , It is transmitted through virtual columns trace Information , But it needs to be upstream  SQL  To rewrite , This may increase the burden on upstream applications , At present, we haven't started to do this .
3.3.3 SQL  Insight
(1) background

at present  Proxy  Although there is printing logic  SQL  And physics  SQL  Log , However, due to the large production demand , Opening the log will affect  IO  There are big challenges . So our production environment is still closed . And this log cannot be connected in series with the upstream , The help for troubleshooting is also relatively limited .

(2) Solution

At present, there is only a general idea , There is no perfect scheme , The effect is to collect all  Proxy  Logic of execution  SQL、 Physics  SQL  And upstream information , Include  JDBCDatabaseCommunicationEngine  Other than  SQL( such as  TCL  wait ), And real-time query through the control console . The final effect is similar to that of Alibaba cloud  RDS  Service charge for -SQL  Insight

3.4 bug  Repair

For historical reasons , We are based on  Apache ShardingSphere 5.0.0-alpha  Secondary development on , In the actual use of the process encountered a lot  bug, Most of them have been mentioned to the authorities  issue, And the rainbow bridge version has been repaired , Of course  ShardingSphere  The small partners in the community also gave a lot of help and repair ideas .

3.5 JDBC&Proxy  Hybrid architecture

(1) background

above  4  Most of the items are for  Proxy  Modification of modules or upstream connection pool modules , But there are still some problems  Proxy  The mode cannot be solved for the time being , Like the one mentioned earlier  SQL  Compatibility and performance issues . And if the whole  Proxy  When all the clusters are down, we don't have a bottom-up mechanism . therefore  Proxy  The mode does not apply to all scenarios . stay  Apache ShardingSphere  You can see such a paragraph in the official document of :



So we are going to  JDBC&Proxy  Further exploration has been made on the hybrid architecture .

(2) Solution

Realize the mode configuration of the logic library on the control console , And through the self-developed connection pool  Rainbow  Sense and start different types of data sources according to different patterns . And it can be dynamically adjusted losslessly after switching modes in the background , This is completely imperceptible to the user . Yes  SQL  performance 、 Applications with high compatibility requirements can be adjusted to  JDBC  Pattern . At the same time when  Proxy  When all clusters are down, there is also a plan to cover all the details , So that the whole station will not collapse .

(3) Realization principle

Rainbow  When the connection pool is started, the corresponding mode of the current logical library will be queried , If it is  Proxy  Mode is directly connected  Proxy  To start the connection pool , If it is  JBDC  According to the data source configuration and fragmentation of the logical library & Read / write separation & Shadow library and other rules to load  JDBC  Data sources for patterns , Corresponding  DataSource  by  GovernanceShardingSphereDataSource. And will listen to this mode configuration , When the mode changes, the current connection pool will be dynamically and losslessly replaced . The specific lossless replacement scheme is similar to the cluster switching mentioned above . At the same time, there are monitoring problems and connection pool resource management that need to be solved ,Proxy  Switch to  JDBC  The exposure of post model indicators is determined by  Proxy  The node becomes an upstream node , The corresponding market also needs to be integrated , Connection pool management also uses a new computing mode to make corresponding adaptation .

4.  The current confusion

For historical reasons , We are based on  Apache ShardingSphere 5.0.0-alpha  Secondary development on , The latest version of the community is  5.1.2-release, from  5.0.0-alpha ~ 5.1.2-release  A lot of optimization and  bug  Repair , But we don't have a good way to incorporate community code into our internal code , Because we can't determine what the community's open source version changes , Will it affect the existing business , I can't enjoy the dividends brought by the community in a short time . At the same time, we are also looking for a way to incorporate some of our optimizations into the community , It can also be regarded as a kind of back feeding community . Make a contribution to open source in China ~

Welcome to the link , Learn more
Apache ShardingSphere  Official website :
https://shardingsphere.apache.org/
Apache ShardingSphere GitHub  Address :
https://github.com/apache/shardingsphere
SphereEx  Official website :
https://www.sphere-ex.com

版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
https://cdmana.com/2022/174/202206231832547804.html

Scroll to Top