In the field of software delivery ,GitOps It's a hot trend in the near future , It follows and expands DevOps、 Infrastructure is code and CI/CD Wait for the trend . We've published a lot about GitOps Introduction to , You can go to 【Rancherlabs】 Official account back office reply 【Git article 】 obtain GitOps Collection of articles .
GitOps The advantages can be summarized as follows ：
- Audit changes freely
- Continuous integration and delivery
- Better control of change management
However , The reality is to build GitOps Assembly line is not easy , It involves a lot of decisions, big and small , And these decisions will bring a lot of trouble to the implementation . We call these decisions “GitOps framework ”, It may lead to many challenges in the implementation process .
The good thing is that as long as you have some planning and experience , You can greatly reduce the transition to GitOps The pain of delivery mode .
In this paper , I'm going to explain some of the challenges through the story of a company . The company adopted... From a small, fragmented start-up GitOps, Grow into a standard multinational enterprise . Although this accelerated growth is rare , But it does reflect many teams in large organizations from proof of concept , To the least viable product （minimum viable product）, And then to the experience of mature systems .
If you just started , The easiest way is to create a single Git repo, Put all the code you need in it . These may include ：
Dockerfile, Used to build an application image
some CI/CD Pipeline code （ for example GitLab CI/CD or GitHub
Terraform, To configure the resources needed to run the application
Besides , All the changes are direct to master change , So the change can take effect directly .
The main advantage of this approach is that you have a single reference point , And all the code will be tightly integrated . If all your developers are fully trusted , And speed is everything , So this method will continue to work .
Unfortunately , As your business grows , The disadvantages of this method will soon show up .
First , As more and more code is added to the code base , The expansion of the code base can confuse Engineers , Because they will encounter more conflicts between changes that have to be resolved . If team members grow significantly , Then the ensuing redistribution of work or consolidation can lead to further confusion .
secondly , If you need to control the pipeline separately , You will encounter difficulties . Sometimes , You just want to quickly test changes to the code , Instead of deploying for complete end-to-end delivery . This approach creates more and more problems to be solved in terms of integrity , As these changes go on, they may affect the work of others .
Third , As you grow , You may want the boundaries of responsibility between the engineer and the team to be more detailed . This can be done through a single repo To achieve , And one repo It's usually a clearer and cleaner boundary .
As the business grows , The assembly line will be more and more crowded ,merge It's starting to get painful . Besides , Your team also needs to be professional , Divide different areas of responsibility into different members .
So you need to put Repo separate from . At this time , You have to face a lot of decisions first , for example repo How far should it be separated ？ Whether you need to create a separate repo？ It seems reasonable ？ And then put Docker Build things together ？ There's no point in such separation .
So all the teams Terraform What about code? ？ It should be put in a new repo Li ？ That sounds reasonable , however ： The newly created Central “ platform ” The team wants to control right AWS Middle core IAM（ Identity and access management ） Rule defined access , And the team RDS The configuration code is also in it , The development team needs to adjust it on a regular basis .
So you decided to Terraform Separate into two repo： One is “ platform ”repo, One is “ Specific applications ”repo. This brings another challenge , Because you need to separate now Terraform Status file of . This is not an unsolvable problem , But it's not the fast feature delivery you're used to , So the product manager will have to explain why the function request takes longer than before .
Unfortunately , these GitOps There is no established best practice or model for decision making .
The problem of separation goes beyond that . before , The coordination between the components built within the pipeline is negligible （ Because all the components needed are co-exist ）, And now you have to coordinate repo The flow of information between . for example ： When building a new Docker Mirror image , This may require triggering a centralized platform repo Deployment in , At the same time, pass the new image name as part of the trigger .
Again , These are not problems that cannot be solved , But building GitOps The early days of the assembly line , These challenges are easier to achieve . later , When step 、 When policies and processes are more mature , It takes more time to make changes .
Distributed vs Centralized
Your business is growing , You're building more and more applications and services . More and more obvious is , In terms of how to build and deploy applications , You need some kind of structural consistency . The central platform team needs to try to implement these standards . But you may face opposition from the development team , They think it's related to DevOps and GitOps Before appearance , In a centralized IT They've been given more autonomy and control in this process .
If the above situation seems familiar to you , That could be because GitOps It is similar to the monomer and microservice debate in the field of application architecture . As you can see in these debates , As the system matures , The expansion of scale and scope , Distributed and centralized IT The tension between them will be more and more frequent .
In a way , Yours GitOps Processes are like other distributed systems , If you don't design well , There is a problem in one of the parts, which may lead to unexpected problems .
Before you decide to separate repo At the same time , You realize that you need a consistent way to manage different deployment environments . And it's not going to work , Because it's time to QA The team , Test the changes before going online .
Now you need to mirror your application in testing and QA Specify different... In the environment Docker label , You may also want to enable instance size or replica functionality of different sizes in different environments . How do you manage the configuration of these different environments in the source code ？ A more straightforward approach is to create a separate... For each environment Git Warehouse （ Such as ：super-app-dev,super-app-qa,super-app-live）.
Separate repo Yes “ be quite distinct from each other ” The benefits of , It's like we divide it up on top Terraform That's what you see when you code . However , Few people will eventually like this solution , Because most teams don't have Git Knowledge and related professional level , And then in different repo Migrate changes between . What's more complicated is ,repo There must be a lot of duplicate code between them , And over time , There could also be a lot of drift （drift）.
If you want to keep things in a single place repo in , You have at least three options ：
- Each environment has a directory
- Every environment has a branch
- Each environment has a label
Synchronization step selection
If you rely heavily on YAML Build tools or templates , You can think of another way . for example ,Kustomize Directory based environment separation is strongly encouraged . If you are using the original YAML, Then the branch or tag method is more suitable for you .
The particle size of the running environment
However , In your runtime environment , You can choose what level of separation you want . At the cluster level , If you are using Kubernetes, You can choose in the following situations ：
- One cluster manages all
- One cluster per environment
- One cluster per team
In extreme cases , You can put all the environments in one cluster . But usually , In most organizations, there is at least one separate cluster for production .
Once you have your clustering strategy in mind , At the namespace level , You can still choose ：
- Every environment has a namespace
- Every application / The service has a namespace
- Each engineer has a namespace
- Every build has a namespace
Platform teams usually start from “dev”、“test”、“prod” Start with the namespace settings for , Then they realized that they wanted to divide the work of the team more finely .
You can also mix and match these options —— for example , For every engineer "desk testing " Namespace , And the namespace of each team .
We're here just for mature GitOps A brief introduction to the decision areas required by the process . If your business really grows into that multinational , You can also consider RBAC/IAM Other requirements .
Usually , Introduction GitOps It's just an investment , There may not be much satisfactory output in the end . But in GitOps Before , Teams often experience confusion and delays , Because no one can be sure what state anything should be . These will lead to secondary costs , Because auditors do spot checks , Interruptions caused by unexpected and unrecorded changes take a lot of employees' attention , It's a high cost .
However , With you GitOps The process is getting more mature , The benefits are multiplied , This process will solve many of the problems that existed before . But more often , The pressure on you is to show more quickly GitOps The advantages of the process .
at present GitOps The biggest challenge is , There is no established pattern or best practice to guide your choice .GitOps adviser , Usually it's just guiding the team to find the best solution for them , And guide the team in certain directions based on experience .
But what I've observed is , Early because it looks like “ Too complicated ” And the abandoned choice , I always regret it later . That doesn't mean you should jump directly to creating a namespace for each build , Each team has a Kubernetes The degree of clustering , There are two reasons ：
Whenever you give GitOps When architecture increases complexity , You'll end up delivering a viable GitOps The cost and time of the solution
You may never really need this setting
Before we accept viable standards in this field , Correct GitOps Architecture is always an art , Not science .
Link to the original text ：
About Rancher Labs
Rancher Labs from CloudStack The father of Liang Sheng created . Flagship products Rancher It's an open source enterprise Kubernetes Management platform , Realized Kubernetes Clusters are mixing clouds + Centralized deployment and management of local data center .Rancher Always because of the intuitive operation experience 、 Extremely simple equipment is favored by users , By Forrester named 2018 Leading manufacturer of global container management platform , By Gartner named 2017 The coolest cloud infrastructure provider in the world .
at present Rancher It has more than 300 million core image downloads worldwide , And owns, including China life 、 Huawei 、 China safe 、 Societe generale 、 Minsheng Bank 、 Ping An Securities 、 HNA technology 、 Xiamen Airlines 、 Shanghai automotive industry corporation 、 haier 、 Michelin 、 Toyota 、 Honda 、 CSIC 、 Zoomlion Heavy Industries Co., Ltd 、 Disney 、IBM、Cisco、Nvidia、 Pfizer pharmaceutical 、 Siemens 、CCTV、 China Unicom and other famous enterprises in the world 40000 Enterprise customers .