In the field of software delivery ,GitOps It's a hot trend in the near future , It follows and expands DevOps、 Infrastructure is code and CI/CD Wait for the trend . We've published a lot about GitOps Introduction to , You can go to 【Rancherlabs】 Official account back office reply 【Git article 】 obtain GitOps Collection of articles .

GitOps The advantages can be summarized as follows :

  • Audit changes freely
  • Continuous integration and delivery
  • Better control of change management

However , The reality is to build GitOps Assembly line is not easy , It involves a lot of decisions, big and small , And these decisions will bring a lot of trouble to the implementation . We call these decisions “GitOps framework ”, It may lead to many challenges in the implementation process .

The good thing is that as long as you have some planning and experience , You can greatly reduce the transition to GitOps The pain of delivery mode .

In this paper , I'm going to explain some of the challenges through the story of a company . The company adopted... From a small, fragmented start-up GitOps, Grow into a standard multinational enterprise . Although this accelerated growth is rare , But it does reflect many teams in large organizations from proof of concept , To the least viable product (minimum viable product), And then to the experience of mature systems .

Simple start

If you just started , The easiest way is to create a single Git repo, Put all the code you need in it . These may include :

  • Application code
  • Dockerfile, Used to build an application image
  • some CI/CD Pipeline code ( for example GitLab CI/CD or GitHub

  • Terraform, To configure the resources needed to run the application

Besides , All the changes are direct to master change , So the change can take effect directly .

The main advantage of this approach is that you have a single reference point , And all the code will be tightly integrated . If all your developers are fully trusted , And speed is everything , So this method will continue to work .

Unfortunately , As your business grows , The disadvantages of this method will soon show up .

First , As more and more code is added to the code base , The expansion of the code base can confuse Engineers , Because they will encounter more conflicts between changes that have to be resolved . If team members grow significantly , Then the ensuing redistribution of work or consolidation can lead to further confusion .

secondly , If you need to control the pipeline separately , You will encounter difficulties . Sometimes , You just want to quickly test changes to the code , Instead of deploying for complete end-to-end delivery . This approach creates more and more problems to be solved in terms of integrity , As these changes go on, they may affect the work of others .

Third , As you grow , You may want the boundaries of responsibility between the engineer and the team to be more detailed . This can be done through a single repo To achieve , And one repo It's usually a clearer and cleaner boundary . Insert picture description here

Separate Repository

As the business grows , The assembly line will be more and more crowded ,merge It's starting to get painful . Besides , Your team also needs to be professional , Divide different areas of responsibility into different members .

So you need to put Repo separate from . At this time , You have to face a lot of decisions first , for example repo How far should it be separated ? Whether you need to create a separate repo? It seems reasonable ? And then put Docker Build things together ? There's no point in such separation .

So all the teams Terraform What about code? ? It should be put in a new repo Li ? That sounds reasonable , however : The newly created Central “ platform ” The team wants to control right AWS Middle core IAM( Identity and access management ) Rule defined access , And the team RDS The configuration code is also in it , The development team needs to adjust it on a regular basis .

So you decided to Terraform Separate into two repo: One is “ platform ”repo, One is “ Specific applications ”repo. This brings another challenge , Because you need to separate now Terraform Status file of . This is not an unsolvable problem , But it's not the fast feature delivery you're used to , So the product manager will have to explain why the function request takes longer than before .

Unfortunately , these GitOps There is no established best practice or model for decision making .

The problem of separation goes beyond that . before , The coordination between the components built within the pipeline is negligible ( Because all the components needed are co-exist ), And now you have to coordinate repo The flow of information between . for example : When building a new Docker Mirror image , This may require triggering a centralized platform repo Deployment in , At the same time, pass the new image name as part of the trigger .

Again , These are not problems that cannot be solved , But building GitOps The early days of the assembly line , These challenges are easier to achieve . later , When step 、 When policies and processes are more mature , It takes more time to make changes .

Distributed vs Centralized

Your business is growing , You're building more and more applications and services . More and more obvious is , In terms of how to build and deploy applications , You need some kind of structural consistency . The central platform team needs to try to implement these standards . But you may face opposition from the development team , They think it's related to DevOps and GitOps Before appearance , In a centralized IT They've been given more autonomy and control in this process .

If the above situation seems familiar to you , That could be because GitOps It is similar to the monomer and microservice debate in the field of application architecture . As you can see in these debates , As the system matures , The expansion of scale and scope , Distributed and centralized IT The tension between them will be more and more frequent .

In a way , Yours GitOps Processes are like other distributed systems , If you don't design well , There is a problem in one of the parts, which may lead to unexpected problems .

Ring   habitat

Before you decide to separate repo At the same time , You realize that you need a consistent way to manage different deployment environments . And it's not going to work , Because it's time to QA The team , Test the changes before going online .

Now you need to mirror your application in testing and QA Specify different... In the environment Docker label , You may also want to enable instance size or replica functionality of different sizes in different environments . How do you manage the configuration of these different environments in the source code ? A more straightforward approach is to create a separate... For each environment Git Warehouse ( Such as :super-app-dev,super-app-qa,super-app-live).

Separate repo Yes “ be quite distinct from each other ” The benefits of , It's like we divide it up on top Terraform That's what you see when you code . However , Few people will eventually like this solution , Because most teams don't have Git Knowledge and related professional level , And then in different repo Migrate changes between . What's more complicated is ,repo There must be a lot of duplicate code between them , And over time , There could also be a lot of drift (drift).

If you want to keep things in a single place repo in , You have at least three options :

  • Each environment has a directory
  • Every environment has a branch
  • Each environment has a label

 Insert picture description here

Synchronization step selection

If you rely heavily on YAML Build tools or templates , You can think of another way . for example ,Kustomize Directory based environment separation is strongly encouraged . If you are using the original YAML, Then the branch or tag method is more suitable for you .
 Insert picture description here

The particle size of the running environment

However , In your runtime environment , You can choose what level of separation you want . At the cluster level , If you are using Kubernetes, You can choose in the following situations :

  • One cluster manages all
  • One cluster per environment
  • One cluster per team

In extreme cases , You can put all the environments in one cluster . But usually , In most organizations, there is at least one separate cluster for production .

Once you have your clustering strategy in mind , At the namespace level , You can still choose :

  • Every environment has a namespace
  • Every application / The service has a namespace
  • Each engineer has a namespace
  • Every build has a namespace

Platform teams usually start from “dev”、“test”、“prod” Start with the namespace settings for , Then they realized that they wanted to divide the work of the team more finely .

You can also mix and match these options —— for example , For every engineer "desk testing " Namespace , And the namespace of each team .

total junction

We're here just for mature GitOps A brief introduction to the decision areas required by the process . If your business really grows into that multinational , You can also consider RBAC/IAM Other requirements .

Usually , Introduction GitOps It's just an investment , There may not be much satisfactory output in the end . But in GitOps Before , Teams often experience confusion and delays , Because no one can be sure what state anything should be . These will lead to secondary costs , Because auditors do spot checks , Interruptions caused by unexpected and unrecorded changes take a lot of employees' attention , It's a high cost .

However , With you GitOps The process is getting more mature , The benefits are multiplied , This process will solve many of the problems that existed before . But more often , The pressure on you is to show more quickly GitOps The advantages of the process .

at present GitOps The biggest challenge is , There is no established pattern or best practice to guide your choice .GitOps adviser , Usually it's just guiding the team to find the best solution for them , And guide the team in certain directions based on experience .

But what I've observed is , Early because it looks like “ Too complicated ” And the abandoned choice , I always regret it later . That doesn't mean you should jump directly to creating a namespace for each build , Each team has a Kubernetes The degree of clustering , There are two reasons :

Whenever you give GitOps When architecture increases complexity , You'll end up delivering a viable GitOps The cost and time of the solution
You may never really need this setting

Before we accept viable standards in this field , Correct GitOps Architecture is always an art , Not science .

Link to the original text :

About Rancher Labs

Rancher Labs from CloudStack The father of Liang Sheng created . Flagship products Rancher It's an open source enterprise Kubernetes Management platform , Realized Kubernetes Clusters are mixing clouds + Centralized deployment and management of local data center .Rancher Always because of the intuitive operation experience 、 Extremely simple equipment is favored by users , By Forrester named 2018 Leading manufacturer of global container management platform , By Gartner named 2017 The coolest cloud infrastructure provider in the world .

at present Rancher It has more than 300 million core image downloads worldwide , And owns, including China life 、 Huawei 、 China safe 、 Societe generale 、 Minsheng Bank 、 Ping An Securities 、 HNA technology 、 Xiamen Airlines 、 Shanghai automotive industry corporation 、 haier 、 Michelin 、 Toyota 、 Honda 、 CSIC 、 Zoomlion Heavy Industries Co., Ltd 、 Disney 、IBM、Cisco、Nvidia、 Pfizer pharmaceutical 、 Siemens 、CCTV、 China Unicom and other famous enterprises in the world 40000 Enterprise customers .