编程知识 cdmana.com

Practice of micro service and continuous delivery based on docker in cloud

This article is based on 〖2016 Global operation and Maintenance Conference • ShenZhen Railway Station 〗 The content shared by the guests of the live speech is organized , The editor is Wu Zhaojun @ tencent .

Welcome to your attention “ Efficient operation and maintenance ( WeChat ID:greatops)” official account , In order to preemptively read a variety of original articles full of dry goods .

About the lecturer


Graduated from Beijing University , Get a bachelor's degree and a master's degree ; Currently responsible for the research and development of products related to Alibaba cloud container technology .

Before joining Ali , I was in IBM China Development Center work 14 year , As senior technical specialist , be responsible for IBM Enterprise platform cloud product line PureApplication System R & D of ; Also responsible for and involved in a series of IBM stay Web 2.0,SOA Middleware development and innovation , We have also provided SOA Technical consultation and project implementation .


Hello everyone , The theme of my speech is 《 The cloud is based on Docker Micro service and continuous delivery practice of 》, I will mainly introduce :

Docker And microservices

Cloud production environment deployment

application Docker Transformation

Continuous delivery process practice


Changes in delivery have changed the global economy

A lot of people start with the container story , I'm no exception . One hundred years ago , Something called a container , Changed the global law , Now in the cross-border trade 90% All the goods are transported in containers .

A humble single , A change in delivery , You can change the whole economy .

Docker In this era, it's a revolution that changes the whole software delivery , Now it is used in almost all operation and maintenance or architecture Docker To deliver , Why? ?


Docker in Alibaba

Alibaba Docker The use of is everywhere ,2011 year , Taobao began to adopt container technology , With some of the internal structure of Ali , It greatly improves the efficiency of development and operation and maintenance , Make the whole development more efficient . stay “ double 11” Such a big promotion , Played a huge role .


Ali Baichuan is an e-commerce company for mobile platform PaaS, It USES Docker Can support different programming environments , So that users can quickly run applications online .

Based on Ali's practice , We have been implementing container services on public cloud computing platforms since last year , You don't have to worry about Docker The underlying technology , Network storage 、 Don't care about resource management , You just care about yourself Docker The application deployment is OK .

meanwhile ,Docker It will play a huge role in many fields .

For example, a while ago, Google's AlphaGo, In fact, one of the most important frameworks behind computing is the deep learning framework , Building a deep learning framework is a very complex learning , You need to configure the software of the corresponding deep computing framework .

But the services provided to customers in Ali , All you have to do is use Docker Images can be quickly grouped together HPC To build a machine , You can get the computing power you need for deep learning on demand , Maybe one day you can develop one for PK.

Actually Docker The use of Alibaba is ubiquitous , You can imagine , A system as big as Ali , Including virtualization 、 database 、 big data 、 The Internet , Such a huge system , Hundreds of components , We're dealing with it now Docker Handle , Speed up its development, operation and output .

We can build the whole alicloud in a few hours . So ,Docker Has completely changed some of our scenarios for software delivery and operation and maintenance .

Why? Docker So important ?

Accelerate application delivery , Shorten the dream distance

We're talking about Internet companies , What is the core competitiveness of Internet companies ?

It's efficiency , It depends on whether you can launch the product as quickly as possible , Lowest cost , The fastest iteration .


And in the What was our software development like ten years ago ?

Our application is a holistic application , It's developed by a large team , This team is divided into development team and operation and maintenance team , The development team has a front-end team 、 Back end team 、 Database team , These teams may argue with each other for business needs , In the end, the whole software development iteration is extremely slow .

In traditional enterprises, the cycle of iteration is from half a year to one year , This speed is far from meeting the needs of the business side , At the same time, it is even more difficult for our operation and maintenance personnel , Until the last moment , Software personnel explain that it will be online soon , Give a bunch of installation scripts to the O & M staff , Let the operation and maintenance personnel to install and deploy , To ensure its high availability .

Can you imagine ? How could such a thing happen !

So , Everyone is carrying a Great challenge :

  • The first is slow change , In the Internet age, who becomes slow , Whoever dies quickly ;

  • second , In order to ensure such a large-scale overall application , It's very hard to scale , Usually in order to adapt to large traffic , We can only add new CPU And so on .

Everybody knows , If you want to guarantee the growth of linear computing performance , It may cost more . in addition , The continuous operation ability of the whole system is also bad , The bigger, the more unstable , One of the components is broken , There's an avalanche effect , The whole system goes down .

It's a very painful process , Ali went through this process , Today's Internet companies, including Alibaba, have evolved into the following structure .

The bottom layer is a computing architecture based on cloud services or virtualization , Every business such as e-commerce , There can be user management 、 Commodity management , Our shopping cart 、 shoppers 、 advertisement .

These are different modules , Each module is made up of a team , Each image can be deployed independently , Components communicate with each other through standardized protocols .

Only in this way can speed be guaranteed , In order to ensure that we can survive in the competition .

Docker And virtualization technology

Docker The emergence of the emergence of further push this matter ,Docker Is a lightweight operating system virtualization solution , Deliver more quickly .

Docker It has good portability , This is more important , We can develop 、 test 、 Use a unified method in production 、 Unified media to deliver software .

Imagine , If it's in a hybrid cloud scenario , for instance “ double 11 Great promotion ”, A lot of computing power now has been moved to the public cloud , Because it's distributed on demand , You can quickly extend the application to the entire data center .

combination Docker Container and virtualization technology

A lot of people say “Docker Changed the life of virtualization ”, We think Docker And virtualization are complementary in many ways .

Docker Technology still has its own limitations , such as Docker It's not good at isolating the system , Like virtualization Docker Can't do , At present, one of the most mature solutions is to combine virtualization technology with container technology .

In the public cloud, we are not recommended to adopt the multi tenant approach , Because there are a lot of security holes .

Cloud Native Computing

With the development of microservice and container technology , Last year, Google took the lead in setting up a Cloud Native Computing The foundation .

It defines some basic elements and frameworks for future native applications , Delivered as a container in a microservice architecture , Support DevOps, This platform is dynamically managed by itself , It's not manual .


Why do you do this ?

We should look at it from the perspective of sustainable development , The single application will touch the ceiling sooner or later , Its complexity 、 Scalability must hit the wall .

therefore , We use microservices , Microservices are not free lunches , It brings complexity as well as benefits , I used to run an app , Now we have to operate dozens of services .

I have been told by my colleagues that there are 20 Services , After decomposing into microservices, there are nearly 400 , His management is a lot more .

How to let services and services monitor their health ?

Once a service fails , We need to isolate it 、 Fuse 、 Downgrade , Then how do we update the version of this microservice , Previous products are not guaranteed to be interrupted ?

These are huge impacts .

If you let people do these things manually , Definitely not , Make sure you use the platform 、 In an automated way . therefore , That's why people are emphasizing the need for a platform to support .

Walking in the clouds : Start Docker The journey

In fact, using servers on the cloud is not complicated , People are very used to their own development 、 test 、 Data center use Docker technology , however Why don't you all on the cloud ?

People think virtual machines are needed 、 It's a lot of trouble with all kinds of things , But it's not like this .


Docker stay 2014 Launched at the end of the year Docker  Machine, utilize Docker  Machine You can quickly create a Docker The cloud environment of .

The thing to do is very simple , download Docker  Machine Then look for the driver of the cloud provider , For example, Alibaba cloud now 、 Amazon 、Aure Some of these Driver, adopt Driver, From the command line, I can deploy my containerized application on it , It's simple .

Used in a production environment Docker

But if it's really on the cloud 、 Choose... In the growing environment Docker, The challenge you face is much more than that .

One Docker Certainly not enough , It must be a cluster , How to manage this cluster , How to bridge the network , What about storage , How to schedule resources , How to arrange , It's a very complicated thing .

For this matter , Many Internet companies offer so-called products , Let me introduce some of them to you .

  • Docker  Cloud,Docker In the last year 11 I bought tutum.co In this year 2 Launched in April Docker  Cloud, It basically provides Docker A choreographed of its own API.

  • In the amazon 2014 year 11 Launched in April EC2 Container Service, It was originally based on a private of its own API Provides container described Services , But in the last year, it gradually began to support a wider range of Docker, To describe a composite containerized application .


  • Google's Compose  template, It's also in 2014 year 7 Open source in , It integrates a lot of ideas from the past , A lot of history of the corresponding scheduling . It was very popular after it was pushed out , But it provides a set of its own unique abstraction of containerized applications .

Container cluster management - Docker Swarm

To better illustrate some of our content , I'm going to introduce Ali container service today . In order to solve the problem that users are developing 、 Seamless migration of test environment , We are completely compatible with Docker Native choreography .
Docker What does the native choreography include ?

  • First of all Docker Swarm, It's a very delicate design , Can I put a group of Docker  engine Become a virtual Docker  engine, I'm all for this virtual Docker  engine Give orders , From its control node to a real node to execute .


  • Its architecture is very simple , Only one is needed on each node Docker  engine And then install another Agent Just fine ,Agent By reporting, you can automatically register a machine , Through this function, there is a node above that can find the node information inside , The cluster can be built automatically .

  • It's a very delicate abstraction , Because it's almost 99% To support all Docker Native API.

It brings two benefits

It can be owned by three parties and Docker Connected tools are integrated , Don't make any changes ;

Another advantage is that it provides a pluggable Architecture , Like its scheduler 、 Both storage and networking can be easily expanded .

And it has a big shortcoming

Swarm and Docker equally , Its basic abstract unit is the container , Not from the perspective of service .

Container arrangement - Docker Compose

Docker Compose From the Docker An acquisition of ,Docker Compose It describes how to combine a set of containers with resources related to the container .

For example, we take Wordpress For example , then Mysql, A simple layout template can describe it , adopt Wordpress Mirror together , Connect to through a link Mysql, adopt volume To create .

In this way, it is very elegant to describe how a set of containers work together in association , And you can start the whole application station with one button . If we scale it a little bit , It's very simple .



  • Simple to use , To facilitate the development . It's a great development tool , stay Docker There are more than 70%   People using Docker  Compose Do image development .

  • Expanded on the network 、 Storage support . Not only can you describe and container , You can also describe some associations between a container and its underlying resources .


For development and deployment , Automatic operation and maintenance is not supported . for instance , How to monitor the operation and maintenance , Is it possible to perform elastic contraction , It didn't do it , Because it's a development tool itself .

Alibaba container service

What is our ideal container development platform ?

One of the capabilities provided by our alicloud container service , First of all, the bottom layer is the public cloud computing platform or the enterprise's proprietary cloud . Above that is the container layer , except Docker outside ,Docker Warehouse , There's storage and networking .


Native Docker It's not enough , It provides a mechanism , We also recommend this mechanism to store blocks in the cloud 、 Object storage 、 Network storage can be easily integrated in .

Above the container layer is the cluster management and scheduling layer , We've done a lot of optimization and improvement , For example, we talked about an application just now , We need to make sure it's moving , We have so many things to do , We can't guarantee it won't go down , What can we guarantee ?

We can schedule resources , Even if one data center loses power , A regional data center is down , It can also guarantee resource scheduling .

in addition , From the perspective of container arrangement , How can we easily expose a service , It's easy to collect and monitor the container log ? We've done a lot of expansion , Still can do very good container application control .

Above that is our service layer , Alibaba's own microservice architecture has been built for a long time , It's very popular in the open source world , We internalize these experiences into our support capabilities .

One of the big keys is how to do service discovery 、 Routing of services , We've expanded a lot , adopt BNS Find out , Through load balancing to achieve dynamic load balancing between service nodes , Through these things , Let your micro service do well .

Above the service layer is the access layer , Give Way Web Applications can be very easy to access to your own applications .

This is the core of our entire container platform , But you know , A closed platform is not enough , Because containers don't solve all the problems , The container must be connected with the existing enterprise applications or cloud services .

We have done a good integration capability , It's very easy to integrate with cloud services , We can integrate with third-party tools , Integrate the container technology into your own development process , At the same time, we provide cloud control .

Control ability , In addition to its own access to our cloud monitoring log , In fact, all the control frameworks in our whole system can be extended at will , Because we think that a platform that is not open enough is basically playing rogue , Containers are not your islands of information , Be sure to keep up with the existing IT Control comes together .

We also provide many examples of how to use open source frameworks , Quickly build a cloud monitoring capability that you need , We have some examples in the following documents , You can go to see .

What applications can run in containers

So much has been said , In fact, people will care about :

How does my app run in the container ?

Which of my applications can be containerized ?

People ask these questions .


This is a very famous classification method that I extracted , It can help you choose what kind of application is suitable for running in the container 、 What kind of application is not suitable for .

It's based on two dimensions :

One dimension is the application of longevity or short life ;

The other is to see whether it is stateful or stateless .

  • The best use of containerized applications is the short-lived, stateless applications on the left , Because such an application is the easiest to deploy .

For example, a Web application , We can get rid of it easily and quickly , We don't care about it , You can deploy one very quickly Web application .

  • The other dimension is the short-lived , Like high performance computing 、 The batch .

Rendering a video must be a lot of state information , But this information can be done through Web Object storage to save , This kind of computing intensive task is also very suitable for doing on it , Because we can quickly identify a large group of clusters , Running such a task through a container like this .

  • There are also long-lived but stateless , For example, our development test environment has been used or our monitoring , It's going to be there all the time , But its state dependence is very small , We can also consider this kind of application .

  • Only the lower left is the most challenging dimension in general , It's a stateful service , Generally, stateful services need some adjustments , Including storage adjustment 、 Adjust the network , You still need to DBA Doing complicated work , It's not completely automated .

For some of these applications , Our suggestion is that container technology can be used in test and development , But it's not recommended to use such technology in production .

Docker It's a practical application : Ghost Blog

Next, I'll take a very popular example to explain how to container an application .

Ghost Blog is one of my favorite blog applications , It's simple , Very light , The mirror is also very easy to use , In a very simple way Ghost Mirror it .


But there are still many problems :

  • It's not scalable ;

  • It's not highly available .

All its data is stored locally Database Inside , If the virtual machine node is down , Migrate to another node , Data status is lost .

How can we solve this problem ?

In fact, you can refer to THE TWELVE-FACTOR standard , This specification is supported by most manufacturers , It's a very important programming rule now .

It has a few Core principles

  • The application should be decoupled from the running environment

  • The application should be decoupled from the external calling service

  • The application should be decoupled from the configuration

Through this decoupling, we can have some applications become stateless , Be able to deploy and operate on the cloud quickly .

Ghost Blog High availability cluster 1

We just need to add one more MySQL, Let it support MySQL It's OK to drive , Start a ghost+MySQ, adopt MySQL Connect , They are shared , Our container service has been optimized a lot .


Ghost Blog High availability cluster 2

We don't recommend using stateful services like databases in a production environment , What should we do ?

We see a lot of articles that say Docker It's very hard to use , Said it couldn't run the database ,Docker It doesn't run a database .

Why don't I connect directly to my database instance ?

Of course , There are different approaches , But I think the best thing to do is to make a minimal change , The application layer can mirror a database without any perception , Put one Docker The running use of becomes a Web, We've added an ability to expand .

We expand to quote Web Service for , We can deploy in the production environment , And your application layer doesn't change .

Ghost Blog High availability cluster 3

In the process, there is still a problem to be solved , Attachments uploaded by users , Like pictures , Still stored in local storage , It's not going to work . In order to do this , We have another thing .
We can go through Docker Of  Volueme  Plugin To solve , It provides a very flexible mechanism to support different storage types , Block storage is now supported 、 Object storage 、 Network file system .


And the more interesting thing is , All of our network drivers and Volume In fact, all the drivers run in the container , Because only in this way , We can carry out unified operation and maintenance and unified management of the whole system .

however Docker There are still flaws in this area ,Docker It's impossible to distinguish some of these network drivers , Will cause a restart Docker  Engine When , It's possible to kill your Volume  Driver, Your data is broken without saving it ;

Or say , It's the same with your network , It may kill your network driver without waiting for your application to kill , Your application and network will be completely disrupted , That's not going to work .

We actually made changes in the community , We can grade different daemons , You can start some containers , It can have a higher system level , It is loaded first at boot time , When it is stopped, it will be stopped finally , In the community 1.11 There will be similar work in the version , The community will solve this kind of work .

By treating both network drivers and volume drivers as containers , It can also bring us greater benefits .

Our whole system is very scalable , For example, we are talking about cooperation with a third-party network storage company , It now takes a container to deliver the storage driver , We can run the storage driver on the server without modifying a single line of code , This will make our system more scalable .

Containerized continuous integration and delivery

Docker One of the important benefits of is portability , Through portability, you can develop 、 Throughout the software life cycle of testing and production , Deliver our software products in the same way .

for instance , That's what our developers are like now , You can start the local development environment with one click , Submit the code when it's done , When it is submitted, it is the corresponding self code and the original .


With this , dependent Docker infrastructure , Such as the container 、 Mirror warehouse , You can compile the relevant code into Docker Mirror image , And throughout the test 、 This image has been used in production , All the steps can be repeated , And ensure consistency .

In the process , Everything can support Docker Managed , And it can be updated and managed quickly .

What's the benefit of doing this ?

The developer thought on the first day “ How my code goes online ”, It's a huge cultural and temporal change .

We used to talk about it DevOps, It's no use just for operation and maintenance personnel , You have to let developers think about it on the first day of development “ How software is delivered 、 How to be highly available in the cloud 、 Telescopic ”, This is a culture and an idea that has to be changed .

If that doesn't change , You can't change... With any technology .

We used to be the same in Ali , Developers are amazing , The operation and maintenance personnel are hard pressed , Developers develop , Operation and maintenance personnel stay up late online , Roll back when something goes wrong , Very inefficient .

But we now require developers to deliver one for each function Docker Mirror image , Put your preconditions 、 Postcondition 、 Check the script 、 The health check is delivered at the beginning , Without delivery , We operators stopped accepting code like this .

Through this , We can evolve quickly .

Simplified continuous delivery process

Source control , We can have a mirror service , It can subscribe to notifications from the source repository , Our container service can also subscribe to notification of mirror changes .

When your code changes , For example, modify a web page , Turn two columns into three columns , It tells the mirror service to pull the corresponding code build , After packaging it into a mirror image , Automatically notify the corresponding container service to update the ready-made application .

In a few minutes , Change online , We can also integrate with other services .


Complete continuous delivery process

How can Alibaba social platform monitor the change of source code warehouse , After code launch changes , It pulls the code and mirrors it , After passing the unit test, mirror it , Notify the continuous delivery server to proceed with the next step , It's an assembly line .

Take... In the assembly line Docker Mirror image and Docker File in test environment 、 Pretest environment 、 Deployment on production environment .


We have the same one Docker Mirror image , The same one Docker Templates , It can be used in different environments , So I can make sure that from the development 、 test 、 Consistency of everything online .

We must adhere to some of these ideas , because DevOps A lot of things we all know , What is lacking is persistence .

Immutable Architecture (immutable infrastructure)

Docker Why do you get DevOps Cheers from people in the field ?

In fact, the way we want it to be operated .

understand OOpenStac We all know the famous fable stock market , Your app is like a pet that needs your care all day , It's like a cow in a herd , It can be killed at any time without sadness , You can always have another cow to add .

Make this system self-sustaining , Very robust , The whole system will not be terminated due to the failure of any node .


Take advantage of immutability to maintain the infrastructure : Once instantiated , Never change ; It will only be replaced correctly by another instance .


  • Avoid inconsistencies between environments . It's a big part of our daily life , exceed 30% The online errors are caused by the inconsistency between the development environment and the test environment and the online environment .

As like as two peas, I can guarantee that all code is exactly the same as the door , Always run the way you expect , Test and launch 、 Production is the same thing .

  • Simplify deployment complexity . It's very difficult to patch and upgrade in place , Especially a lot of system software , There are many pairs. How many .

  • Low cost rollback . It's a lot more complicated to write a rollback code , And it's hard to be right , Because we never test rollback .

In fact, it's not new , There used to be a virtual machine , It's also possible to have an automated tool at home , however Docker Make it faster , Make it easier .

It's like saying , Every Docker The mirror image is actually immutable , It's a process , You can always replace it with a new image .

Docker It starts very fast , It used to take a few minutes to roll back a virtual machine , But if you use Docker The rollback time is in seconds , Users can't feel the interruption .

Docker: Immutable architecture dream come true

To achieve this, we also need to pay attention to , Because I know a lot of people still put Docker Containers are used as lightweight virtual machines , There's nothing wrong with this , It's just that everyone's scene is different .

however , I ask you to think twice when you do this , When you think of the container as a lightweight virtual machine , You have to think about it again , You may lose Docker A lot of good features , The most important thing is immutability .

To achieve this , We need to do something right :

  • Never manually modify the contents of the container , Your container should always be built out of code .

  • Try not to use latest As a mirror label , You have to roll back in production , You need to know what issues to roll back 、 Which version , One of the easiest ways to do this is to use Git   Commit As a mirror image tag Part of the , Easy to track , Make sure your online product knows exactly what version is running .


  • Don't store any variable data in a mirror image , To pass the Volume abstract , You can combine the application changes with the lifecycle of your container , The benefits it brings far outweigh laziness 、 The benefits of saving time .

Cross host container network Cloud practice

On the Internet, you can see a lot about Docker A lot of discussion on the Internet , Most of the discussion is based on your own data center , Because it's very simple in your own data center , You can even control the switch .

But on the public cloud, it's hard , We have to choose on the cloud Docker If the containers are interconnected , It must be adapted according to the maximum optimization of cloud manufacturer's configuration .

Generally, there are two solutions on the cloud to realize the interconnection between container networks across virtual machines :

  • adopt Overlay Methods , As long as the three layers are connected , adopt Overlay Realize virtual network .

Overlay This is a very general way , It can be used in different network environments , Even across different cloud providers . But it also has weaknesses , Its own performance is limited .

We're at Amazon 、 Alibaba cloud 、IBM It's been measured on the cloud , adopt Overlay Container interconnection performance and container communication through native virtual machine , Only bandwidth 70%, Also increases 20%-30% Delay of , For people who are sensitive to network performance, we don't recommend this .


  • Through the network characteristics of cloud provider network itself , such as VPC,VPC It's very different from today's Internet , stay VPC We can control some IT Distribute 、 Routing rules .

We get a lot of benefits in this way , Because in one VSwitch Inside, the whole two were once connected , We can even avoid Web technology , In the corresponding node with the corresponding routing table can achieve interoperability between containers .

But in one VSwitch If you want to do it, you have to bear a consequence , When a data center loses the chain, the application will be suspended .

In general, we recommend this method in production , Through one VRouter Deploy your application in different VSwitch On , And then in VRouter Configure routing rules , This is a very general way , Whether it's on Amazon or alicloud , We all use this way .

Its benefits , The bandwidth of communication between containers , On Alibaba cloud and Amazon , There is no difference between bandwidth and native speed , But the delay will be a little bit more , May be in 10% about .

therefore , If you pursue performance, you may consider , Of course, it's also limited by a VRouter How many routing tables are there , Finally, limit the maximum size of your cluster nodes .

Everybody knows , There is no best solution in the world , Be sure to choose the right plan according to your own situation .

My speech is over today , This is my blog address ( The cloud community Docker Team blog ), There are often a lot of and Docker Related examples , And these examples don't necessarily need to run on alicloud , Our big goal is any use of Docker Mirror image 、Docker Template applications can run in the cloud . Thank you for your time .

本文为[EXP 993tr4xp]所创,转载请带上原文链接,感谢

Scroll to Top