The underlying implementation of the container is deeply dependent on many features of the kernel , Such as overlay file system ,namespace, cgroup etc. , So the functionality and stability of the kernel , For the most part , Determines the whole container PaaS The function and stability of the platform . from TKE It's been online for more than three years , Tens of thousands of clusters , From the operation experience of hundreds of thousands of nodes , Kernel problems account for about one-third of all node problems .
So which container scenario to choose Linux The best version ？ Tencent's cloud , The answer is Tencent Linux.
At this time, we should have a lot of questions , Next FAQ I will try my best to answer your questions .
Tencent Linux Who maintained ？ Based on what distribution ？
Tencent kernel and virtualization team are responsible for maintaining .Tencent Linux 2.4 be based on CentOS 7, User mode software package keeps up with the latest version CentOS 7 compatible ,CentOS 7 Version of the software package can be directly in Tencent Linux 2.4 Use in
Tencent Linux Follow CentOS What's the difference? ？
The key difference is in the kernel version , This article will introduce in detail . There are a few adjustments in user mode , Such as YUM Source configuration, etc , Please refer to the official introduction document for details ：https://cloud.tencent.com/document/product/213/38027
What version of the kernel is ？
Tencent Linux 2.4 At present, it is 4.14 kernel .
Code and rpm Wrapped in GitHub Can get ： https://github.com/Tencent/TencentOS-kernel.
At the end of the year 5.4 edition .
Tencent Linux Follow TKE Of Optimized What is the relationship between mirror images ？
They have the same kernel , but Tencent Linux 2.4 yes CVM Public image , TKE Optimized Mirror image is market mirror image .
TKE Will use
Tencent Linux2.4 To replace
CentOS 7.6 TKE Optimized as well as
Ubuntu18.04 TKE Optimized. Already in use
CentOS 7.6 TKE Optimized and
Ubuntu18.04 TKE Optimized The cluster can still be used , But in the future, the new cluster will no longer support these two kinds of images .
expect 11 From the end of the month ,TKE Users can operate on the console by themselves , Cluster the newly created nodes OS The image is switched to
Tencent Linux2.4, The operation entrance is shown in the figure below ( Change the operating system to
comparison CentOS and Ubuntu What's the advantage of the distribution ？
The main advantages are as follows , I'll give you a detailed introduction later ：
- After many years of internal business validation and polishing of Tencent .
- Support from a team of top kernel experts .
- Contains some key performance optimizations and customization features for container scenarios .
Tencent's internal business has been verified and polished for many years
Tencent Linux from 2010 R & D was launched in , Inside Tencent, it has been put into operation 10 year , The number of headquarters has reached the level of one million , Inside Tencent Linux In the system 99%, Tencent has covered all its business , At the same time, Tencent has the most diverse business ecology in China , From social , game , To financial payments ,AI, Safety, etc , So the stability of the underlying operating system , performance , Compatibility and so on have stronger requirements .
For container scenarios , Tencent's large number of core businesses have been containerized in part or in whole , For example, all logical services of wechat are containerized , According to the characteristics of wechat business , Carry out series optimization , Successfully guaranteed the annual Spring Festival red envelope peak operation , At the same time, in terms of data security , We also work closely with wechat , Provide solutions .
Support from kernel expert team
More than 30 full-time kernel experts currently support this kernel version . Among them is kvm Maintainer , And a lot of kernel networks , Storage ,cgroup, Experts in scheduling and other subsystems .
Support is also reflected in the rhythm of version update and hot patch service .
From the version record (https://github.com/Tencent/TencentOS-kernel/releases) You can see , from 7 Month to 10 month , Tencent Linux 4.14 The kernel family has released 5 A version . Tencent internal business and Tencent cloud external customers encountered the vast majority of problems , Can be located and repaired in time .
And we're looking at some important fixes , It will provide online repair of kernel hot patches . Hot patch installation and validation , No need to restart the machine , Without interrupting customer business , Improve the delay of customer service SLA.
For bug fixes ,Tencent Linux With a full set of hot patch solutions , Include ： Application level hot patches , Kernel level hot patch, etc . Every year 100 Multiple hot patches . Most vulnerabilities offer fixes within a week .
Tecent Linux According to the problems encountered in large-scale landing practice by internal and external users , A lot of performance optimization has been done for container scenarios , Including but not limited to :
- solve IPVS Mode high concurrency scenario , Connection reuse causes connection exceptions (#81775).
- solve IPVS Mode in high configuration node ( Many kernels ) Next IPVS Too many rules lead to network glitches .
- Solve the problem in the container intensive scenario ( There are many containers on a single node ),cAdvisor Read memcg Being trapped in the kernel state for a long time causes network glitch .
- Solve big Pod ( It takes up a lot of cores , High single core occupancy ) In high configuration nodes ( Many kernels ) scenario ,CPU Load balancing causes network glitches .
- Solve the problem of high concurrency TCP Connection monitoring ( For example, deploy separately cAdvisor Configure monitoring TCP Connect ) This leads to periodic network jitter .
- Optimize network packet receiving soft interrupt , Improve network performance .
These optimization results for various container scenarios are very significant , By the end of 3 Point for example ,ping The effect picture of time delay monitoring is as follows (11:00 And then there's optimization ):
Container customization features
Container resource display isolation
quite a lot golang, java The efficient operation of a program depends on getting the available CPU And memory resources . But what this kind of program gets in the container is node's CPU And memory resources , Does not match the resource allocated by the actual container , The thread pool and other parameters of the process are often unreasonable , And that leads to problems .
The mainstream solution for the community is through deployment FUSE Realized LXCFS To achieve /proc/cpuinfo, /proc/meminfo And other resources are displayed in containers . This solution needs to be deployed on nodes LXCFS file system , Also need to go to POD sepc Insert the relevant volume And mount point configuration . For details, please refer to ：Kubernetes Demystified: Using LXCFS to Improve Container Resource Visibility
Tencnet Linux The kernel implements something like LXCFS characteristic , Users do not need to deploy on the node LXCFS file system , There is no need to modify POD spec. Just turn on a global switch at the node （
sysctl -w kernel.stats_isolated=1）, Read from the container /proc/cpuinfo, /proc/meminfo The file is isolated by container , It's that simple .
in addition , Considering some special containers , For example, node monitoring components , Maybe you need to read node level information . To solve this problem , Specially added container level switch
kernel.container_stats_isolated. With the master level switch on , Just in the container's startup script , Turn off the container level switch (
sysctl -w kernel.container_stats_isolated=0), Read it in this container later /proc/cpuinfo, /proc/meminfo The information of the host computer is the file .( notes ： The container level switch must be set in the container , To work with this container )
Please refer to the detailed usage document ： In container CPU、 Memory 、 process 、 Disk and other information isolation
More isolation of kernel parameters
These kernel parameters are often customized and modified by business . But the community kernel does not do these parameters namespace Isolation . Modification of the above parameters by a container , Will work on the host and all other containers . Tencent Linux According to the needs of internal and external customers , Implementation of these kernel parameters namespace Isolation , The business container can be assured to personalize these parameters without worrying about interference with other services .
Container default kernel parameter optimization
In the case of high concurrency , Packet loss may occur when the semi connection queue is full , You can turn it up
net.core.somaxconn To ease the problem . But container networks namespace Inside
net.core.somaxconn The default value is only 128, And it's code dead . stay Tencent Linux The kernel , We set the default value to 4096, So it can reduce the packet loss problem when the semi connection queue is full in the case of high concurrency .
stay TKE How to use Tencent Linux
If you want to TKE Cluster nodes use Tencent Linux Operating system of , Need to create TKE When the cluster , Operating system selection
Tencent Linux Of :
notes : Tencent Linux In addition to supporting ordinary cloud server models , It also supports Blackstone physics and Nvidia GPU Model of .
- Tencent Linux Official profile
- Tencent Linux Kernel code
- Container resource display isolation usage document
【 Tencent cloud native 】 Cloud said new products 、 Cloud research new technology 、 Travel new life 、 Cloud View information , Scan code is concerned about the official account number of the same name , Get more dry goods in time ！！