欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

istio sidecar自动注入过程分析

程序员文章站 2023-01-18 19:06:48
istio sidecar自动注入过程分析 sidecar自动注入检查 istio通过 "mutating webhook admission controller" 机制实现sidecar的自动注入.istio sidecard在每个服务创建pod时都会被自动注入. 检查kube apiserve ......

istio sidecar自动注入过程分析

sidecar自动注入检查

istio通过机制实现sidecar的自动注入.istio sidecard在每个服务创建pod时都会被自动注入.

检查kube-apiserver

webhook支持需要kubernets1.9或者更高的版本,使用以下命令查看

[root@test1 ~]# kubectl api-versions | grep admissionregistration
admissionregistration.k8s.io/v1beta1

同时检查kube-apiserver有没按顺序加入参数mutatingadmissionwebhookvalidatingadmissionwebhook

检查sidecar-injector的configmap

在sidecar-injector的configmap中设置policy=enabled字段来查看是否启用自动注入

[root@test1 ~]# kubectl describe cm istio-sidecar-injector -n istio-system
name:         istio-sidecar-injector
namespace:    istio-system
labels:       app=istio
              chart=istio-1.0.3
              heritage=tiller
              istio=sidecar-injector
              release=istio
...
data
====
config:
----
policy: enabled

检查namespace标签

为需要自动注入的namespace打上标签istio-injection: enabled

[root@test1 ~]# kubectl get namespace -l istio-injection
name           status    age       istio-injection
default        active    3d        enabled
istio-system   active    3d        
kube-public    active    3d        
kube-system    active    3d 

kubectl label namespace default istio-injection=enabled

sidecar自动注入过程

webhook过程

查看sidecar的webhook

[root@test1 ~]# kubectl get mutatingwebhookconfiguration -n istio-system
name                     created at
istio-sidecar-injector   2018-11-12t09:14:44z

[root@test1 ~]# kubectl describe mutatingwebhookconfiguration istio-sidecar-injector -n istio-system
name:         istio-sidecar-injector
namespace:    
labels:       app=istio-sidecar-injector
              chart=sidecarinjectorwebhook-1.0.3
              heritage=tiller
              release=istio
... ...
webhooks:
client config:
    ... ...
    service:
      name:        istio-sidecar-injector
      namespace:   istio-system
      path:        /inject
  failure policy:  fail
  name:            sidecar-injector.istio.io
  namespace selector:
    match labels:
      istio - injection:  enabled
  rules:
    api groups:
      
    api versions:
      v1
    operations:
      create
    resources:
      pods

由上面可以看出创建pod时会调用sidecar的webhook,接着向istio-sidecar-injector的服务发送inject注册(post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s).

查看istio-sidecar-injector的日志

[root@test-1 ~]# kubectl get pods -n istio-system | grep istio-sidecar
istio-sidecar-injector-d96cd9459-lbf66      1/1       running       0          13d

[root@test-1 ~]# kubectl logs istio-sidecar-injector-d96cd9459-lbf66 -n istio-system
2018-11-09t06:40:53.895979z info  admissionreview for kind=/v1, kind=pod namespace=default name= () uid=67d96021-e3ea-11e8-a721-00163e0c1d10 rfc6902patchoperation=create userinfo={system:unsecured  [system:masters system:authenticated] map[]}
2018-11-09t06:40:53.897821z info  admissionresponse: patch=[{"op":"add","path":"/spec/initcontainers","value":[{"name":"istio-init","image":"docker.io/istio/proxy_init:1.0.0","args":["-p","15001","-u","1337","-m","redirect","-i","10.0.0.1/24","-x","","-b","80,","-d",""] ... ...},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"docker.io/istio/proxyv2:1.0.0","args":["proxy","sidecar",... ...\"initcontainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagepullsecrets\":null}"}}]

hook发送inject后,sidecar会返回两个container,istio-init和istio-proxy.下面我们来具体分析下.

获取pod具体信息

[root@test-1 ~]#kubectl describe pod nginx-dm-fff68d674-9tv9w 
name:           nginx-dm-fff68d674-9tv9w
namespace:      default
node:           10.0.3.126/10.0.3.126
start time:     fri, 09 nov 2018 14:40:53 +0800
labels:         name=nginx
                pod-template-hash=999248230
annotations:    sidecar.istio.io/status={"version":"5aa52d92ced8dab93e04a5a4701773b2f3d78968c04b05bb430f32e80a4d9be1","initcontainers":["istio-init"],"containers":["istio-proxy"],...
status:         running
ip:             172.30.2.21
controlled by:  replicaset/nginx-dm-fff68d674
init containers:
  istio-init:
    container id:  docker://43668b6cf4bb331542b8d98348a7670dad99b735aa0ef0ca572bf4ee1966538b
    image:         docker.io/istio/proxy_init:1.0.0
    image id:      docker-pullable://istio/proxy_init@sha256:345c40053b53b7cc70d12fb94379e5aa0befd979a99db80833cde671bd1f9fad
    port:          <none>
    host port:     <none>
    args:
      -p
      15001
      ... ...

containers:
  containers:
  nginx:
    container id:   docker://d917ffa9282bc4f82a0af1c8cbd6b51c0392fca6a85de6f8db6da128700db204
    image:          nginx:alpine
    image id:       
    port:           80/tcp
    host port:      0/tcp

  istio-proxy:
    container id:  docker://932a8bc6b85f1106cde057bd55598337bf7f9963fc4e796d3d88907d717a8eff
    image:         docker.io/istio/proxyv2:1.0.0
    image id:      docker-pullable://istio/proxyv2@sha256:77915a0b8c88cce11f04caf88c9ee30300d5ba1fe13146ad5ece9abf8826204c
    port:          <none>
    host port:     <none>
    args:
      proxy
      sidecar
      --configpath
      /etc/istio/proxy
      --binarypath
      /usr/local/bin/envoy
      --servicecluster
      ... ...

由具体信息可知,pod除了自身的容器外,还额外注入了两个容器.这就是由istio-sidecar-injector完成的.

proxy_init

proxy_init是一个init containers.init containers用于pod中执行初始化的任务,执行完毕退出后,才会执行后面的containers.

[root@test-1 ~]# docker inspect  docker.io/istio/proxy_init:1.0.0
[
    {
        "repotags": [
            "istio/proxy_init:1.0.0",
            "gcr.io/istio-release/proxy_init:1.0.0"
        ],
        "containerconfig": {
            ...
            "cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "entrypoint [\"/usr/local/bin/istio-iptables.sh\"]"
            ],
            ...
        },
]

如上cmd可以知道,这个容器主要执行的是的脚本.

查看脚本内容

...
while getopts ":p:u:g:m:b:d:i:x:h" opt; do
  case ${opt} in
    p)
      proxy_port=${optarg}
      ;;
    u)
    ...

该脚本通过配置iptable来劫持pod中的流量.结合前面的-p 15001可知pod的数据流量被转发向envoy的15001端口.

proxyv2

查看pod内istio-proxy的进程

[root@test-1 ~]# kubectl exec nginx-dm-fff68d674-9tv9w -c istio-proxy -- ps -ef
uid        pid  ppid  c stime tty          time cmd
istio-p+     1     0  0 nov09 ?        00:00:12 /usr/local/bin/pilot-agent proxy sidecar --configpath /etc/istio/proxy --binarypath /usr/local/bin/envoy --servicecluster istio-proxy --drainduration 45s --parentshutdownduration 1m0s --discoveryaddress istio-pilot.istio-system:15007 --discoveryrefreshdelay 1s --zipkinaddress zipkin.istio-system:9411 --connecttimeout 10s --statsdudpaddress istio-statsd-prom-bridge.istio-system:9125 --proxyadminport 15000 --controlplaneauthpolicy none
istio-p+    24     1  0 nov09 ?        00:42:50 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-proxy --service-node sidecar~172.30.2.21~nginx-dm-fff68d674-9tv9w.default~default.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only

上面有两个进程pilot-agent和envoy.
pilot-agent根据k8s api生成配置信息,并负责管理(启动,热更新,关闭等)整个envoy.生成的配置信息在 /etc/istio/proxy/envoy-rev0.json,具体内容可自己查看.

envoy由pilot-agent进程启动,envoy读取pilot-agent为它生成的配置文件(envoy-rev0.json),然后根据该文件的配置获取到pilot的地址,通过数据面标准api的xds接口从pilot拉取动态配置信息.

参考文档:
1.https://istio.io/docs/setup/kubernetes/sidecar-injection/
2.https://zhaohuabing.com/post/2018-09-25-istio-traffic-management-impl-intro/