使用 bpf 监控 kubernetes 集群(k8s bpf 工具 kubectl-爱代码爱编程 (2024)

写在前面

学习中遇到，整理分享，博文内容涉及：
kubectl-trace 安装，在节点，容器中如何使用
需要注意的问题：job 闪完成，一直 Pending 状态解决
理解不足小伙伴帮忙指正

不必太纠结于当下，也不必太忧虑未来，当你经历过一些事情的时候，眼前的风景已经和从前不一样了。——村上春树

kubectl-trace 安装

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$curl -L -o kubectl-trace.tar.gz https://github.com/iovisor/kubectl-trace/releases/download/v0.1.0-rc.1/kubectl-trace_0.1.0-rc.1_linux_amd64.tar.gz└─$pwd/root/ansible/trace┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$lskubectl-trace.tar.gz

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$tar -xvf kubectl-trace.tar.gzLICENSEREADME.mdkubectl-tracetrace-runner┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$mv kubectl-trace /usr/local/bin/kubectl-trace┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$

查看版本，检查安装是否成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl trace versiongit commit: d34d1d586b110af718aeadc9f3213c78e543a961build date: 2019-09-19 23:00:13 +0800 CST

如何使用

这里我们看一下帮助文档

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl trace run --helpExecute a bpftrace program on resourcesUsage: trace run (POD | TYPE/NAME) [-c CONTAINER] [--attach] [flags]Examples: # Count system calls using tracepoints on a specific node kubectl trace run node/kubernetes-node-emt8.c.myproject.internal -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }' # Execute a bpftrace program from file on a specific node kubectl trace run node/kubernetes-node-emt8.c.myproject.internal -f read.bt # Run an bpftrace inline program on a pod container kubectl trace run pod/nginx -c nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" # Run a bpftrace inline program on a pod container with a custom image for the init container responsible to fetch linux headers kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); } --init-imagename=quay.io/custom-init-image-name --fetch-headers" # Run a bpftrace inline program on a pod container with a custom image for the bpftrace container that will run your program in the cluster kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); } --imagename=quay.io/custom-bpftrace-image-name"...............┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$

在节点上使用

在使用之前，可能需要做一些准备工作，提前拉去镜像

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible k8s_node -m shell -a "docker pull quay.io/iovisor/kubectl-trace-bpftrace:latest" -i host.yaml┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible k8s_node -m shell -a "docker pull quay.io/iovisor/kubectl-trace-init:latest" -i host.yaml

使用 tracepoints 在特定节点上计算系统调用次数(帮助文档中的Demo,这里实际上是动态跟踪)：

kubectl trace run node/kubernetes-node-emt8.c.myproject.internal -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }'

kprobe:do_sys_open 是一个 bpftrace 的探针（probe），用于跟踪 Linux 内核中的 do_sys_open 系统调用。

实际上kprobe 是一种用于在内核函数入口处执行的探针类型,属于动态跟踪。do_sys_open 是一个内核函数，它在打开文件时被调用。而 tracepoint 属于静态跟踪

实际的 动态跟踪 Demo

运行方式，这里使用编码方式，传递字符串

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl trace run node/vms103.liruilongs.github.io -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }'trace a659c18c-e50c-11ee-ba0d-000c290e5d5f created

会启动一个 job 在对应的节点上

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl get jobsNAME COMPLETIONS DURATION AGEkubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f 0/1 20s 20swebhook-cert-setup 0/1 124d 124d

获取 pod 信息

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl get jobs | grep a659c18c-e50c-11ee-ba0d-000c290e5d5fkubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f 0/1 29s 29s┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl get pods | grep a659c18c-e50c-11ee-ba0d-000c290e5d5fkubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f-7292n 1/1 Running 0 65s

查看 Job 日志，即上面的跟踪日志

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl logs kubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f-7292n --tail=3dockerd: /docker/data/overlay2/b212e7a41dd9417f549f234ce768a3fd0fc0e0bafdockerd: /docker/data/image/overlay2/imagedb/content/sha256/1f0ce7a730e7dockerd: /docker/data/image/overlay2/imagedb/metadata/sha256/1f0ce7a730e┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$

静态跟踪 Demo

这里的静态跟踪实际上是帮助文档中讲的 tracepoint 内核跟踪点，需要注意部分机器可能需要添加 --fetch-headers

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl trace run vms105.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }" --fetch-headerstrace 305a7d60-e5a0-11ee-ba88-000c290e5d5f created┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get pods -wNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18dkubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 0/1 Init:0/1 0 4skubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 0/1 PodInitializing 0 15skubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 1/1 Running 0 21s┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$

查看日志信息

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl logs kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprtDefaulted container "kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f" out of: kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f, kubectl-trace-init (init)if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two timesAttaching 1 probe...

在跟踪节点传递结束信息（重新打个页面执行）

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible 192.168.26.105 -m shell -a "pgrep bpftrace" -i host.yaml192.168.26.105 | CHANGED | rc=0 >>37312┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$ansible 192.168.26.105 -m shell -a "kill -2 37312" -i host.yaml192.168.26.105 | CHANGED | rc=0 >>

对应的跟踪日志信息输出

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl logs kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 -fDefaulted container "kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f" out of: kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f, kubectl-trace-init (init)if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two timesAttaching 1 probe...@[bash]: 1@[systemd-udevd]: 1@[calico]: 2@[containerd]: 6@[nsenter]: 15@[kubelet]: 18@[kube-proxy]: 34@[runc:[2:INIT]]: 207@[exe]: 207@[runc]: 208@[containerd-shim]: 220@[calico-node]: 303@[cri-dockerd]: 313@[dockerd]: 3907

pod 状态为完成

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get podsNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18dkubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Completed 0 31m

运行方式通过文件的方式，使用 bpftrace 自带的脚步执行

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl trace run vms105.liruilongs.github.io -f opensnoop.bt --fetch-headerstrace e0b7df1e-e5a1-11ee-898f-000c290e5d5f created┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl get pods | grep e0b7df1e-e5a1-11ee-898f-000c290e5d5fkubectl-trace-e0b7df1e-e5a1-11ee-898f-000c290e5d5f-wv8w2 0/1 PodInitializing 0 12s┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl trace logs e0b7df1e-e5a1-11ee-898f-000c290e5d5fif your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two timesAttaching 6 probes...Tracing open syscalls... Hit Ctrl-C to end.PID COMM FD ERR PATH1038 dockerd 133 0 /docker/data/image/overlay2/imagedb/content/sha256/4873874c08ef1038 dockerd 133 0 /docker/data/image/overlay2/imagedb/content/sha256/4873874c08ef1038 dockerd -1 2 /docker/data/image/overlay2/imagedb/metadata/sha256/4873874c08e1316 kubelet 17 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.stat1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.usage1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.max_u1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.failc1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.limit1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw............................

bpftrace 自带了opensnoop.bt，这个工具可以同时对每个系统调用的开始和结束位置进行跟踪，然后将结果分列输出:

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$cat opensnoop.bt#!/usr/bin/bpftrace/* * opensnoop Trace open() syscalls. * For Linux, uses bpftrace and eBPF. * * Also a basic example of bpftrace. * * USAGE: opensnoop.bt * * This is a bpftrace version of the bcc tool of the same name. * * Copyright 2018 Netflix, Inc. * Licensed under the Apache License, Version 2.0 (the "License") * * 08-Sep-2018 Brendan Gregg Created this. */BEGIN{ printf("Tracing open syscalls... Hit Ctrl-C to end.\n"); printf("%-6s %-16s %4s %3s %s\n", "PID", "COMM", "FD", "ERR", "PATH");}tracepoint:syscalls:sys_enter_open,tracepoint:syscalls:sys_enter_openat{ @filename[tid] = args->filename;}tracepoint:syscalls:sys_exit_open,tracepoint:syscalls:sys_exit_openat/@filename[tid]/{ $ret = args->ret; $fd = $ret >= 0 ? $ret : -1; $errno = $ret >= 0 ? 0 : - $ret; printf("%-6d %-16s %4d %3d %s\n", pid, comm, $fd, $errno, str(@filename[tid])); delete(@filename[tid]);}END{ clear(@filename);}

容器中使用

容器相关的这里不做演示，下面是帮助文档的 Demo

在 Pod 容器上运行内联的 bpftrace 程序：

kubectl trace run pod/nginx -c nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"

在 Pod 容器上使用自定义镜像运行内联的 bpftrace 程序：

kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" --imagename=quay.io/custom-bpftrace-image-name

需要注意的问题

job 闪完成

在编译或构建过程中，表示缺少了必要的头文件,job 启 Pod 之后会马上闪退

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl trace run node/vms103.liruilongs.github.io -f opensnoop.bttrace fc86c785-e513-11ee-8e04-000c290e5d5f created

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl get pods | grep fc86c785-e513-11ee-8e04-000c290e5d5fkubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 ContainerCreating 0 8s┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl get pods kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm -wNAME READY STATUS RESTARTS AGEkubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Completed 0 19skubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Completed 0 20skubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Completed 0 21skubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Terminating 0 26skubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Terminating 0 26s^C┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$kubectl logs kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gmError from server (NotFound): pods "kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm" not found

日志信息

if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two ││ times ││/bpftrace/include/clang_workarounds.h:14:10: fatal error: 'linux/types.h' file not found ││exit status 1

执行环境

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]└─$hostnamectl Static hostname: vms100.liruilongs.github.io Icon name: computer-vm Chassis: vm Machine ID: e93ae3f6cb354f3ba509eeb73568087e Boot ID: 51ad5f1933914654affd2dcf9ebca862 Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 5.4.266-1.el7.elrepo.x86_64 Architecture: x86-64

解决办法：添加 --fetch-headers 尝试 https://github.com/iovisor/kubectl-trace/issues/177

┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl trace run vms105.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }" --fetch-headerstrace 3d9981a0-e59b-11ee-a570-000c290e5d5f created┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get podsNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18dkubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 12s┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get pods -wNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18dkubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 33skubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 41s┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get pods -wNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18dkubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 64s┌──[root@vms100.liruilongs.github.io]-[~/ansible]└─$kubectl get podsNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18dkubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 1/1 Running 0 16m

一直 Pending 状态

这里需要注意，如果是 master 节点做分析，需要把对应的 master 的污点去掉，负责无法调度

┌──[root@vms100.liruilongs.github.io]-[~]└─$kubectl trace run vms102.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }"trace 4f30d986-e595-11ee-a5ef-000c290e5d5f created┌──[root@vms100.liruilongs.github.io]-[~]└─$kubectl get podsNAME READY STATUS RESTARTS AGEhello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d15h ago) 18dkubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f-xw6qm 0/1 Pending 0 7s

┌──[root@vms100.liruilongs.github.io]-[~]└─$kubectl describe pods kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f-xw6qm..................Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 23s default-scheduler 0/6 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.

┌──[root@vms100.liruilongs.github.io]-[~]└─$kubectl trace delete 4f30d986-e595-11ee-a5ef-000c290e5d5ftrace job kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f deletedtrace configuration kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f deleted

在实际生产中可以考虑通过 sidecar 边车的方式运行，监视 pod 中的所有容器。这里不做分享，感兴趣小伙伴可以研究下 ^_

博文部分内容参考

https://github.com/iovisor/kubectl-trace

https://medium.com/@calavera/spy-on-your-kubernetes-cluster-with-bpf-b09032bd1cdc

https://github.com/iovisor/kubectl-trace/issues/177