Docker

起因 # 在上一篇《从进程视角重新理解 Docker》中，我总结了 Docker 最核心的价值：它通过 namespace 和 cgroup 给进程构造了独立的视图和资源限制，并把这套运行环境固化成了可以直接分发的 artifact。

起因 # 第一次接触 Docker 是 2017 年，在深圳网警机房驻场做审核项目。机房完全没有外网，模型、依赖、运行环境全靠 U 盘拷贝 docker image 来传输。后来做模型转换系统，再到量化行业负责机器学习平台，和 Docker、K8s 打了不少交道。但距离上一次系统梳理 namespace、cgroup 这些底层机制，已经过去好几年了。

背景 # 需要在gitlab pipelines中跑一堆测试 其中某些测试需要与rancher交互，在训练集群上执行一个训练任务 gitlab pipelines的默认用户是root，而rancher会默认以当前操作的user进行调度。

本来不知道写什么所以不打算写了，不过后来觉得可以把今年做的一些重大的决定写出来，把当时的分析和想法记录下来。这样若干年后再回看，就能找到：是哪些明智或愚蠢的决定对人生产生了巨大的影响。

继续把 k8s 用于模型转换和部署的自动化流程，然后发现之前安装 k8s 的文档不 work 了…… 时间是 2020 年 5 月 7 日，当前最新的 k8s 版本是 v1.18.2。 报错如下： 详细代码 1 2 3<2kzzqw6rsjid0 --discovery-token-ca-cert-hash sha256:c6c72bdc96c0ff4d59559ff915eee61ba7ac5e8b93c0b2f9e11e813412387ec2 --v=5 4W0507 15:45:12.608784 4768 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. 5I0507 15:45:12.608822 4768 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName 6I0507 15:45:12.608853 4768 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock 7[preflight] Running pre-flight checks 8I0507 15:45:12.608902 4768 preflight.go:90] [preflight] Running general checks 9I0507 15:45:12.608933 4768 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests 10I0507 15:45:12.608966 4768 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf 11I0507 15:45:12.608975 4768 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf 12I0507 15:45:12.608985 4768 checks.go:102] validating the container runtime 13I0507 15:45:12.685381 4768 checks.go:128] validating if the service is enabled and active 14 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ 15I0507 15:45:12.765669 4768 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables 16I0507 15:45:12.765720 4768 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward 17I0507 15:45:12.765752 4768 checks.go:649] validating whether swap is enabled or not 18I0507 15:45:12.765780 4768 checks.go:376] validating the presence of executable conntrack 19I0507 15:45:12.765804 4768 checks.go:376] validating the presence of executable ip 20I0507 15:45:12.765826 4768 checks.go:376] validating the presence of executable iptables 21I0507 15:45:12.765844 4768 checks.go:376] validating the presence of executable mount 22I0507 15:45:12.765864 4768 checks.go:376] validating the presence of executable nsenter 23I0507 15:45:12.765882 4768 checks.go:376] validating the presence of executable ebtables 24I0507 15:45:12.765902 4768 checks.go:376] validating the presence of executable ethtool 25I0507 15:45:12.765920 4768 checks.go:376] validating the presence of executable socat 26I0507 15:45:12.765935 4768 checks.go:376] validating the presence of executable tc 27I0507 15:45:12.765953 4768 checks.go:376] validating the presence of executable touch 28I0507 15:45:12.765973 4768 checks.go:520] running all checks 29I0507 15:45:12.844881 4768 checks.go:406] checking whether the given node name is reachable using net.LookupHost 30I0507 15:45:12.845030 4768 checks.go:618] validating kubelet version 31I0507 15:45:12.888056 4768 checks.go:128] validating if the service is enabled and active 32I0507 15:45:12.893254 4768 checks.go:201] validating availability of port 10250 33I0507 15:45:12.893373 4768 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt 34I0507 15:45:12.893388 4768 checks.go:432] validating if the connectivity type is via proxy or direct 35I0507 15:45:12.893414 4768 join.go:441] [preflight] Discovering cluster-info 36I0507 15:45:12.893440 4768 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "172.20.52.117:6443" 37I0507 15:45:13.033539 4768 token.go:116] [discovery] Requesting info from "172.20.52.117:6443" again to validate TLS against the pinned public key 38I0507 15:45:13.172634 4768 token.go:133] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.20.52.117:6443" 39I0507 15:45:13.172653 4768 discovery.go:51] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process 40I0507 15:45:13.172660 4768 join.go:455] [preflight] Fetching init configuration 41I0507 15:45:13.172669 4768 join.go:493] [preflight] Retrieving KubeConfig objects 42[preflight] Reading configuration from the cluster... 43[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' 44I0507 15:45:13.541858 4768 interface.go:400] Looking for default routes with IPv4 addresses 45I0507 15:45:13.541871 4768 interface.go:405] Default route transits interface "eth0" 46I0507 15:45:13.541941 4768 interface.go:208] Interface eth0 is up 47I0507 15:45:13.541978 4768 interface.go:256] Interface "eth0" has 2 addresses :[10.198.21.97/22 fe80::f816:3eff:fe5e:88f1/64]. 48I0507 15:45:13.541998 4768 interface.go:223] Checking addr 10.198.21.97/22. 49I0507 15:45:13.542008 4768 interface.go:230] IP found 10.198.21.97 50I0507 15:45:13.542016 4768 interface.go:262] Found valid IPv4 address 10.198.21.97 for interface "eth0". 51I0507 15:45:13.542023 4768 interface.go:411] Found active IP 10.198.21.97 52I0507 15:45:13.542057 4768 preflight.go:101] [preflight] Running configuration dependant checks 53I0507 15:45:13.542072 4768 controlplaneprepare.go:211] [download-certs] Skipping certs download 54I0507 15:45:13.542080 4768 kubelet.go:111] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf 55I0507 15:45:13.542775 4768 kubelet.go:119] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt 56I0507 15:45:13.543283 4768 kubelet.go:145] [kubelet-start] Checking for an existing Node in the cluster with name "host-10-198-21-97" and status "Ready" 57nodes "host-10-198-21-97" is forbidden: User "system:bootstrap:0752yx" cannot get resource "nodes" in API group "" at the cluster scope 58cannot get Node "host-10-198-21-97" 59k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletStartJoinPhase 60 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/kubelet.go:148 61k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1 62 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234 63k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll 64 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422 65k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run 66 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207 67k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1 68 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170 69k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute 70 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826 71k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC 72 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 73k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute 74 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864 75k8s.io/kubernetes/cmd/kubeadm/app.Run 76 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50 77main.main 78 _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25 79runtime.main 80 /usr/local/go/src/runtime/proc.go:203 81runtime.goexit 82 /usr/local/go/src/runtime/asm_amd64.s:1357 83error execution phase kubelet-start 84k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1 85 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235 86k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll 87 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422 88k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run 89 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207 90k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1 91 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170 92k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute 93 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826 94k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC 95 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 96k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute 97 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864 98k8s.io/kubernetes/cmd/kubeadm/app.Run 99 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50 100main.main 101 _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25 102runtime.main 103 /usr/local/go/src/runtime/proc.go:203 104runtime.goexit 105 /usr/local/go/src/runtime/asm_amd64.s:1357 看起来关键的报错在这一句

年中的时候接了离职的同事模型转换的锅，在不断地更新迭代的过程中，发现了一些痛点。 发现k8s能够解决一部分痛点，因此来分享一下在这方面的探索。 什么是模型转换 # 简单来说，深度学习模型的流程分为training和inference两部分。训练时用的一般是pytorch等框架，这些框架训练出的model是没办法直接部署在各个硬件平台上做inference的。因此需要将使用训练框架得到的模型，转换为能够部署到各个硬件平台上的模型。这个过程就是模型转换。

TL;DR * 依靠人的小心谨慎是不靠谱的，人总有失误的时候 * 看了下docker volume的权限机制，貌似是从docker image中继承。 * 写了两个脚本，用来把rm alias到mv，避免手滑 又是一个可以摸鱼的周五晚上，sensespider系统测试了一天，fix了几个Bug，似乎可以发布了。系统一直是部署在了docker中..这几天测试产生了不少结果文件在host的volume中… 看着不舒服，干脆删一下好了

起因： # 公司部署在 hk 的爬虫服务器突然挂掉了。后来发现只是在深圳办公区无法访问。排查后发现原因是 docker 的网络（包括 docker network 的 subnet 或者是某个容器的 ip）与该 host 在内网的 ip 段相同，导致冲突。

现象： # 使用docker compose 挂载 named volume 无效（且没有错误提示) 排查过程： # 一开始是没有使用docker-compose命令，直接使用docker run -v 命令，挂载两个绝对路径，没有问题。