↓ 跳过正文
  1. Posts/

【施工中】gitlab ci docker executor指定用户执行

·966 字·2 分钟

背景
#

需要在gitlab pipelines中跑一堆测试 其中某些测试需要与rancher交互,在训练集群上执行一个训练任务

gitlab pipelines的默认用户是root,而rancher会默认以当前操作的user进行调度。

这里还涉及若干奇奇怪怪的,比较特异性的原因。 导致gitlab pipeline无法调度成功rancher.

总之,需求是,要在gitlab上以一个non-root 执行一些任务

由于缺少官方支持 (参考这里 ), 最终用了非常tricky的办法解决,踩了非常多的坑,因此记录下来,以飨后人

过程
#

为了方便表述,假设这个non-root user就叫"renkz"

在docker中添加renkz为默认user
#

如果我想以renkz作为user来运行docker,那么docker中肯定要有这个用户。

最初我是这样在dockerfile设置的。

1RUN useradd -ms /bin/bash renkz -d /home/renkz  -G sudo
2USER renkz
3WORKDIR /home/renkz

然后以如下方式启动:

1docker run -it -v /dssg:/dssg -u renkz  image

其中挂载的/dssg是数据的存储。然而由于/dssg的权限问题,我需要在我的镜像中也存在这些能访问/dssg的用户。尤其是renkz这个用户。(不要问我为什么这么奇怪。公司现状就是如此。。。) 询问了公司的IT的同事,他们的做法是给所有需要挂载/dssg的容器同时挂载了全局的/etc/group和全局的/etc/passwd

于是启动方式变成了这样:

1docker run -it -v /dssg:/dssg -v /global-group:/etc/group -v /global-passwd:/etc/passwd -u renkz  image

其中/global-passwd和/global-group为两个公司全局的passwd和group文件路径

然而,奇怪的发现,启动docker后,“whoami"得到的不是renkz,而是"qwe”。。对应着/etc/passwd里user id为1000的用户名。

排查后认为,是因为 docker –user选项,会把user name先转换成user id。 由于我在docker中创建user renkz的时候是使用的默认user id(1000),因此docker run的时候其实是以user id 1000启动的。在这个过程发生后,/etc/passwd被替换,此时user id为1000的user变成了qwe

指定gitlab runner以renkz为user启动
#

参考Change Gitlab CI Runner user

以及gitlab runner advanced config

相关文章

k8s nodes is forbidden user cannot list resource nodes in api group at the cluster scope

·998 字·2 分钟
继续把 k8s 用于模型转换和部署的自动化流程,然后发现之前安装 k8s 的文档不 work 了…… 时间是 2020 年 5 月 7 日,当前最新的 k8s 版本是 v1.18.2。 报错如下: 详细代码 1 2 3<2kzzqw6rsjid0 --discovery-token-ca-cert-hash sha256:c6c72bdc96c0ff4d59559ff915eee61ba7ac5e8b93c0b2f9e11e813412387ec2 --v=5 4W0507 15:45:12.608784 4768 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. 5I0507 15:45:12.608822 4768 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName 6I0507 15:45:12.608853 4768 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock 7[preflight] Running pre-flight checks 8I0507 15:45:12.608902 4768 preflight.go:90] [preflight] Running general checks 9I0507 15:45:12.608933 4768 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests 10I0507 15:45:12.608966 4768 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf 11I0507 15:45:12.608975 4768 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf 12I0507 15:45:12.608985 4768 checks.go:102] validating the container runtime 13I0507 15:45:12.685381 4768 checks.go:128] validating if the service is enabled and active 14 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ 15I0507 15:45:12.765669 4768 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables 16I0507 15:45:12.765720 4768 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward 17I0507 15:45:12.765752 4768 checks.go:649] validating whether swap is enabled or not 18I0507 15:45:12.765780 4768 checks.go:376] validating the presence of executable conntrack 19I0507 15:45:12.765804 4768 checks.go:376] validating the presence of executable ip 20I0507 15:45:12.765826 4768 checks.go:376] validating the presence of executable iptables 21I0507 15:45:12.765844 4768 checks.go:376] validating the presence of executable mount 22I0507 15:45:12.765864 4768 checks.go:376] validating the presence of executable nsenter 23I0507 15:45:12.765882 4768 checks.go:376] validating the presence of executable ebtables 24I0507 15:45:12.765902 4768 checks.go:376] validating the presence of executable ethtool 25I0507 15:45:12.765920 4768 checks.go:376] validating the presence of executable socat 26I0507 15:45:12.765935 4768 checks.go:376] validating the presence of executable tc 27I0507 15:45:12.765953 4768 checks.go:376] validating the presence of executable touch 28I0507 15:45:12.765973 4768 checks.go:520] running all checks 29I0507 15:45:12.844881 4768 checks.go:406] checking whether the given node name is reachable using net.LookupHost 30I0507 15:45:12.845030 4768 checks.go:618] validating kubelet version 31I0507 15:45:12.888056 4768 checks.go:128] validating if the service is enabled and active 32I0507 15:45:12.893254 4768 checks.go:201] validating availability of port 10250 33I0507 15:45:12.893373 4768 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt 34I0507 15:45:12.893388 4768 checks.go:432] validating if the connectivity type is via proxy or direct 35I0507 15:45:12.893414 4768 join.go:441] [preflight] Discovering cluster-info 36I0507 15:45:12.893440 4768 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "172.20.52.117:6443" 37I0507 15:45:13.033539 4768 token.go:116] [discovery] Requesting info from "172.20.52.117:6443" again to validate TLS against the pinned public key 38I0507 15:45:13.172634 4768 token.go:133] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.20.52.117:6443" 39I0507 15:45:13.172653 4768 discovery.go:51] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process 40I0507 15:45:13.172660 4768 join.go:455] [preflight] Fetching init configuration 41I0507 15:45:13.172669 4768 join.go:493] [preflight] Retrieving KubeConfig objects 42[preflight] Reading configuration from the cluster... 43[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' 44I0507 15:45:13.541858 4768 interface.go:400] Looking for default routes with IPv4 addresses 45I0507 15:45:13.541871 4768 interface.go:405] Default route transits interface "eth0" 46I0507 15:45:13.541941 4768 interface.go:208] Interface eth0 is up 47I0507 15:45:13.541978 4768 interface.go:256] Interface "eth0" has 2 addresses :[10.198.21.97/22 fe80::f816:3eff:fe5e:88f1/64]. 48I0507 15:45:13.541998 4768 interface.go:223] Checking addr 10.198.21.97/22. 49I0507 15:45:13.542008 4768 interface.go:230] IP found 10.198.21.97 50I0507 15:45:13.542016 4768 interface.go:262] Found valid IPv4 address 10.198.21.97 for interface "eth0". 51I0507 15:45:13.542023 4768 interface.go:411] Found active IP 10.198.21.97 52I0507 15:45:13.542057 4768 preflight.go:101] [preflight] Running configuration dependant checks 53I0507 15:45:13.542072 4768 controlplaneprepare.go:211] [download-certs] Skipping certs download 54I0507 15:45:13.542080 4768 kubelet.go:111] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf 55I0507 15:45:13.542775 4768 kubelet.go:119] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt 56I0507 15:45:13.543283 4768 kubelet.go:145] [kubelet-start] Checking for an existing Node in the cluster with name "host-10-198-21-97" and status "Ready" 57nodes "host-10-198-21-97" is forbidden: User "system:bootstrap:0752yx" cannot get resource "nodes" in API group "" at the cluster scope 58cannot get Node "host-10-198-21-97" 59k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletStartJoinPhase 60 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/kubelet.go:148 61k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1 62 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234 63k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll 64 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422 65k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run 66 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207 67k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1 68 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170 69k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute 70 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826 71k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC 72 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 73k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute 74 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864 75k8s.io/kubernetes/cmd/kubeadm/app.Run 76 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50 77main.main 78 _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25 79runtime.main 80 /usr/local/go/src/runtime/proc.go:203 81runtime.goexit 82 /usr/local/go/src/runtime/asm_amd64.s:1357 83error execution phase kubelet-start 84k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1 85 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235 86k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll 87 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422 88k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run 89 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207 90k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1 91 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170 92k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute 93 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826 94k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC 95 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 96k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute 97 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864 98k8s.io/kubernetes/cmd/kubeadm/app.Run 99 /workspace/anago-v1.18.2-beta.0.14+a78cd082e8c913/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50 100main.main 101 _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25 102runtime.main 103 /usr/local/go/src/runtime/proc.go:203 104runtime.goexit 105 /usr/local/go/src/runtime/asm_amd64.s:1357 看起来关键的报错在这一句

Kubernetes(k8s)在深度学习模型转换方面的探索

·1886 字·4 分钟
年中的时候接了离职的同事模型转换的锅,在不断地更新迭代的过程中,发现了一些痛点。 发现k8s能够解决一部分痛点,因此来分享一下在这方面的探索。 什么是模型转换 # 简单来说,深度学习模型的流程分为training和inference两部分。训练时用的一般是pytorch等框架,这些框架训练出的model是没办法直接部署在各个硬件平台上做inference的。因此需要将使用训练框架得到的模型,转换为能够部署到各个硬件平台上的模型。这个过程就是模型转换。

docker network 与 本地 network 网段冲突

·537 字·2 分钟
起因: # 公司部署在 hk 的爬虫服务器突然挂掉了。后来发现只是在深圳办公区无法访问。排查后发现原因是 docker 的网络(包括 docker network 的 subnet 或者是某个容器的 ip)与该 host 在内网的 ip 段相同,导致冲突。