96. kubeadm 설치 에러 : kubelet error

 

 

 

failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\"

 

상황

kubeadm을 사용해서 kubernetes cluster 구성 시 error 발생함

 

[사용한 명령어]

kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.0.2.15

 

 

ERROR MSG

 

[kubeadm Err msg]

Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get    "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

 

[systemctl status kubelet]

● kubelet.service - kubelet: The Kubernetes Node Agent

   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)

  Drop-In: /usr/lib/systemd/system/kubelet.service.d

           └─10-kubeadm.conf

   Active: activating (auto-restart) (Result: exit-code) since Thu 2022-02-10 21:18:56 EST; 6s ago

     Docs: https://kubernetes.io/docs/

  Process: 14649 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)

Main PID: 14649 (code=exited, status=1/FAILURE)

 

 

[journalctl -qu kubelet -n 100]

Feb 10 21:19:58 kube.master1.node kubelet[15048]: I0210 21:19:58.013605   15048 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"

Feb 10 21:19:58 kube.master1.node kubelet[15048]: I0210 21:19:58.015544   15048 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"

Feb 10 21:19:58 kube.master1.node kubelet[15048]: I0210 21:19:58.015585   15048 docker_service.go:258] "Docker cri networking managed by the network plugin" networkPluginName="cni"

Feb 10 21:19:58 kube.master1.node kubelet[15048]: I0210 21:19:58.015622   15048 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"

Feb 10 21:19:58 kube.master1.node kubelet[15048]: I0210 21:19:58.019963   15048 docker_service.go:264] "Docker Info" dockerInfo=&{ID:2R3V:YYTR:E564:UJ7A:KTPI:RRNQ:TIIS:EIGZ:7CLK:ZXJH:QHCG:U5B5 Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:7 Driver:overlay2 D

Feb 10 21:19:58 kube.master1.node kubelet[15048]: E0210 21:19:58.020005   15048 server.go:302] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""

 

 

 

원인

설치 시 container runtime을 docker로 사용했는데

kubelet과 docker의 cgroup이 일치하지 않아서 발생하는 오류임.

 

kubelet과 Container runtime에는 "cgroup driver"라는 속성이 있음.

linux 시스템에서 cgroup을 관리하는데 중요한 설정임.

cgroup 종류는 cgroupfs, systemd 2개가 있음.

 

kubernetes cluster 구성에 kubelet이 정상동작해야하는데

이때 kubelet 과 container runtime의 cgroup 드라이버가 일치되어 있어야함.

 

kubelet의 cgroup을 "cgroupfs"로 변경하는 방법도 있지만

kubernetes 공식문서에서는 "cgroupfs"보다 "systemd"를 권장한다고 함.

 

Docker의 경우 cgroup이 default로 cgroupfs 로 설정되어 있고

kubelet의 경우 cgroup이 default로 systemd로 설정되어 있음.

이를 일치 시켜주기 위해 

설치 시 container runtime을 docker를 "cgroupfs"에서 "systemd"로 변경해줘야함.

 

[docker cgroup 확인 방법]

docker info|grep "Cgroup Driver"

 

[kubelet cgroup 확인 방법]

kubectl describe cm  kubelet-config -n kube-system  | grep cgroupDriver

 

 

 

해결책

 

sudo mkdir /etc/docker

cat <<EOF | sudo tee /etc/docker/daemon.json

{

  "exec-opts": ["native.cgroupdriver=systemd"],

  "log-driver": "json-file",

  "log-opts": {

    "max-size": "100m"

  },

  "storage-driver": "overlay2"

}

EOF

 

systemctl daemon-reload

systemctl restart docker

systemctl restart kubelet

 

[변경 확인]

docker info|grep "Cgroup Driver"

 

kubeadm reset

 

[재설치]

kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.0.2.10