Validating Admission Controller

Kubernetes/Kubernetes 이론

Validating Admission Controller

이쿠우우 2020. 11. 5. 19:34

Validating Admission Controller 생성해보기

[Admission Controller 관련 글목록]

Kubernetes Version 1.18 에 해당함.

Kubernetes Admission Controller
Mutating Admission Controller 생성해보기
Validating Admission Controller

예제로

iksoon-ns namespace에 Pod가 생성 요청이 오면

labels가

validate: allow

인 경우만 pod가 생성되는

validating Admission Controller를 생성해보겠음.

0. ValidatingAdmissionWebhook 활성화

kubernetes에서는ValidatingAdmissionWebhook은

default로 이미 활성화과 되어있는 상태라

default 환경이라면 따로 설정하지 않고 진행해도 됨.

[활성화된 admission controller 조회 명령어]

kube-apiserver -h | grep enable-admission-plugins

만약 목록에 ValidatingAdmissionWebhook항목이 없다면

kube-apiserver container에 들어가서 아래의 명령을 실행

[명령어]

kube-apiserver --enable-admission-plugins=ValidatingAdmissionWebhook

1. iksoon-ns namespace생성

iksoon-ns namespace에 pod가 생성됐을 때를 고려해야하니

먼저 namespace를 미리 생성해놓음

[명령어]

kubectl create ns iksoon-ns

[생성 확인]

kubectl get ns

항목 중 iksoon-ns가 있는지 확인

2. Webhook Server Container 생성

2.1. Python Flask를 사용해서 Webhook Server를 구현.

[iksoon-webhook-validating.py 예제]

코드 최적화는 생략함. 그저 참고용으로 만들어봄...

from flask import Flask, request, jsonify
import pprint
import requests
import logging
import copy
import base64
import jsonpatch

from requests.auth import HTTPBasicAuth

logging.basicConfig(filename = "admission_controller.log", level = logging.DEBUG)

app = Flask(__name__)

@app.route('/iksoon/validate', methods=['POST'])
def validate():

message_text = ""
admissionReview = ""
label_validating = ""

        # kubernetes API Server의 Request를 해당 webhook으로 가져옴
kube_api_request= request.json
logging.info("\n")
        logging.info('--Kube-apiserver request---')
logging.info(kube_api_request)
        logging.info("\n")

        #namespace data를 알아옴
request_namespace = kube_api_request["request"]["namespace"]

        #namesapce 가 "iksoon-ns"과 동일한지 확인
        logging.info("----namespace name check result----")
logging.info("namespace = %s", request_namespace)

        #namesapce 가 "iksoon-ns"가 아니라면 Pod생성 안되도록 함.
if "iksoon-ns" != request_namespace:
message_text = "namespace is not iksoon-ns (Don't create Pod)"
response = admission_response(False, message_text)
           logging.info(response)
return jsonify(response)
else:
message_text = "namespace is iksoon-ns (validating start)"
logging.info(message_text)

# Pod labels 정보 확인
        # labels ( validate: allow )
        # 먼저 pod 생성 yaml에 labels 항목이 있는지 확인
        # (labels 항목이 없이 생성될 수도 있음)
if 'labels' in kube_api_request["request"]["object"]["metadata"]:
           message_text = "labels is exsist"
           logging.info(message_text)
else:
message_text = "labels is null (Don't create Pod)"
           response = admission_response(False, message_text)
           logging.info(response)
return jsonify(response)

# labels 항목 중에 validate가 있는지 확인
        if 'validate' in kube_api_request["request"]["object"]["metadata"]["labels"]:
           message_text = "validate labels is exsist"
           logging.info(message_text)
label_validating = kube_api_request["request"]["object"]["metadata"]["labels"]["validate"]
           logging.info("label_validating = %s", label_validating)
        else:
           message_text = "validate labels is null (Don't create Pod)"
           response = admission_response(False, message_text)
           logging.info(response)
return jsonify(response)

        # validate의 value가 allow인지 확인
        if 'allow' != label_validating:
           message_text = "validate labels value is not allow (Don't create Pod)"
           response = admission_response(False, message_text)
           logging.info(response)
return jsonify(response)

message_text = "validate labels value is allow (create Pod)"
response = admission_response(True, message_text)
logging.info(response)
return jsonify(response)

def admission_response(result, message_text):
admissionReview = {
              "response": {
                 "allowed": result,
"status": {"message": message_text}
              }
}
return admissionReview

if __name__ == '__main__':
app.run(host= '0.0.0.0', port=6000, debug=True, ssl_context=('/run/secrets/tls/tls.crt', '/run/secrets/tls/tls.key'))

[response 참고]

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

log 확인을 위해 logging을 사용함.

ssl_context=('/run/secrets/tls/tls.crt', '/run/secrets/tls/tls.key') 은

해당 flask webhook에 HTTPS로 접근하기 위한 인증서을 설정하는 부분으로

아래에서 인증서를 생성하고

deployment 시 해당 경로에 volume 형태로 mount할 것임.

[python flask test 쿼리]

curl -d '{"request": {"uid":"de47", "namespace":"iksoon-ns", "object": {"spec":{"containers": [{"image": "peksoon/iksoon_tomcat:1.0.6"}] }, "metadata": { "labels": {"test": "data1", "validate": "allow"}} }}}' -H "Content-Type: application/json" -X POST http://10.0.2.5:6000/iksoon/validate

2.2. 상위 Webhook Server가 실행되는 Container image 생성

DockerFile을 사용해서 생성

[Dockerfile 생성]

원활한 디버깅을 위해 필요한 tool을 전부 설치한 이미지를 생성

FROM centos:7

RUN yum install -y net-tools
RUN yum install -y gcc openssl-devel bzip2-devel libffi-devel wget
RUN yum install -y make
RUN yum update
RUN wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz
RUN tar xvf Python-3.8.5.tgz
WORKDIR Python-3.8.5/
RUN ./configure --enable-optimizations
RUN make altinstall
RUN unlink /bin/python
RUN ln -s /usr/local/bin/python3.8 /bin/python

RUN pip3.8 install flask
RUN pip3.8 install jsonpatch
RUN pip3.8 install requests

RUN mkdir /admission_test
WORKDIR /admission_test
ADD ./iksoon-webhook-validating.py .

CMD ["python", "/admission_test/iksoon-webhook-validating.py"]

[Dockerfile 빌드 명령어]

docker build --tag mutatingadmission:1.0.0 .

[Docker hub에 push 해놓음]

docker tag [생성된 image id] peksoon/admission_test:1.0.0

docker login

docker push peksoon/admission_test:1.0.0

3. Flask Webhook HTTPS 접속용 인증서 생성

[명령어]

webhook-server-tls.crt 생성 시 CN을 iksoon-admission-service.default.svc로 설정했는데

이는 아래에서 생성할 webhook deployment 접속 service object의 domain 을 넣어줘야함.

openssl req -nodes -new -x509 -keyout ca.key -out ca.crt -subj "/CN=iksoon admission controller"

openssl genrsa -out webhook-server-tls.key 2048

openssl req -new -key webhook-server-tls.key -subj "/CN=iksoon-admission-service.default.svc" \
| openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -out webhook-server-tls.crt

[결과]

4. TLS Secret 생성

Secret 을 생성해서 deployment에서 해당 secret 를

volume 행태로 mount해서

flask webhook container에 인증서를 전달할 것임.

[명령어]

kubectl -n default create secret tls webhook-certs \

--cert "webhook-server-tls.crt" \

--key "webhook-server-tls.key"

[결과]

5. Webhook Server deployment 와 Service 생성

[Deployment, Servcie yaml 예시]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iksoon-admission-webhook
  namespace: default
  labels:
    app: mutating-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admission-pod
  template:
    metadata:
      labels:
        app: admission-pod
    spec:
      containers:
      - name: webhook
        image: peksoon/admission_test:1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 6000
name: flask-webhook
        volumeMounts:
        - name: webhook-tls
          mountPath: /run/secrets/tls
          readOnly: true
volumes:
      - name: webhook-tls
        secret:
          secretName: webhook-certs

---

apiVersion: v1
kind: Service
metadata:
  name: iksoon-admission-service
  namespace: default
spec:
  ports:
    - port: 443
      targetPort: flask-webhook
  selector:
    app: admission-pod

6. ca.crt 파일 base64 인코딩

[명령어]

openssl base64 -A <"ca.crt"

결과 복사 해놓음

7. Webhook Server deployment 와 Service 생성

MutatingWebhookConfiguration name 은

상위 webhook deployment와 연결되어있는 service의 domain이여야함

service name = iksoon-admission-service

namespace name = default

임으로 생성되는 domain = iksoon-admission-service.default.svc

clientConfig의 service name과 namespace는

상위 webhook deployment와 연결되어있는 service의

service name = iksoon-admission-service

namespace name = default

을 입력함.

path: "/iksoontest"

에서 경로를 "/iksoontest" 로 설정한 이유는

상위 webhook python 코드의

@app.route('/iksoontest', methods=['POST'])

부분에서 경로를 "/iksoontest"로 설정했기 때문.

[validating_admission_controller.yaml]

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating-admission-webhook
webhooks:
  - name: iksoon-admission-service.default.svc
    clientConfig:
      service:
        name: iksoon-admission-service
        namespace: default
  path: "/iksoon/validate"
caBundle: [여기에 상위에서 복사한 base64 encoding ca.crt 값이 들어감.]
    rules:  # Pod를 생성하는 경우에만 해당 mutating admission이 적용됨. 예) deployment로 생성하면 안됨.
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]

8. 결과 확인 pod 생성 test

[test.yaml]

apiVersion: v1
kind: Pod
metadata:
  name: iksoon-pod
  namespace: default
  labels:
    app: test
spec:
  containers:
  - name: iksoon-tomcat
    image: peksoon/iksoon_tomcat:1.0.6
    ports:
    - containerPort: 8080

[Pod생성 결과 확인]

iksoon-ns namespace에 Pod를 배포했는데

labels 에 validate: allow 가 없어서 배포가 안됨

[test.yaml]

apiVersion: v1
kind: Pod
metadata:
  name: iksoon-pod
  namespace: default
  labels:
    app: test
    validate: allow
spec:
  containers:
  - name: iksoon-tomcat
    image: peksoon/iksoon_tomcat:1.0.6
    ports:
    - containerPort: 8080

[Pod생성 결과 확인]

labels 에 validate: allow를 추가하면 정상적으로 배포됨

참고용 Mutating, Validating admission controller webhook 소스코드

from flask import Flask, request, jsonify

import pprint

import requests

import logging

import copy

import base64

import jsonpatch

from requests.auth import HTTPBasicAuth

logging.basicConfig(filename = "admission_controller.log", level = logging.DEBUG)

app = Flask(__name__)

@app.route('/iksoon/mutate', methods=['POST'])

def mutate():

message_text = ""

patch = ""

admissionReview = ""

container_image= ""

mutating_result = []

# kubernetes API Server의 Request를 해당 webhook으로 가져옴

kube_api_request= request.json

logging.info("\n")

logging.info('--Kube-apiserver request---')

logging.info(kube_api_request)

logging.info("\n")

#namespace data를 알아옴

request_namespace = kube_api_request["request"]["namespace"]

#namesapce 가 "iksoon-ns"과 동일한지 확인

logging.info("----namespace name check result----")

logging.info("namespace = %s", request_namespace)

#namesapce 가 "iksoon-ns"가 아니라면 mutating 하지 않고 return

if "iksoon-ns" != request_namespace:

message_text = "namespace is not iksoon-ns (Don't Mutating)"

admissionReview = admission_response(True, message_text)

logging.info(admissionReview)

return jsonify(admissionReview)

else:

message_text = "namespace is iksoon-ns (Mutating start)"

logging.info(message_text)

# 첫번째 container image정보를 확인

container_image = kube_api_request["request"]["object"]["spec"]["containers"][0]["image"]

logging.info("container image = %s", container_image)

# 첫번째 container image가 "peksoon/iksoon_tomcat:1.0.6"과 동일한지 확인

if "peksoon/iksoon_tomcat:1.0.6" == container_image:

container_mutating={'op': 'replace', 'path': '/spec/containers/0/image', 'value': 'peksoon/iksoon_mysql:1.0.2'}

mutating_result.append(container_mutating)

# labels 추가 ( mutating: admission-test )

# 먼저 pod 생성 yaml에 labels 항목이 있는지 확인

# (labels 항목이 없이 생성될 수도 있음)

if 'labels' in kube_api_request["request"]["object"]["metadata"]:

message_text = "labels is exsist"

logging.info(message_text)

# labels 항목 중에 mutate가 있는지 확인

# mutate key가 있다면 value를 무조건 admission-test로 변경

if 'mutate' in kube_api_request["request"]["object"]["metadata"]["labels"]:

message_text = "mutate labels is exsist (labels mutate)"

logging.info(message_text)

container_mutating={'op': 'replace', 'path': '/metadata/labels/mutate', 'value': 'admission-test'}

# labels 항목 중에 mutate가 없다면 labels 추가

else:

message_text = "mutate labels is null (labels add)"

logging.info(message_text)

labelAdd_mutating={'op': 'add', 'path': '/metadata/labels/mutate', 'value': 'admission-test'}

mutating_result.append(labelAdd_mutating)

logging.info(labelAdd_mutating)

else:

message_text = "labels is null (Don't Mutating)"

logging.info(message_text)

if 0 != len(mutating_result):

message_text = "Data mutating"

logging.info(mutating_result)

patch = jsonpatch.JsonPatch(mutating_result)

#jsonpath를 base64 인코딩함.

base64_patch = base64.b64encode(patch.to_string().encode("utf-8")).decode("utf-8")

uid = kube_api_request["request"]["uid"]

admissionReview = admission_response_mutating(True, message_text, uid, base64_patch)

else:

message_text = "nothing to change"

admissionReview = admission_response(True, message_text)

# Webhook Server에서 Kubernetes API Server로 response

logging.info("\n")

logging.info('--Kube-apiserver response result---')

logging.info(admissionReview)

logging.info("\n")

return jsonify(admissionReview)

@app.route('/iksoon/validate', methods=['POST'])

def validate():

message_text = ""

admissionReview = ""

label_validating = ""

# kubernetes API Server의 Request를 해당 webhook으로 가져옴

kube_api_request= request.json

logging.info("\n")

logging.info('--Kube-apiserver request---')

logging.info(kube_api_request)

logging.info("\n")

#namespace data를 알아옴

request_namespace = kube_api_request["request"]["namespace"]

#namesapce 가 "iksoon-ns"과 동일한지 확인

logging.info("----namespace name check result----")

logging.info("namespace = %s", request_namespace)

#namesapce 가 "iksoon-ns"가 아니라면 Pod생성 안되도록 함.

if "iksoon-ns" != request_namespace:

message_text = "namespace is not iksoon-ns (Don't create Pod)"

response = admission_response(False, message_text)

logging.info(response)

return jsonify(response)

else:

message_text = "namespace is iksoon-ns (validating start)"

logging.info(message_text)

# Pod labels 정보 확인

# labels ( validate: allow )

# 먼저 pod 생성 yaml에 labels 항목이 있는지 확인

# (labels 항목이 없이 생성될 수도 있음)

if 'labels' in kube_api_request["request"]["object"]["metadata"]:

message_text = "labels is exsist"

logging.info(message_text)

else:

message_text = "labels is null (Don't create Pod)"

response = admission_response(False, message_text)

logging.info(response)

return jsonify(response)

# labels 항목 중에 validate가 있는지 확인

if 'validate' in kube_api_request["request"]["object"]["metadata"]["labels"]:

message_text = "validate labels is exsist"

logging.info(message_text)

label_validating = kube_api_request["request"]["object"]["metadata"]["labels"]["validate"]

logging.info("label_validating = %s", label_validating)

else:

message_text = "validate labels is null (Don't create Pod)"

response = admission_response(False, message_text)

logging.info(response)

return jsonify(response)

# validate의 value가 allow인지 확인

if 'allow' != label_validating:

message_text = "validate labels value is not allow (Don't create Pod)"

response = admission_response(False, message_text)

logging.info(response)

return jsonify(response)

message_text = "validate labels value is allow (create Pod)"

response = admission_response(True, message_text)

logging.info(response)

return jsonify(response)

def admission_response(result, message_text):

admissionReview = {

"response": {

"allowed": result,

"status": {"message": message_text}

}

return admissionReview

def admission_response_mutating(result, message_text, uid_data, path_data):

#Kube-apiserver로 보내는 response 필드로 아래와 같이 data 포맷이 정해져있어서

#해당 규칙대로 return 해줘야함

#return 값은 patch 항목도 base64로 설정되는것에 주의함.

#allowed 항목을 보면

#원래 요청 거절을 validate에서 하지만 해당 값이 false면 mutate 단계에서도 요청을 거절할 수 있음

#거절할 경우 deny를 추가해서 거절에 대한 이유를 같이 return 해주는것이 좋음

admissionReview = {

"response": {

"allowed": True,

"uid": uid_data,

"patch": path_data,

"patchtype": "JSONPatch",

"status": {"message": message_text}

}

return admissionReview

if __name__ == '__main__':

app.run(host= '0.0.0.0', port=6000, debug=True, ssl_context=('/run/secrets/tls/tls.crt', '/run/secrets/tls/tls.key'))

제 글을 복사할 시 출처를 명시해주세요.
글에 오타, 오류가 있다면 댓글로 알려주세요! 바로 수정하겠습니다!