Validating Admission Controller 생성해보기
[Admission Controller 관련 글목록]
Kubernetes Version 1.18 에 해당함.
Kubernetes Admission Controller
Mutating Admission Controller 생성해보기
Validating Admission Controller
예제로
iksoon-ns namespace에 Pod가 생성 요청이 오면
labels가
validate: allow
인 경우만 pod가 생성되는
validating Admission Controller를 생성해보겠음.
0. ValidatingAdmissionWebhook 활성화
kubernetes에서는ValidatingAdmissionWebhook은
default로 이미 활성화과 되어있는 상태라
default 환경이라면 따로 설정하지 않고 진행해도 됨.
[활성화된 admission controller 조회 명령어]
kube-apiserver -h | grep enable-admission-plugins
만약 목록에 ValidatingAdmissionWebhook항목이 없다면
kube-apiserver container에 들어가서 아래의 명령을 실행
[명령어]
kube-apiserver --enable-admission-plugins=ValidatingAdmissionWebhook
1. iksoon-ns namespace생성
iksoon-ns namespace에 pod가 생성됐을 때를 고려해야하니
먼저 namespace를 미리 생성해놓음
[명령어]
kubectl create ns iksoon-ns
[생성 확인]
kubectl get ns
항목 중 iksoon-ns가 있는지 확인
2. Webhook Server Container 생성
2.1. Python Flask를 사용해서 Webhook Server를 구현.
[iksoon-webhook-validating.py 예제]
코드 최적화는 생략함. 그저 참고용으로 만들어봄...
| from flask import Flask, request, jsonify import pprint import requests import logging import copy import base64 import jsonpatch from requests.auth import HTTPBasicAuth logging.basicConfig(filename = "admission_controller.log", level = logging.DEBUG) app = Flask(__name__) @app.route('/iksoon/validate', methods=['POST']) def validate(): message_text = "" admissionReview = "" label_validating = "" # kubernetes API Server의 Request를 해당 webhook으로 가져옴 kube_api_request= request.json logging.info("\n") logging.info('--Kube-apiserver request---') logging.info(kube_api_request) logging.info("\n") #namespace data를 알아옴 request_namespace = kube_api_request["request"]["namespace"] #namesapce 가 "iksoon-ns"과 동일한지 확인 logging.info("----namespace name check result----") logging.info("namespace = %s", request_namespace) #namesapce 가 "iksoon-ns"가 아니라면 Pod생성 안되도록 함. if "iksoon-ns" != request_namespace: message_text = "namespace is not iksoon-ns (Don't create Pod)" response = admission_response(False, message_text) logging.info(response) return jsonify(response) else: message_text = "namespace is iksoon-ns (validating start)" logging.info(message_text) # Pod labels 정보 확인 # labels ( validate: allow ) # 먼저 pod 생성 yaml에 labels 항목이 있는지 확인 # (labels 항목이 없이 생성될 수도 있음) if 'labels' in kube_api_request["request"]["object"]["metadata"]: message_text = "labels is exsist" logging.info(message_text) else: message_text = "labels is null (Don't create Pod)" response = admission_response(False, message_text) logging.info(response) return jsonify(response) # labels 항목 중에 validate가 있는지 확인 if 'validate' in kube_api_request["request"]["object"]["metadata"]["labels"]: message_text = "validate labels is exsist" logging.info(message_text) label_validating = kube_api_request["request"]["object"]["metadata"]["labels"]["validate"] logging.info("label_validating = %s", label_validating) else: message_text = "validate labels is null (Don't create Pod)" response = admission_response(False, message_text) logging.info(response) return jsonify(response) # validate의 value가 allow인지 확인 if 'allow' != label_validating: message_text = "validate labels value is not allow (Don't create Pod)" response = admission_response(False, message_text) logging.info(response) return jsonify(response) message_text = "validate labels value is allow (create Pod)" response = admission_response(True, message_text) logging.info(response) return jsonify(response) def admission_response(result, message_text): admissionReview = { "response": { "allowed": result, "status": {"message": message_text} } } return admissionReview if __name__ == '__main__': app.run(host= '0.0.0.0', port=6000, debug=True, ssl_context=('/run/secrets/tls/tls.crt', '/run/secrets/tls/tls.key')) |
[response 참고]
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
log 확인을 위해 logging을 사용함.
ssl_context=('/run/secrets/tls/tls.crt', '/run/secrets/tls/tls.key') 은
해당 flask webhook에 HTTPS로 접근하기 위한 인증서을 설정하는 부분으로
아래에서 인증서를 생성하고
deployment 시 해당 경로에 volume 형태로 mount할 것임.
[python flask test 쿼리]
curl -d '{"request": {"uid":"de47", "namespace":"iksoon-ns", "object": {"spec":{"containers": [{"image": "peksoon/iksoon_tomcat:1.0.6"}] }, "metadata": { "labels": {"test": "data1", "validate": "allow"}} }}}' -H "Content-Type: application/json" -X POST http://10.0.2.5:6000/iksoon/validate
2.2. 상위 Webhook Server가 실행되는 Container image 생성
DockerFile을 사용해서 생성
[Dockerfile 생성]
원활한 디버깅을 위해 필요한 tool을 전부 설치한 이미지를 생성
| FROM centos:7 RUN yum install -y net-tools RUN yum install -y gcc openssl-devel bzip2-devel libffi-devel wget RUN yum install -y make RUN yum update RUN wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz RUN tar xvf Python-3.8.5.tgz WORKDIR Python-3.8.5/ RUN ./configure --enable-optimizations RUN make altinstall RUN unlink /bin/python RUN ln -s /usr/local/bin/python3.8 /bin/python RUN pip3.8 install flask RUN pip3.8 install jsonpatch RUN pip3.8 install requests RUN mkdir /admission_test WORKDIR /admission_test ADD ./iksoon-webhook-validating.py . CMD ["python", "/admission_test/iksoon-webhook-validating.py"] |
[Dockerfile 빌드 명령어]
docker build --tag mutatingadmission:1.0.0 .
[Docker hub에 push 해놓음]
docker tag [생성된 image id] peksoon/admission_test:1.0.0
docker login
docker push peksoon/admission_test:1.0.0
3. Flask Webhook HTTPS 접속용 인증서 생성
[명령어]
webhook-server-tls.crt 생성 시 CN을 iksoon-admission-service.default.svc로 설정했는데
이는 아래에서 생성할 webhook deployment 접속 service object의 domain 을 넣어줘야함.
| openssl req -nodes -new -x509 -keyout ca.key -out ca.crt -subj "/CN=iksoon admission controller" openssl genrsa -out webhook-server-tls.key 2048 openssl req -new -key webhook-server-tls.key -subj "/CN=iksoon-admission-service.default.svc" \ | openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -out webhook-server-tls.crt |
[결과]
4. TLS Secret 생성
Secret 을 생성해서 deployment에서 해당 secret 를
volume 행태로 mount해서
flask webhook container에 인증서를 전달할 것임.
[명령어]
kubectl -n default create secret tls webhook-certs \
--cert "webhook-server-tls.crt" \
--key "webhook-server-tls.key"
[결과]
5. Webhook Server deployment 와 Service 생성
[Deployment, Servcie yaml 예시]
| apiVersion: apps/v1 kind: Deployment metadata: name: iksoon-admission-webhook namespace: default labels: app: mutating-webhook spec: replicas: 1 selector: matchLabels: app: admission-pod template: metadata: labels: app: admission-pod spec: containers: - name: webhook image: peksoon/admission_test:1.0.0 imagePullPolicy: Always ports: - containerPort: 6000 name: flask-webhook volumeMounts: - name: webhook-tls mountPath: /run/secrets/tls readOnly: true volumes: - name: webhook-tls secret: secretName: webhook-certs --- apiVersion: v1 kind: Service metadata: name: iksoon-admission-service namespace: default spec: ports: - port: 443 targetPort: flask-webhook selector: app: admission-pod |
6. ca.crt 파일 base64 인코딩
[명령어]
openssl base64 -A <"ca.crt"
결과 복사 해놓음
7. Webhook Server deployment 와 Service 생성
MutatingWebhookConfiguration name 은
상위 webhook deployment와 연결되어있는 service의 domain이여야함
service name = iksoon-admission-service
namespace name = default
임으로 생성되는 domain = iksoon-admission-service.default.svc
clientConfig의 service name과 namespace는
상위 webhook deployment와 연결되어있는 service의
service name = iksoon-admission-service
namespace name = default
을 입력함.
path: "/iksoontest"
에서 경로를 "/iksoontest" 로 설정한 이유는
상위 webhook python 코드의
@app.route('/iksoontest', methods=['POST'])
부분에서 경로를 "/iksoontest"로 설정했기 때문.
[validating_admission_controller.yaml]
| apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration metadata: name: validating-admission-webhook webhooks: - name: iksoon-admission-service.default.svc clientConfig: service: name: iksoon-admission-service namespace: default path: "/iksoon/validate" caBundle: [여기에 상위에서 복사한 base64 encoding ca.crt 값이 들어감.] rules: # Pod를 생성하는 경우에만 해당 mutating admission이 적용됨. 예) deployment로 생성하면 안됨. - operations: [ "CREATE" ] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] |
8. 결과 확인 pod 생성 test
[test.yaml]
| apiVersion: v1 kind: Pod metadata: name: iksoon-pod namespace: default labels: app: test spec: containers: - name: iksoon-tomcat image: peksoon/iksoon_tomcat:1.0.6 ports: - containerPort: 8080 |
[Pod생성 결과 확인]
iksoon-ns namespace에 Pod를 배포했는데
labels 에 validate: allow 가 없어서 배포가 안됨
[test.yaml]
| apiVersion: v1 kind: Pod metadata: name: iksoon-pod namespace: default labels: app: test validate: allow spec: containers: - name: iksoon-tomcat image: peksoon/iksoon_tomcat:1.0.6 ports: - containerPort: 8080 |
[Pod생성 결과 확인]
labels 에 validate: allow를 추가하면 정상적으로 배포됨
참고용 Mutating, Validating admission controller webhook 소스코드
from flask import Flask, request, jsonify
import pprint
import requests
import logging
import copy
import base64
import jsonpatch
from requests.auth import HTTPBasicAuth
logging.basicConfig(filename = "admission_controller.log", level = logging.DEBUG)
app = Flask(__name__)
@app.route('/iksoon/mutate', methods=['POST'])
def mutate():
message_text = ""
patch = ""
admissionReview = ""
container_image= ""
mutating_result = []
# kubernetes API Server의 Request를 해당 webhook으로 가져옴
kube_api_request= request.json
logging.info("\n")
logging.info('--Kube-apiserver request---')
logging.info(kube_api_request)
logging.info("\n")
#namespace data를 알아옴
request_namespace = kube_api_request["request"]["namespace"]
#namesapce 가 "iksoon-ns"과 동일한지 확인
logging.info("----namespace name check result----")
logging.info("namespace = %s", request_namespace)
#namesapce 가 "iksoon-ns"가 아니라면 mutating 하지 않고 return
if "iksoon-ns" != request_namespace:
message_text = "namespace is not iksoon-ns (Don't Mutating)"
admissionReview = admission_response(True, message_text)
logging.info(admissionReview)
return jsonify(admissionReview)
else:
message_text = "namespace is iksoon-ns (Mutating start)"
logging.info(message_text)
# 첫번째 container image정보를 확인
container_image = kube_api_request["request"]["object"]["spec"]["containers"][0]["image"]
logging.info("container image = %s", container_image)
# 첫번째 container image가 "peksoon/iksoon_tomcat:1.0.6"과 동일한지 확인
if "peksoon/iksoon_tomcat:1.0.6" == container_image:
container_mutating={'op': 'replace', 'path': '/spec/containers/0/image', 'value': 'peksoon/iksoon_mysql:1.0.2'}
mutating_result.append(container_mutating)
# labels 추가 ( mutating: admission-test )
# 먼저 pod 생성 yaml에 labels 항목이 있는지 확인
# (labels 항목이 없이 생성될 수도 있음)
if 'labels' in kube_api_request["request"]["object"]["metadata"]:
message_text = "labels is exsist"
logging.info(message_text)
# labels 항목 중에 mutate가 있는지 확인
# mutate key가 있다면 value를 무조건 admission-test로 변경
if 'mutate' in kube_api_request["request"]["object"]["metadata"]["labels"]:
message_text = "mutate labels is exsist (labels mutate)"
logging.info(message_text)
container_mutating={'op': 'replace', 'path': '/metadata/labels/mutate', 'value': 'admission-test'}
# labels 항목 중에 mutate가 없다면 labels 추가
else:
message_text = "mutate labels is null (labels add)"
logging.info(message_text)
labelAdd_mutating={'op': 'add', 'path': '/metadata/labels/mutate', 'value': 'admission-test'}
mutating_result.append(labelAdd_mutating)
logging.info(labelAdd_mutating)
else:
message_text = "labels is null (Don't Mutating)"
logging.info(message_text)
if 0 != len(mutating_result):
message_text = "Data mutating"
logging.info(mutating_result)
patch = jsonpatch.JsonPatch(mutating_result)
#jsonpath를 base64 인코딩함.
base64_patch = base64.b64encode(patch.to_string().encode("utf-8")).decode("utf-8")
uid = kube_api_request["request"]["uid"]
admissionReview = admission_response_mutating(True, message_text, uid, base64_patch)
else:
message_text = "nothing to change"
admissionReview = admission_response(True, message_text)
# Webhook Server에서 Kubernetes API Server로 response
logging.info("\n")
logging.info('--Kube-apiserver response result---')
logging.info(admissionReview)
logging.info("\n")
return jsonify(admissionReview)
@app.route('/iksoon/validate', methods=['POST'])
def validate():
message_text = ""
admissionReview = ""
label_validating = ""
# kubernetes API Server의 Request를 해당 webhook으로 가져옴
kube_api_request= request.json
logging.info("\n")
logging.info('--Kube-apiserver request---')
logging.info(kube_api_request)
logging.info("\n")
#namespace data를 알아옴
request_namespace = kube_api_request["request"]["namespace"]
#namesapce 가 "iksoon-ns"과 동일한지 확인
logging.info("----namespace name check result----")
logging.info("namespace = %s", request_namespace)
#namesapce 가 "iksoon-ns"가 아니라면 Pod생성 안되도록 함.
if "iksoon-ns" != request_namespace:
message_text = "namespace is not iksoon-ns (Don't create Pod)"
response = admission_response(False, message_text)
logging.info(response)
return jsonify(response)
else:
message_text = "namespace is iksoon-ns (validating start)"
logging.info(message_text)
# Pod labels 정보 확인
# labels ( validate: allow )
# 먼저 pod 생성 yaml에 labels 항목이 있는지 확인
# (labels 항목이 없이 생성될 수도 있음)
if 'labels' in kube_api_request["request"]["object"]["metadata"]:
message_text = "labels is exsist"
logging.info(message_text)
else:
message_text = "labels is null (Don't create Pod)"
response = admission_response(False, message_text)
logging.info(response)
return jsonify(response)
# labels 항목 중에 validate가 있는지 확인
if 'validate' in kube_api_request["request"]["object"]["metadata"]["labels"]:
message_text = "validate labels is exsist"
logging.info(message_text)
label_validating = kube_api_request["request"]["object"]["metadata"]["labels"]["validate"]
logging.info("label_validating = %s", label_validating)
else:
message_text = "validate labels is null (Don't create Pod)"
response = admission_response(False, message_text)
logging.info(response)
return jsonify(response)
# validate의 value가 allow인지 확인
if 'allow' != label_validating:
message_text = "validate labels value is not allow (Don't create Pod)"
response = admission_response(False, message_text)
logging.info(response)
return jsonify(response)
message_text = "validate labels value is allow (create Pod)"
response = admission_response(True, message_text)
logging.info(response)
return jsonify(response)
def admission_response(result, message_text):
admissionReview = {
"response": {
"allowed": result,
"status": {"message": message_text}
}
}
return admissionReview
def admission_response_mutating(result, message_text, uid_data, path_data):
#Kube-apiserver로 보내는 response 필드로 아래와 같이 data 포맷이 정해져있어서
#해당 규칙대로 return 해줘야함
#return 값은 patch 항목도 base64로 설정되는것에 주의함.
#allowed 항목을 보면
#원래 요청 거절을 validate에서 하지만 해당 값이 false면 mutate 단계에서도 요청을 거절할 수 있음
#거절할 경우 deny를 추가해서 거절에 대한 이유를 같이 return 해주는것이 좋음
admissionReview = {
"response": {
"allowed": True,
"uid": uid_data,
"patch": path_data,
"patchtype": "JSONPatch",
"status": {"message": message_text}
}
}
return admissionReview
if __name__ == '__main__':
app.run(host= '0.0.0.0', port=6000, debug=True, ssl_context=('/run/secrets/tls/tls.crt', '/run/secrets/tls/tls.key'))
제 글을 복사할 시 출처를 명시해주세요.
글에 오타, 오류가 있다면 댓글로 알려주세요! 바로 수정하겠습니다!
'클라우드 > Kubernetes' 카테고리의 다른 글
| 35. Custom Resource Definition (CRD) Python (0) | 2020.11.05 |
|---|---|
| 34. Kubernetes Custom Resource 개념 (0) | 2020.11.05 |
| 32. Mutating Admission Controller (0) | 2020.11.05 |
| 31. Admission Controller (0) | 2020.11.05 |
| 30. kubernetes PriorityClass ( Priority & Preemption ) (0) | 2020.10.11 |