kubernetes kubelet cAdvisor 분석

Kubernetes/Monitoring

kubernetes kubelet cAdvisor 분석

이쿠우우 2021. 7. 17. 17:32

kubernetes kubelet cAdvisor

kubernetes cluster의 모든 Node에는 kubelet이 설치되어 있음.

해당 kubelet에는 cAdvisor가 탐재되어 있음.

cAdvisor는 kubelet 바이너리에 통합된 오픈 소스 에이전트로

cAdvisor를 통해서 Node의 자원 사용량을 모니터링하고 컨테이너의 성능을 분석함.

특정 Node(Pod 레벨에서는 실행되지 않음)에서 실행되고 있는

모든 Container의 CPU와 Memory, File, Network 사용량 등 System metrics 정보를 수집하고

그 외에 시스템 이벤트도 모니터링함.

cAdvisor는정보를 수집해서 kubelet의 메모리에 올려놓음.

아래 예제에서는

curl 명령을 통해 해당 cAdvisor가 수집한 metrics 정보를 가져오는 것을 알아봄.

test 환경

kubelet Port = 10250

test환경 master node ip = 10.0.2.15

test환경 worker node ip = 10.0.2.4

1. matser node의 /etc/kubernetes/pki 경로에 있는 인증서로 kubelet https 인증

curl https://localhost:10250/metrics -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key

curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key

2. SA를 생성해서 token으로 kubelet https 인증

참고 : https://kubernetes.io/ko/docs/concepts/cluster-administration/system-metrics/

[SA생성 명령어]

kubectl create sa kubelet-api-test

kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test

SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')

TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')

echo ${TOKEN}

[SA Token을 사용한 curl 예시]

curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/metrics

curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/stats/summary

3. cAdvisor Metrics 정보 조회 (모든 정보)

/metrics

/metrics/cadvisor

/metrics/resource

/metrics/probes

엔드포인트에서 메트릭을 노출함.

이러한 메트릭은 동일한 라이프사이클을 가지지 않음.

curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/metrics/cadvisor

[metrics/cadvisor 명령어로 확인 할 수 있는 정보는 아래 링크에 나와 있음]

Monitoring cAdvisor with Prometheus

https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md

3.1. Metrics 중 Network 정보

Metric name	Type	Description	Unit	disable_metrics parameter
container_network_receive_bytes_total	Counter	Cumulative count of bytes received	bytes	network
container_network_receive_packets_dropped_total	Counter	Cumulative count of packets dropped while receiving		network
container_network_receive_packets_total	Counter	Cumulative count of packets received		network
container_network_receive_errors_total	Counter	Cumulative count of errors encountered while receiving		network
container_network_transmit_bytes_total	Counter	Cumulative count of bytes transmitted	bytes	network
container_network_transmit_packets_total	Counter	Cumulative count of packets transmitted		network
container_network_transmit_packets_dropped_total	Counter	Cumulative count of packets dropped while transmitting		network
container_network_transmit_errors_total	Counter	Cumulative count of errors encountered while transmitting		network
container_network_tcp_usage_total	Gauge	tcp connection usage statistic for container		tcp
container_network_tcp6_usage_total	Gauge	tcp6 connection usage statistic for container		tcp
container_network_udp_usage_total	Gauge	udp connection usage statistic for container		udp
container_network_udp6_usage_total	Gauge	udp6 connection usage statistic for container		udp

한개의 deployment 안에 1개의 Pod를 생성했고

해당 Pod안에는 mysql, tomcat 2개의 container가 생성되어 있지만

각 container에 해당하는 network 정보는 없고

Pause Container의 Network 정보만 있음

즉 Pod network 정보만 있음

Pod 내부의 container 간 통신에 대한 정보는 알 수 없음

# HELP container_network_receive_bytes_total Cumulative count of bytes received

# TYPE container_network_receive_bytes_total counter

container_network_receive_bytes_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 648 1604892032886

# HELP container_network_receive_errors_total Cumulative count of errors encountered while receiving

# TYPE container_network_receive_errors_total counter

container_network_receive_errors_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886

# HELP container_network_receive_packets_dropped_total Cumulative count of packets dropped while receiving

# TYPE container_network_receive_packets_dropped_total counter

container_network_receive_packets_dropped_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886

# HELP container_network_receive_packets_total Cumulative count of packets received

# TYPE container_network_receive_packets_total counter

container_network_receive_packets_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 8 1604892032886

# HELP container_network_transmit_bytes_total Cumulative count of bytes transmitted

# TYPE container_network_transmit_bytes_total counter

container_network_transmit_bytes_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 300 1604892032886

# HELP container_network_transmit_errors_total Cumulative count of errors encountered while transmitting

# TYPE container_network_transmit_errors_total counter

container_network_transmit_errors_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886

# HELP container_network_transmit_packets_dropped_total Cumulative count of packets dropped while transmitting

# TYPE container_network_transmit_packets_dropped_total counter

container_network_transmit_packets_dropped_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886

# HELP container_network_transmit_packets_total Cumulative count of packets transmitted

# TYPE container_network_transmit_packets_total counter

container_network_transmit_packets_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 4 1604892032886

3.2. container_network_tcp_usage_total 정보 조회 방법

3.2.1. container_network_tcp_usage_total , udp 에 대한 정보를 확인할 수 없는 이유

기본적으로 kubelet에 내장되어있는 cAdvisor에서

container_network_tcp_usage_total , udp 에 대한 정보를 확인할 수가 없었는데

그 이유는 kubelet에서 사용하는 cAdvisor의 경우 해당 정보 조회 옵션이 default 로 disable 이기 때문임.

참고 페이지 : https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#metrics

3.2.2. cAdvisor 에서 container_network_tcp_usage_total , udp 정보를 enable하는 방법

참고 페이지 : https://github.com/google/cadvisor/issues/2380

참고 페이지 중 핵심이 되는 부분을 가져옴.

Option 1 and option 2 are the same endpoint.
One is just proxied by the API Server.
Prefer (2) when possible because it is more direct.
If you want to customize the set of metrics exposed by cAdvisor,
you can run it yourself as a daemonset.
If you just want the metrics in the kubelet's metrics/cadvisor endpoint,
I would just use that to save on resource consumption.

[cAdvisor runtime option]
--disable_metrics=tcp,advtcp,udp,sched,process,hugetlb
: comma-separated list of metrics to be disabled.
Options are 'disk', 'network', 'tcp', 'advtcp', 'udp', 'sched', 'process', 'hugetlb'.
Note: tcp and udp are disabled by default due to high CPU usage. (default tcp,advtcp,udp,sched,process,hugetlb)

위에 글을 확인한 결과 cadvisor를 통해 tcp, udp metrics 정보를 확인하기 위해서는

별도로 사용자가 cAdvisor를 Daemonset으로 실행시켜야함.

[--disable_metrics 설정가능 옵션 : cAdvisor v0.37.0 기준]

disk, network, tcp, udp, sched, process, advtcp, hugetlb

3.2.3. cAdvisor Daemonset 배포

[cAdvisor Daemonset 참고]

https://github.com/google/cadvisor/blob/master/deploy/kubernetes/base/daemonset.yaml

[cAdvisor images 참고]

https://github.com/google/cadvisor/releases

namespace 먼저 생성해줌 : kubectl create ns monitoring

주의)

tcp,udp 정보를 조회하게 되면 data양이 굉장히 많아져서 각 worker node에 리소스 과부하가 올 수 있음

반드시 resource 설정으로 리소스 사용량을 제한해야함.

[예제 cAdvisor_Daemonset.yaml]

apiVersion: apps/v1 # for Kubernetes versions before 1.9.0 use apps/v1beta2
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: cadvisor
spec:
  selector:
    matchLabels:
      name: cadvisor
  template:
    metadata:
      labels:
        name: cadvisor
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
containers:
      - name: cadvisor
        args:
          - --disable_metrics=disk,percpu,network,sched,process # enable only tcp,udp
          - --docker_only
          - --housekeeping_interval=30s
          - --max_housekeeping_interval=60s
        image: gcr.io/cadvisor/cadvisor:v0.37.0
        resources:
          requests:
            memory: 200Mi
            cpu: 150m
          limits:
            memory: 2000Mi
            cpu: 300m
        volumeMounts:
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: true
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: disk
          mountPath: /dev/disk
          readOnly: true
        - name: cgroup
          mountPath: /sys/fs/cgroup
          readOnly: true
        ports:
          - name: http
            containerPort: 8080
            protocol: TCP
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      volumes:
      - name: rootfs
        hostPath:
          path: /
      - name: var-run
        hostPath:
          path: /var/run
      - name: sys
        hostPath:
          path: /sys
      - name: docker
        hostPath:
          path: /var/lib/docker
      - name: disk
        hostPath:
          path: /dev/disk
      - name: cgroup
        hostPath:
          path: /cgroup

[metrics 조회 명령어]

curl <배포된 cAdvisor pod ip>:8080/metrics

예) curl 10.244.1.127:8080/metrics

3.2.4. cAdvisor Daemonset 배포 결과

한개의 pod안에 2의 container(tomcat, mysql)가 있는경우

tcp와 udp로 조회되는 정보는

pause container

tomcat container

mysql container

에대한 정보가 조회됨.

UDP 의 경우 dropped, listen, rxqueued, txqueued,

TCP의 경우 close, closewait, closing, established, finwait1, finwait2, lastack, listen, synrecv, synsent, timewait

정보를 확인할 수 있음.

container_network_tcp_usage_total 정보

"container_network_tcp_usage_total"{

"container_label_annotation_checksum_config=""",

"container_label_annotation_checksum_dashboards_json_config=""",

"container_label_annotation_checksum_sc_dashboard_provider_config=""",

"container_label_annotation_checksum_secret=""",

"container_label_annotation_io_kubernetes_container_hash=""191b1040",

"container_label_annotation_io_kubernetes_container_ports=""[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",

"container_label_annotation_io_kubernetes_container_restartCount=""0",

"container_label_annotation_io_kubernetes_container_terminationMessagePath=""/dev/termination-log",

"container_label_annotation_io_kubernetes_container_terminationMessagePolicy=""File",

"container_label_annotation_io_kubernetes_pod_terminationGracePeriod=""30",

"container_label_annotation_kubernetes_io_config_seen=""",

"container_label_annotation_kubernetes_io_config_source=""",

"container_label_app=""",

"container_label_app_kubernetes_io_instance=""",

"container_label_app_kubernetes_io_name=""",

"container_label_chart=""",

"container_label_component=""",

"container_label_controller_revision_hash=""",

"container_label_heritage=""",

"container_label_io_kubernetes_container_logpath=""/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",

"container_label_io_kubernetes_container_name=""iksoon-mysql",

"container_label_io_kubernetes_docker_type=""container",

"container_label_io_kubernetes_pod_name=""iksoon-deployment-tomcat-6769c7766f-tzr28",

"container_label_io_kubernetes_pod_namespace=""default",

"container_label_io_kubernetes_pod_uid=""ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",

"container_label_io_kubernetes_sandbox_id=""6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",

container_label_k8s_app="",

"container_label_maintainer=""",

"container_label_name=""",

"container_label_pod_template_generation=""",

"container_label_pod_template_hash=""",

"container_label_release=""",

"container_label_tier=""",

"id=""/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",

"image=""sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",

"name=""k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",

"tcp_state=""close"

container_network_udp_usage_total 정보

"container_network_udp_usage_total"{

"container_label_annotation_checksum_config=""",

"container_label_annotation_checksum_dashboards_json_config=""",

"container_label_annotation_checksum_sc_dashboard_provider_config=""",

"container_label_annotation_checksum_secret=""",

"container_label_annotation_io_kubernetes_container_hash=""191b1040",

"container_label_annotation_io_kubernetes_container_ports=""[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",

"container_label_annotation_io_kubernetes_container_restartCount=""0",

"container_label_annotation_io_kubernetes_container_terminationMessagePath=""/dev/termination-log",

"container_label_annotation_io_kubernetes_container_terminationMessagePolicy=""File",

"container_label_annotation_io_kubernetes_pod_terminationGracePeriod=""30",

"container_label_annotation_kubernetes_io_config_seen=""",

"container_label_annotation_kubernetes_io_config_source=""",

"container_label_app=""",

"container_label_app_kubernetes_io_instance=""",

"container_label_app_kubernetes_io_name=""",

"container_label_chart=""",

"container_label_component=""",

"container_label_controller_revision_hash=""",

"container_label_heritage=""",

"container_label_io_kubernetes_container_logpath=""/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",

"container_label_io_kubernetes_container_name=""iksoon-mysql",

"container_label_io_kubernetes_docker_type=""container",

"container_label_io_kubernetes_pod_name=""iksoon-deployment-tomcat-6769c7766f-tzr28",

"container_label_io_kubernetes_pod_namespace=""default",

"container_label_io_kubernetes_pod_uid=""ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",

"container_label_io_kubernetes_sandbox_id=""6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",

container_label_k8s_app="",

"container_label_maintainer=""",

"container_label_name=""",

"container_label_pod_template_generation=""",

"container_label_pod_template_hash=""",

"container_label_release=""",

"container_label_tier=""",

"id=""/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",

"image=""sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",

"name=""k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",

"udp_state=""listen"

참고용 container_network_udp,tcp_usage_total 확인정보

# HELP container_network_tcp_usage_total tcp connection usage statistic for container

# TYPE container_network_tcp_usage_total gauge

# (pause container 정보)

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="close"} 0

# (mysql container 정보)

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="close"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closewait"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closing"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="established"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait1"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait2"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="lastack"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="listen"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synrecv"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synsent"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="timewait"} 0

# (tomcat container 정보)

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="close"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closewait"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closing"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="established"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait1"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait2"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="lastack"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="listen"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synrecv"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synsent"} 0

container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="timewait"} 0

# HELP container_network_udp_usage_total udp connection usage statistic for container

# TYPE container_network_udp_usage_total gauge

# (pause container 정보)

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="dropped"} 0

# (mysql container 정보)

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="dropped"} 0

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="listen"} 0

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="rxqueued"} 0

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="txqueued"} 0

# (tomcat container 정보)

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="dropped"} 0

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="listen"} 0

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="rxqueued"} 0

container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="txqueued"} 0

3.3. Metrics 중 CPU 정보

한개의 deployment 안에 1개의 Pod를 생성했고

해당 Pod안에는 mysql, tomcat 2개의 container가 생성되어 있어서

각 tomcat, mysql Container에 해당하는 CPU정보가 나옴.

즉 Pod 내부에 있는 모든 Container에 대한 정보를 모두 확인 할 수 있음

CPU정보 중 예제로 몇가지만 가져와봄

# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.

# TYPE container_cpu_load_average_10s gauge

container_cpu_load_average_10s{container="iksoon-mysql",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-5f6f330a50042b61476c280f9b44df17554e77738571eb21e97c3e429a924341.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892019441

container_cpu_load_average_10s{container="iksoon-tomcat",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-c9d1625dff77b640ffc9331717c5207ee2e5e26c8ed9405a70ff497817822344.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892023537

# HELP container_cpu_system_seconds_total Cumulative system cpu time consumed in seconds.

# TYPE container_cpu_system_seconds_total counter

container_cpu_system_seconds_total{container="iksoon-mysql",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-5f6f330a50042b61476c280f9b44df17554e77738571eb21e97c3e429a924341.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 1.24 1604892019441

container_cpu_system_seconds_total{container="iksoon-tomcat",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-c9d1625dff77b640ffc9331717c5207ee2e5e26c8ed9405a70ff497817822344.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0.49 1604892023537

4. cAdvisor /stats/summary 정보 조회 (요약 정보)

curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/stats/summary

[stat/summary 정보 중 cpu와 memory 정보만 조회하고 싶은 경우]

url에 사용자쿼리(querystring)를 사용해서

?only_cpu_and_memory=true를 붙여주면 됨.

curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/stats/summary?only_cpu_and_memory=true

[stat/summary로 확인할 수 있는 pod에 대한 정보는 아래와 같음]

pod의 CPU, Memory, Network, Volume, ephemeral-storage, process_stats 정보,

pod 내부에 생성된 container의 CPU, Memory, rootfs, logs 정보를 확인할 수 있음

(확인할 수 있는 Network 정보)

rxBytes

rxErrors

txBytes

txErrors

RX: Receive Data (수신 신호)

Tx : Transmit Data (송신 신호)

(rootfs란?)

rootfs 라고 하는 root 파일 시스템 layer이다.

Root 파일 시스템은 실제 OS 가 설치된다.

(예를 들어 Debian 이나 Ubuntu).

원래 리눅스에서는 root filesystem은 처음 mount될 때 read-only로 mount가 된 후

integrity check후 read-write으로 바뀐다.

하지만 Docker에서는 계속해서 read-only 모드이다.

Read-write 모드로 변환하지 않는 이유는 Docker는 union mount를 사용해서

read-only 파일 시스템들을 root 파일 시스템 위에 덮는 구조로 이루어져 있기 때문이다.

참고로 union mount는 여러게의 파일 시스템을 mount하되

하나의 파일 시스템만이 mount된것 처럼 사용하는 방법이다.

[stat/summary 정보]

"podRef": {

"name": "iksoon-deployment-tomcat-85567fbc4-h24xc",

"namespace": "default",

"uid": "beec608d-28c2-4c34-906c-7025b109556f"

"startTime": "2020-11-06T03:13:52Z",

"containers": [

{

"name": "iksoon-tomcat",

"startTime": "2020-11-06T03:13:54Z",

"cpu": {

"time": "2020-11-08T23:47:42Z",

"usageNanoCores": 1522200,

"usageCoreNanoSeconds": 321392726196

"memory": {

"time": "2020-11-08T23:47:42Z",

"usageBytes": 104632320,

"workingSetBytes": 104595456,

"rssBytes": 104583168,

"pageFaults": 64407,

"majorPageFaults": 0

"rootfs": {

"time": "2020-11-08T23:47:42Z",

"availableBytes": 3283169280,

"capacityBytes": 18238930944,

"usedBytes": 49152,

"inodesFree": 6412639,

"inodes": 6730584,

"inodesUsed": 13

"logs": {

"time": "2020-11-08T23:47:42Z",

"availableBytes": 3283169280,

"capacityBytes": 18238930944,

"usedBytes": 24576,

"inodesFree": 6412639,

"inodes": 6730584,

"inodesUsed": 317945

}

"cpu": {

"time": "2020-11-08T23:47:35Z",

"usageNanoCores": 1417477,

"usageCoreNanoSeconds": 321411574632

"memory": {

"time": "2020-11-08T23:47:35Z",

"usageBytes": 104673280,

"workingSetBytes": 104636416,

"rssBytes": 104624128,

"pageFaults": 0,

"majorPageFaults": 0

"network": {

"time": "2020-11-08T23:47:42Z",

"name": "eth0",

"rxBytes": 648,

"rxErrors": 0,

"txBytes": 300,

"txErrors": 0,

"interfaces": [

{

"name": "eth0",

"rxBytes": 648,

"rxErrors": 0,

"txBytes": 300,

"txErrors": 0

}

]

"volume": [

{

"time": "2020-11-06T03:14:24Z",

"availableBytes": 1987076096,

"capacityBytes": 1987088384,

"usedBytes": 12288,

"inodesFree": 485120,

"inodes": 485129,

"inodesUsed": 9,

"name": "default-token-zr6hw"

}

"ephemeral-storage": {

"time": "2020-11-08T23:47:42Z",

"availableBytes": 3283169280,

"capacityBytes": 18238930944,

"usedBytes": 73728,

"inodesFree": 6412639,

"inodes": 6730584,

"inodesUsed": 13

"process_stats": {

"process_count": 0

}

kubernetes에서 사용하는 cAdvisor 분석 결과

1. Metrics-Server는 왜 kubelet의 cAdvisor정보를 /metrics로 모두 가져오는게 아니라 /stat/summary 정보만 가져올까?

Metrics-Server 코드를 분석해보면 /stat/summary request로 요약정보만 가져오고

그 중 cpu, memory 정보만 kubelet으로 보냄.

그 이유는 cAdvisor 에서 모든 정보를 가져오게되면 대량의 log로 인해 과부하가 발생할 수 있기때문.

(참고)

https://github.com/google/cadvisor/issues/2144

2. 그렇다면 사용자가 직접 cAdvisor배포하고 수집하는 자료 중 특정항목을 disabled 시키고 원하는 정보만 수집하면 되지 않나?

cAdvisor의 -disable_metrics flag를 사용해도 일부 metrics만 제외시킬 수 있음.

실제로 -disable_metrics flag에 disk, network, tcp, udp, sched, process, advtcp, hugetlb 옵션을 모두 적용하고

test 해본 결과 그래도 상당량의 metrics 정보를 생성하고 있음.

그로 인해 기본적인 metrics정보만 수집하더라도 과부하가 발생할 수 있음.

cAdvisor 공식 page에서 패치 note를 보면 verison이 증가할 때 마다

disable_metrics 옵션을 추가하고 있는데

왜 애초에 모든 metrics정보를 추가해주지 않는건지는 의문임.

(참고)

https://github.com/google/cadvisor/issues/2602

3. kubelet의 cAdvisor는 왜 tcp, udp 정보를 수집하지 않도록 설정했을까?

/stat/summary 결과를 보면 network 트래픽 사용량 정보만 조회할 수 있고

pod안에 생성된 container의 tcp, udp 정보는 조회할 수 없음

이 또한 과부하를 고려해서 설정해놓은것이라고 함.

결론

개발자가 원하는 정보만 추출하기 위해서는

직접 exporters 를 생성해야함.

ex) prometheus exporter

제 글을 복사할 시 출처를 명시해주세요.
글에 오타, 오류가 있다면 댓글로 알려주세요! 바로 수정하겠습니다!

[참고]

https://stackoverflow.com/questions/58369289/how-to-get-real-time-resource-usage-of-a-pod-in-k8s

https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/3

https://labs.f-secure.com/blog/attacking-kubernetes-through-kubelet/

https://blog.ziki.cn/post/kubelet-api/

https://docs.netapp.com/us-en/cloudinsights/pdfs/pages/task_config_telegraf_kubernetes.pdf

[cAdvisor]

https://github.com/google/cadvisor

[Prometheus cAdvisor 참고]

https://engineering.linecorp.com/ko/blog/prometheus-container-kubernetes-cluster/