kubernetes kubelet cAdvisor
kubernetes cluster의 모든 Node에는 kubelet이 설치되어 있음.
해당 kubelet에는 cAdvisor가 탐재되어 있음.
cAdvisor는 kubelet 바이너리에 통합된 오픈 소스 에이전트로
cAdvisor를 통해서 Node의 자원 사용량을 모니터링하고 컨테이너의 성능을 분석함.
특정 Node(Pod 레벨에서는 실행되지 않음)에서 실행되고 있는
모든 Container의 CPU와 Memory, File, Network 사용량 등 System metrics 정보를 수집하고
그 외에 시스템 이벤트도 모니터링함.
cAdvisor는정보를 수집해서 kubelet의 메모리에 올려놓음.
아래 예제에서는
curl 명령을 통해 해당 cAdvisor가 수집한 metrics 정보를 가져오는 것을 알아봄.
test 환경
kubelet Port = 10250
test환경 master node ip = 10.0.2.15
test환경 worker node ip = 10.0.2.4
1. matser node의 /etc/kubernetes/pki 경로에 있는 인증서로 kubelet https 인증
curl https://localhost:10250/metrics -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key
curl https://localhost:10250/stats/summary -k --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key
2. SA를 생성해서 token으로 kubelet https 인증
참고 : https://kubernetes.io/ko/docs/concepts/cluster-administration/system-metrics/
[SA생성 명령어]
kubectl create sa kubelet-api-test
kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
echo ${TOKEN}
[SA Token을 사용한 curl 예시]
curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/metrics
curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/stats/summary
3. cAdvisor Metrics 정보 조회 (모든 정보)
/metrics
/metrics/cadvisor
/metrics/resource
/metrics/probes
엔드포인트에서 메트릭을 노출함.
이러한 메트릭은 동일한 라이프사이클을 가지지 않음.
curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/metrics/cadvisor
[metrics/cadvisor 명령어로 확인 할 수 있는 정보는 아래 링크에 나와 있음]
Monitoring cAdvisor with Prometheus
https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md
3.1. Metrics 중 Network 정보
| Metric name | Type | Description | Unit | disable_metrics parameter |
| container_network_receive_bytes_total | Counter | Cumulative count of bytes received | bytes | network |
| container_network_receive_packets_dropped_total | Counter | Cumulative count of packets dropped while receiving | network | |
| container_network_receive_packets_total | Counter | Cumulative count of packets received | network | |
| container_network_receive_errors_total | Counter | Cumulative count of errors encountered while receiving | network | |
| container_network_transmit_bytes_total | Counter | Cumulative count of bytes transmitted | bytes | network |
| container_network_transmit_packets_total | Counter | Cumulative count of packets transmitted | network | |
| container_network_transmit_packets_dropped_total | Counter | Cumulative count of packets dropped while transmitting | network | |
| container_network_transmit_errors_total | Counter | Cumulative count of errors encountered while transmitting | network | |
| container_network_tcp_usage_total | Gauge | tcp connection usage statistic for container | tcp | |
| container_network_tcp6_usage_total | Gauge | tcp6 connection usage statistic for container | tcp | |
| container_network_udp_usage_total | Gauge | udp connection usage statistic for container | udp | |
| container_network_udp6_usage_total | Gauge | udp6 connection usage statistic for container | udp |
한개의 deployment 안에 1개의 Pod를 생성했고
해당 Pod안에는 mysql, tomcat 2개의 container가 생성되어 있지만
각 container에 해당하는 network 정보는 없고
Pause Container의 Network 정보만 있음
즉 Pod network 정보만 있음
Pod 내부의 container 간 통신에 대한 정보는 알 수 없음
# HELP container_network_receive_bytes_total Cumulative count of bytes received
# TYPE container_network_receive_bytes_total counter
container_network_receive_bytes_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 648 1604892032886
# HELP container_network_receive_errors_total Cumulative count of errors encountered while receiving
# TYPE container_network_receive_errors_total counter
container_network_receive_errors_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886
# HELP container_network_receive_packets_dropped_total Cumulative count of packets dropped while receiving
# TYPE container_network_receive_packets_dropped_total counter
container_network_receive_packets_dropped_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886
# HELP container_network_receive_packets_total Cumulative count of packets received
# TYPE container_network_receive_packets_total counter
container_network_receive_packets_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 8 1604892032886
# HELP container_network_transmit_bytes_total Cumulative count of bytes transmitted
# TYPE container_network_transmit_bytes_total counter
container_network_transmit_bytes_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 300 1604892032886
# HELP container_network_transmit_errors_total Cumulative count of errors encountered while transmitting
# TYPE container_network_transmit_errors_total counter
container_network_transmit_errors_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886
# HELP container_network_transmit_packets_dropped_total Cumulative count of packets dropped while transmitting
# TYPE container_network_transmit_packets_dropped_total counter
container_network_transmit_packets_dropped_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892032886
# HELP container_network_transmit_packets_total Cumulative count of packets transmitted
# TYPE container_network_transmit_packets_total counter
container_network_transmit_packets_total{container="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-580a2b201d710d7669a82e94406ea16154903683a269720a4847a556b111530a.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 4 1604892032886
3.2. container_network_tcp_usage_total 정보 조회 방법
3.2.1. container_network_tcp_usage_total , udp 에 대한 정보를 확인할 수 없는 이유
기본적으로 kubelet에 내장되어있는 cAdvisor에서
container_network_tcp_usage_total , udp 에 대한 정보를 확인할 수가 없었는데
그 이유는 kubelet에서 사용하는 cAdvisor의 경우 해당 정보 조회 옵션이 default 로 disable 이기 때문임.
참고 페이지 : https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#metrics
3.2.2. cAdvisor 에서 container_network_tcp_usage_total , udp 정보를 enable하는 방법
참고 페이지 : https://github.com/google/cadvisor/issues/2380
참고 페이지 중 핵심이 되는 부분을 가져옴.
| Option 1 and option 2 are the same endpoint. One is just proxied by the API Server. Prefer (2) when possible because it is more direct. If you want to customize the set of metrics exposed by cAdvisor, you can run it yourself as a daemonset. If you just want the metrics in the kubelet's metrics/cadvisor endpoint, I would just use that to save on resource consumption. [cAdvisor runtime option] --disable_metrics=tcp,advtcp,udp,sched,process,hugetlb : comma-separated list of metrics to be disabled. Options are 'disk', 'network', 'tcp', 'advtcp', 'udp', 'sched', 'process', 'hugetlb'. Note: tcp and udp are disabled by default due to high CPU usage. (default tcp,advtcp,udp,sched,process,hugetlb) |
위에 글을 확인한 결과 cadvisor를 통해 tcp, udp metrics 정보를 확인하기 위해서는
별도로 사용자가 cAdvisor를 Daemonset으로 실행시켜야함.
[--disable_metrics 설정가능 옵션 : cAdvisor v0.37.0 기준]
disk, network, tcp, udp, sched, process, advtcp, hugetlb
3.2.3. cAdvisor Daemonset 배포
[cAdvisor Daemonset 참고]
https://github.com/google/cadvisor/blob/master/deploy/kubernetes/base/daemonset.yaml
[cAdvisor images 참고]
https://github.com/google/cadvisor/releases
namespace 먼저 생성해줌 : kubectl create ns monitoring
주의)
tcp,udp 정보를 조회하게 되면 data양이 굉장히 많아져서 각 worker node에 리소스 과부하가 올 수 있음
반드시 resource 설정으로 리소스 사용량을 제한해야함.
[예제 cAdvisor_Daemonset.yaml]
| apiVersion: apps/v1 # for Kubernetes versions before 1.9.0 use apps/v1beta2 kind: DaemonSet metadata: name: cadvisor namespace: cadvisor spec: selector: matchLabels: name: cadvisor template: metadata: labels: name: cadvisor spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: cadvisor args: - --disable_metrics=disk,percpu,network,sched,process # enable only tcp,udp - --docker_only - --housekeeping_interval=30s - --max_housekeeping_interval=60s image: gcr.io/cadvisor/cadvisor:v0.37.0 resources: requests: memory: 200Mi cpu: 150m limits: memory: 2000Mi cpu: 300m volumeMounts: - name: rootfs mountPath: /rootfs readOnly: true - name: var-run mountPath: /var/run readOnly: true - name: sys mountPath: /sys readOnly: true - name: docker mountPath: /var/lib/docker readOnly: true - name: disk mountPath: /dev/disk readOnly: true - name: cgroup mountPath: /sys/fs/cgroup readOnly: true ports: - name: http containerPort: 8080 protocol: TCP automountServiceAccountToken: false terminationGracePeriodSeconds: 30 volumes: - name: rootfs hostPath: path: / - name: var-run hostPath: path: /var/run - name: sys hostPath: path: /sys - name: docker hostPath: path: /var/lib/docker - name: disk hostPath: path: /dev/disk - name: cgroup hostPath: path: /cgroup |
[metrics 조회 명령어]
curl <배포된 cAdvisor pod ip>:8080/metrics
예) curl 10.244.1.127:8080/metrics
3.2.4. cAdvisor Daemonset 배포 결과
한개의 pod안에 2의 container(tomcat, mysql)가 있는경우
tcp와 udp로 조회되는 정보는
pause container
tomcat container
mysql container
에대한 정보가 조회됨.
UDP 의 경우 dropped, listen, rxqueued, txqueued,
TCP의 경우 close, closewait, closing, established, finwait1, finwait2, lastack, listen, synrecv, synsent, timewait
정보를 확인할 수 있음.
container_network_tcp_usage_total 정보
"container_network_tcp_usage_total"{
"container_label_annotation_checksum_config=""",
"container_label_annotation_checksum_dashboards_json_config=""",
"container_label_annotation_checksum_sc_dashboard_provider_config=""",
"container_label_annotation_checksum_secret=""",
"container_label_annotation_io_kubernetes_container_hash=""191b1040",
"container_label_annotation_io_kubernetes_container_ports=""[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",
"container_label_annotation_io_kubernetes_container_restartCount=""0",
"container_label_annotation_io_kubernetes_container_terminationMessagePath=""/dev/termination-log",
"container_label_annotation_io_kubernetes_container_terminationMessagePolicy=""File",
"container_label_annotation_io_kubernetes_pod_terminationGracePeriod=""30",
"container_label_annotation_kubernetes_io_config_seen=""",
"container_label_annotation_kubernetes_io_config_source=""",
"container_label_app=""",
"container_label_app_kubernetes_io_instance=""",
"container_label_app_kubernetes_io_name=""",
"container_label_chart=""",
"container_label_component=""",
"container_label_controller_revision_hash=""",
"container_label_heritage=""",
"container_label_io_kubernetes_container_logpath=""/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",
"container_label_io_kubernetes_container_name=""iksoon-mysql",
"container_label_io_kubernetes_docker_type=""container",
"container_label_io_kubernetes_pod_name=""iksoon-deployment-tomcat-6769c7766f-tzr28",
"container_label_io_kubernetes_pod_namespace=""default",
"container_label_io_kubernetes_pod_uid=""ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",
"container_label_io_kubernetes_sandbox_id=""6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",
container_label_k8s_app="",
"container_label_maintainer=""",
"container_label_name=""",
"container_label_pod_template_generation=""",
"container_label_pod_template_hash=""",
"container_label_release=""",
"container_label_tier=""",
"id=""/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",
"image=""sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",
"name=""k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",
"tcp_state=""close"
}0
container_network_udp_usage_total 정보
"container_network_udp_usage_total"{
"container_label_annotation_checksum_config=""",
"container_label_annotation_checksum_dashboards_json_config=""",
"container_label_annotation_checksum_sc_dashboard_provider_config=""",
"container_label_annotation_checksum_secret=""",
"container_label_annotation_io_kubernetes_container_hash=""191b1040",
"container_label_annotation_io_kubernetes_container_ports=""[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",
"container_label_annotation_io_kubernetes_container_restartCount=""0",
"container_label_annotation_io_kubernetes_container_terminationMessagePath=""/dev/termination-log",
"container_label_annotation_io_kubernetes_container_terminationMessagePolicy=""File",
"container_label_annotation_io_kubernetes_pod_terminationGracePeriod=""30",
"container_label_annotation_kubernetes_io_config_seen=""",
"container_label_annotation_kubernetes_io_config_source=""",
"container_label_app=""",
"container_label_app_kubernetes_io_instance=""",
"container_label_app_kubernetes_io_name=""",
"container_label_chart=""",
"container_label_component=""",
"container_label_controller_revision_hash=""",
"container_label_heritage=""",
"container_label_io_kubernetes_container_logpath=""/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",
"container_label_io_kubernetes_container_name=""iksoon-mysql",
"container_label_io_kubernetes_docker_type=""container",
"container_label_io_kubernetes_pod_name=""iksoon-deployment-tomcat-6769c7766f-tzr28",
"container_label_io_kubernetes_pod_namespace=""default",
"container_label_io_kubernetes_pod_uid=""ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",
"container_label_io_kubernetes_sandbox_id=""6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",
container_label_k8s_app="",
"container_label_maintainer=""",
"container_label_name=""",
"container_label_pod_template_generation=""",
"container_label_pod_template_hash=""",
"container_label_release=""",
"container_label_tier=""",
"id=""/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",
"image=""sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",
"name=""k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",
"udp_state=""listen"
}0
참고용 container_network_udp,tcp_usage_total 확인정보
# HELP container_network_tcp_usage_total tcp connection usage statistic for container
# TYPE container_network_tcp_usage_total gauge
# (pause container 정보)
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="close"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closewait"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closing"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="established"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait1"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait2"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="lastack"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="listen"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synrecv"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synsent"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="timewait"} 0
# (mysql container 정보)
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="close"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closewait"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closing"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="established"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait1"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait2"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="lastack"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="listen"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synrecv"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synsent"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="timewait"} 0
# (tomcat container 정보)
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="close"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closewait"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="closing"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="established"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait1"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="finwait2"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="lastack"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="listen"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synrecv"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="synsent"} 0
container_network_tcp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",tcp_state="timewait"} 0
# HELP container_network_udp_usage_total udp connection usage statistic for container
# TYPE container_network_udp_usage_total gauge
# (pause container 정보)
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="dropped"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="listen"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="rxqueued"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubernetes_io_config_seen="2020-11-09T16:37:42.073870025+09:00",container_label_annotation_kubernetes_io_config_source="api",container_label_app="iksoon-pod-tomcat",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="6769c7766f",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773.scope",image="k8s.gcr.io/pause:3.2",name="k8s_POD_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="txqueued"} 0
# (mysql container 정보)
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="dropped"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="listen"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="rxqueued"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="191b1040",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":3306,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-mysql/0.log",container_label_io_kubernetes_container_name="iksoon-mysql",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-52aeba91ef97e86332a474faba308a1c17c32bdae1945c102c1598e755cd2d4f.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="txqueued"} 0
# (tomcat container 정보)
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="dropped"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="listen"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="rxqueued"} 0
container_network_udp_usage_total{container_label_annotation_checksum_config="",container_label_annotation_checksum_dashboards_json_config="",container_label_annotation_checksum_sc_dashboard_provider_config="",container_label_annotation_checksum_secret="",container_label_annotation_io_kubernetes_container_hash="b2e5d0a0",container_label_annotation_io_kubernetes_container_ports="[{\"containerPort\":8080,\"protocol\":\"TCP\"}]",container_label_annotation_io_kubernetes_container_restartCount="0",container_label_annotation_io_kubernetes_container_terminationMessagePath="/dev/termination-log",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="File",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="30",container_label_annotation_kubernetes_io_config_seen="",container_label_annotation_kubernetes_io_config_source="",container_label_app="",container_label_app_kubernetes_io_instance="",container_label_app_kubernetes_io_name="",container_label_chart="",container_label_component="",container_label_controller_revision_hash="",container_label_heritage="",container_label_io_kubernetes_container_logpath="/var/log/pods/default_iksoon-deployment-tomcat-6769c7766f-tzr28_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0/iksoon-tomcat/0.log",container_label_io_kubernetes_container_name="iksoon-tomcat",container_label_io_kubernetes_docker_type="container",container_label_io_kubernetes_pod_name="iksoon-deployment-tomcat-6769c7766f-tzr28",container_label_io_kubernetes_pod_namespace="default",container_label_io_kubernetes_pod_uid="ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0",container_label_io_kubernetes_sandbox_id="6abfb10014942b15ae2bc4303f17843354950d8b14917357dd36ac7c42460773",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_tier="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcde1cd_66d8_4bfd_96f4_f4b75cb679b0.slice/docker-4be0b17ccb819b527a1b0e904ffc4766d578ce494268bc3aa2fbb5197e6bd52a.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-tzr28_default_ffcde1cd-66d8-4bfd-96f4-f4b75cb679b0_0",udp_state="txqueued"} 0
3.3. Metrics 중 CPU 정보
한개의 deployment 안에 1개의 Pod를 생성했고
해당 Pod안에는 mysql, tomcat 2개의 container가 생성되어 있어서
각 tomcat, mysql Container에 해당하는 CPU정보가 나옴.
즉 Pod 내부에 있는 모든 Container에 대한 정보를 모두 확인 할 수 있음
CPU정보 중 예제로 몇가지만 가져와봄
# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
container_cpu_load_average_10s{container="iksoon-mysql",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-5f6f330a50042b61476c280f9b44df17554e77738571eb21e97c3e429a924341.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892019441
container_cpu_load_average_10s{container="iksoon-tomcat",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-c9d1625dff77b640ffc9331717c5207ee2e5e26c8ed9405a70ff497817822344.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0 1604892023537
# HELP container_cpu_system_seconds_total Cumulative system cpu time consumed in seconds.
# TYPE container_cpu_system_seconds_total counter
container_cpu_system_seconds_total{container="iksoon-mysql",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-5f6f330a50042b61476c280f9b44df17554e77738571eb21e97c3e429a924341.scope",image="sha256:f5e785cdc6faac016dec0d03b8eea7ede527271296933bfb269a328804561757",name="k8s_iksoon-mysql_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 1.24 1604892019441
container_cpu_system_seconds_total{container="iksoon-tomcat",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1c745_ef1c_41dc_87f0_9d43cd7253f3.slice/docker-c9d1625dff77b640ffc9331717c5207ee2e5e26c8ed9405a70ff497817822344.scope",image="sha256:9c1794f71b568f2004da0afaa77fa797eb187fc11fed9236e144cfab8a531a9c",name="k8s_iksoon-tomcat_iksoon-deployment-tomcat-6769c7766f-2dp62_default_51b1c745-ef1c-41dc-87f0-9d43cd7253f3_0",namespace="default",pod="iksoon-deployment-tomcat-6769c7766f-2dp62"} 0.49 1604892023537
4. cAdvisor /stats/summary 정보 조회 (요약 정보)
curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/stats/summary
[stat/summary 정보 중 cpu와 memory 정보만 조회하고 싶은 경우]
url에 사용자쿼리(querystring)를 사용해서
?only_cpu_and_memory=true를 붙여주면 됨.
curl -Ssk --header "Authorization: Bearer ${TOKEN}" https://10.0.2.4:10250/stats/summary?only_cpu_and_memory=true
[stat/summary로 확인할 수 있는 pod에 대한 정보는 아래와 같음]
pod의 CPU, Memory, Network, Volume, ephemeral-storage, process_stats 정보,
pod 내부에 생성된 container의 CPU, Memory, rootfs, logs 정보를 확인할 수 있음
(확인할 수 있는 Network 정보)
rxBytes
rxErrors
txBytes
txErrors
RX: Receive Data (수신 신호)
Tx : Transmit Data (송신 신호)
(rootfs란?)
rootfs 라고 하는 root 파일 시스템 layer이다.
Root 파일 시스템은 실제 OS 가 설치된다.
(예를 들어 Debian 이나 Ubuntu).
원래 리눅스에서는 root filesystem은 처음 mount될 때 read-only로 mount가 된 후
integrity check후 read-write으로 바뀐다.
하지만 Docker에서는 계속해서 read-only 모드이다.
Read-write 모드로 변환하지 않는 이유는 Docker는 union mount를 사용해서
read-only 파일 시스템들을 root 파일 시스템 위에 덮는 구조로 이루어져 있기 때문이다.
참고로 union mount는 여러게의 파일 시스템을 mount하되
하나의 파일 시스템만이 mount된것 처럼 사용하는 방법이다.
[stat/summary 정보]
"podRef": {
"name": "iksoon-deployment-tomcat-85567fbc4-h24xc",
"namespace": "default",
"uid": "beec608d-28c2-4c34-906c-7025b109556f"
},
"startTime": "2020-11-06T03:13:52Z",
"containers": [
{
"name": "iksoon-tomcat",
"startTime": "2020-11-06T03:13:54Z",
"cpu": {
"time": "2020-11-08T23:47:42Z",
"usageNanoCores": 1522200,
"usageCoreNanoSeconds": 321392726196
},
"memory": {
"time": "2020-11-08T23:47:42Z",
"usageBytes": 104632320,
"workingSetBytes": 104595456,
"rssBytes": 104583168,
"pageFaults": 64407,
"majorPageFaults": 0
},
"rootfs": {
"time": "2020-11-08T23:47:42Z",
"availableBytes": 3283169280,
"capacityBytes": 18238930944,
"usedBytes": 49152,
"inodesFree": 6412639,
"inodes": 6730584,
"inodesUsed": 13
},
"logs": {
"time": "2020-11-08T23:47:42Z",
"availableBytes": 3283169280,
"capacityBytes": 18238930944,
"usedBytes": 24576,
"inodesFree": 6412639,
"inodes": 6730584,
"inodesUsed": 317945
}
}
],
"cpu": {
"time": "2020-11-08T23:47:35Z",
"usageNanoCores": 1417477,
"usageCoreNanoSeconds": 321411574632
},
"memory": {
"time": "2020-11-08T23:47:35Z",
"usageBytes": 104673280,
"workingSetBytes": 104636416,
"rssBytes": 104624128,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-11-08T23:47:42Z",
"name": "eth0",
"rxBytes": 648,
"rxErrors": 0,
"txBytes": 300,
"txErrors": 0,
"interfaces": [
{
"name": "eth0",
"rxBytes": 648,
"rxErrors": 0,
"txBytes": 300,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-11-06T03:14:24Z",
"availableBytes": 1987076096,
"capacityBytes": 1987088384,
"usedBytes": 12288,
"inodesFree": 485120,
"inodes": 485129,
"inodesUsed": 9,
"name": "default-token-zr6hw"
}
],
"ephemeral-storage": {
"time": "2020-11-08T23:47:42Z",
"availableBytes": 3283169280,
"capacityBytes": 18238930944,
"usedBytes": 73728,
"inodesFree": 6412639,
"inodes": 6730584,
"inodesUsed": 13
},
"process_stats": {
"process_count": 0
}
},
kubernetes에서 사용하는 cAdvisor 분석 결과
1. Metrics-Server는 왜 kubelet의 cAdvisor정보를 /metrics로 모두 가져오는게 아니라 /stat/summary 정보만 가져올까?
Metrics-Server 코드를 분석해보면 /stat/summary request로 요약정보만 가져오고
그 중 cpu, memory 정보만 kubelet으로 보냄.
그 이유는 cAdvisor 에서 모든 정보를 가져오게되면 대량의 log로 인해 과부하가 발생할 수 있기때문.
(참고)
https://github.com/google/cadvisor/issues/2144
2. 그렇다면 사용자가 직접 cAdvisor배포하고 수집하는 자료 중 특정항목을 disabled 시키고 원하는 정보만 수집하면 되지 않나?
cAdvisor의 -disable_metrics flag를 사용해도 일부 metrics만 제외시킬 수 있음.
실제로 -disable_metrics flag에 disk, network, tcp, udp, sched, process, advtcp, hugetlb 옵션을 모두 적용하고
test 해본 결과 그래도 상당량의 metrics 정보를 생성하고 있음.
그로 인해 기본적인 metrics정보만 수집하더라도 과부하가 발생할 수 있음.
cAdvisor 공식 page에서 패치 note를 보면 verison이 증가할 때 마다
disable_metrics 옵션을 추가하고 있는데
왜 애초에 모든 metrics정보를 추가해주지 않는건지는 의문임.
(참고)
https://github.com/google/cadvisor/issues/2602
3. kubelet의 cAdvisor는 왜 tcp, udp 정보를 수집하지 않도록 설정했을까?
/stat/summary 결과를 보면 network 트래픽 사용량 정보만 조회할 수 있고
pod안에 생성된 container의 tcp, udp 정보는 조회할 수 없음
이 또한 과부하를 고려해서 설정해놓은것이라고 함.
결론
개발자가 원하는 정보만 추출하기 위해서는
직접 exporters 를 생성해야함.
ex) prometheus exporter
제 글을 복사할 시 출처를 명시해주세요.
글에 오타, 오류가 있다면 댓글로 알려주세요! 바로 수정하겠습니다!
[참고]
https://stackoverflow.com/questions/58369289/how-to-get-real-time-resource-usage-of-a-pod-in-k8s
https://discuss.elastic.co/t/kubernetes-module-kubelet-ssl/147770/3
https://labs.f-secure.com/blog/attacking-kubernetes-through-kubelet/
https://blog.ziki.cn/post/kubelet-api/
https://docs.netapp.com/us-en/cloudinsights/pdfs/pages/task_config_telegraf_kubernetes.pdf
[cAdvisor]
https://github.com/google/cadvisor
[Prometheus cAdvisor 참고]
https://engineering.linecorp.com/ko/blog/prometheus-container-kubernetes-cluster/
'클라우드 > Kubernetes' 카테고리의 다른 글
| 80. kubectl 축약어 정리 (0) | 2021.07.24 |
|---|---|
| 79. 오류 : inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory (0) | 2021.07.17 |
| 77. Kubernetes API Aggregation (1) | 2021.06.20 |
| 76. kubernetes NodePort Networking 분석 (kube-proxy : IPVS mode) (0) | 2021.05.07 |
| 75. kubernetes LoadBalancer Networking 분석 (kube-proxy : iptable mode) (0) | 2021.05.06 |