> For the complete documentation index, see [llms.txt](https://anida-huang.gitbook.io/cloud-communication/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://anida-huang.gitbook.io/cloud-communication/qi-mo/20210105-kubernetes-liu.md).
# 20210105 期末週
## 課堂資料
{% embed url="" %}
{% embed url="" %}
{% file src="/files/-MQGZZoT1EJDbKruh40U" %}
### Dockerfile + Flask
{% tabs %}
{% tab title="vm1" %}
```
docker login
```
```
cd
```
```
ls
```
```
mkdir mydocker
```
```
cd mydocker
```
> #### 撰寫 dockerfile
```
gedit Dockerfile main.py requirements.txt &
```
{% hint style="info" %}
#### Dockerfile
```
FROM python:3.7.2-stretch
WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt
CMD python main.py
```
* `FROM`：基底映像檔 (base image)
* `WORKDIR`：建立工作目錄
* `ADD`：複製指定的檔案、目錄或遠端檔案 URL，將其加入映像檔檔案系統中的指定位置
* `RUN`：每一個 RUN 指令會在現有映像檔之上加入新的一層，是在建立 (build) 映像檔的過程中會執行的指令
* `CMD`：一個 Dockerfile 中只能有一個 CMD 指令，CMD 則是在容器運行時所執行的指令
#### main.py
```
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
return 'Flask Dockerized'
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8888)
```
#### requirements.txt
```
Click==7.0
Flask==1.1.1
itsdangerous==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
Werkzeug==0.16.0
```
{% endhint %}
> #### 將 dockerfile 打包成 image
```
docker build -t mydocker:1.0.0 .
```
> #### 透過 image 產生隔離的執行環境 container
```
docker run -d -p 8081:8888 --name mydocker mydocker:1.0.0
```
* `-d`：背景執行
* `-p`：將主機 8888 port 與 container 的 80 port 綁定
* `–name`：為 container 命名
```
curl 127.0.0.1:8081
```
```
docker images | grep mydocker
```
```
docker tag [鏡像ID] xiaoji850312/mydocker:1.0.0
```
```
docker images | grep mydocker
```
```
docker push xiaoji850312/mydocker:1.0.0
```
```
kubectl create deployment mydocker --image=xiaoji850312/mydocker:1.0.0 --dry-run -o yaml > mydocker-deployment.yaml
```
```
gedit mydocker-deployment.yaml &
```
```
kubectl apply -f mydocker-deployment.yaml
```
```
kubectl get deployment
```
```
kubectl get pod
```
```
kubectl get pod -o wide
```
```
kubectl describe pod [podName]
```
```
kubectl get deployment
```
```
kubectl get pod
```
```
kubectl expose deployment mydocker --port=8888 --target-port=8888 --type=NodePort
```
```
kubectl get svc
```
> #### promethus 安裝
```
cd promethus
```
```
kubectl get svc
```
```
kubectl get svc -n ns-monitor
```
```
gedit prometheus.yaml node-exporter.yaml namespace.yaml
```
{% hint style="info" %}
#### prometheus.yaml
```
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs:
- get
- watch
- list
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- watch
- list
- nonResourceURLs: ["/metrics"]
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: ns-monitor
labels:
app: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: ns-monitor
roleRef:
kind: ClusterRole
name: prometheus
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-conf
namespace: ns-monitor
labels:
app: prometheus
data:
prometheus.yml: |-
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'grafana'
static_configs:
- targets:
- 'grafana-service.ns-monitor:3000'
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
# Default to scraping over https. If required, just disable this or change to
# `http`.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# .
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# If your node certificates are self-signed or use a different CA to the
# master CA, then disable certificate verification below. Note that
# certificate verification is an integral part of a secure infrastructure
# so this should only be disabled in a controlled environment. You can
# disable certificate verification by uncommenting the line below.
#
# insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# Keep only the default/kubernetes service endpoints for the https port. This
# will add targets for each API server which Kubernetes adds an endpoint to
# the default/kubernetes service.
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
# Scrape config for nodes (kubelet).
#
# Rather than connecting directly to the node, the scrape is proxied though the
# Kubernetes apiserver. This means it will work if Prometheus is running out of
# cluster, or can't connect to nodes for some other reason (e.g. because of
# firewalling).
- job_name: 'kubernetes-nodes'
# Default to scraping over https. If required, just disable this or change to
# `http`.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# .
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
# Scrape config for Kubelet cAdvisor.
#
# This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
# (those whose names begin with 'container_') have been removed from the
# Kubelet metrics endpoint. This job scrapes the cAdvisor endpoint to
# retrieve those metrics.
#
# In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
# HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
# in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
# the --cadvisor-port=0 Kubelet flag).
#
# This job is not necessary and should be removed in Kubernetes 1.6 and
# earlier versions, or it will cause the metrics to be scraped twice.
- job_name: 'kubernetes-cadvisor'
# Default to scraping over https. If required, just disable this or change to
# `http`.
scheme: https
# This TLS & bearer token file config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# .
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
# Scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
# via the following annotations:
#
# * `prometheus.io/scrape`: Only scrape services that have a value of `true`
# * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
# to set this to `https` & most likely set the `tls_config` of the scrape config.
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
# * `prometheus.io/port`: If the metrics are exposed on a different port to the
# service then set this appropriately.
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
# Example scrape config for probing services via the Blackbox Exporter.
#
# The relabeling allows the actual service scrape endpoint to be configured
# via the following annotations:
#
# * `prometheus.io/probe`: Only probe services that have a value of `true`
- job_name: 'kubernetes-services'
metrics_path: /probe
params:
module: [http_2xx]
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
action: keep
regex: true
- source_labels: [__address__]
target_label: __param_target
- target_label: __address__
replacement: blackbox-exporter.example.com:9115
- source_labels: [__param_target]
target_label: instance
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
target_label: kubernetes_name
# Example scrape config for probing ingresses via the Blackbox Exporter.
#
# The relabeling allows the actual ingress scrape endpoint to be configured
# via the following annotations:
#
# * `prometheus.io/probe`: Only probe services that have a value of `true`
- job_name: 'kubernetes-ingresses'
metrics_path: /probe
params:
module: [http_2xx]
kubernetes_sd_configs:
- role: ingress
relabel_configs:
- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
action: keep
regex: true
- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
regex: (.+);(.+);(.+)
replacement: ${1}://${2}${3}
target_label: __param_target
- target_label: __address__
replacement: blackbox-exporter.example.com:9115
- source_labels: [__param_target]
target_label: instance
- action: labelmap
regex: __meta_kubernetes_ingress_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_ingress_name]
target_label: kubernetes_name
# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
# * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
# pod's declared ports (default is a port-free target if none are declared).
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-rules
namespace: ns-monitor
labels:
app: prometheus
data:
cpu-usage.rule: |
groups:
- name: NodeCPUUsage
rules:
- alert: NodeCPUUsage
expr: (100 - (avg by (instance) (irate(node_cpu{name="node-exporter",mode="idle"}[5m])) * 100)) > 75
for: 2m
labels:
severity: "page"
annotations:
summary: "{{$labels.instance}}: High CPU usage detected"
description: "{{$labels.instance}}: CPU usage is above 75% (current value is: {{ $value }})"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: "prometheus-data-pv"
labels:
name: prometheus-data-pv
release: stable
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
nfs:
path: /nfs/prometheus/data
server: 192.168.8.129
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-data-pvc
namespace: ns-monitor
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
name: prometheus-data-pv
release: stable
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: prometheus
name: prometheus
namespace: ns-monitor
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
securityContext:
runAsUser: 0
containers:
- name: prometheus
image: prom/prometheus:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /prometheus
name: prometheus-data-volume
- mountPath: /etc/prometheus/prometheus.yml
name: prometheus-conf-volume
subPath: prometheus.yml
- mountPath: /etc/prometheus/rules
name: prometheus-rules-volume
ports:
- containerPort: 9090
protocol: TCP
volumes:
- name: prometheus-data-volume
persistentVolumeClaim:
claimName: prometheus-data-pvc
- name: prometheus-conf-volume
configMap:
name: prometheus-conf
- name: prometheus-rules-volume
configMap:
name: prometheus-rules
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
app: prometheus
name: prometheus-service
namespace: ns-monitor
spec:
ports:
- port: 9090
targetPort: 9090
selector:
app: prometheus
type: NodePort
```
{% endhint %}
```
mkdir /nfs/prometheus/data/ -p
```
```
vim /etc/exports
```
{% hint style="info" %}
#### /etc/exports
```
/data/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/var/nfsshare/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/nfs/prometheus/data 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
```
{% endhint %}
```
systemctl restart nfs
```
```
showmount -e localhost
```
```
ls
```
```
kubectl apply -f namespace.yaml
```
```
kubectl apply -f prometheus.yaml
```
```
kubectl apply -f node-exporter.yaml
```
```
kubectl get svc -n ns-monitor
```
```
vim /etc/exports
```
{% hint style="info" %}
#### /etc/exports
```
/data/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/var/nfsshare/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/nfs/prometheus/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
```
{% endhint %}
```
systemctl restart nfs
```
```
showmount -e localhost
```
```
kubectl apply -f prometheus.yaml
```
```
kubectl get svc -n ns-monitor
```
```
cd /nfs/prometheus/
```
```
ls
```
```
cd data/
```
```
ls
```
```
cat /proc/cpuinfo
```
```
cd promethus
```
```
top
```
```
kubectl get svc -n ns-monitor
```
```
ls
```
```
gedit grafana.yaml
```
{% hint style="info" %}
#### grafana.yaml
```
apiVersion: v1
kind: PersistentVolume
metadata:
name: "grafana-data-pv"
labels:
name: grafana-data-pv
release: stable
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
nfs:
path: /nfs/grafana/data
server: 192.168.8.129
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-data-pvc
namespace: ns-monitor
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
name: grafana-data-pv
release: stable
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: grafana
name: grafana
namespace: ns-monitor
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
securityContext:
runAsUser: 0
containers:
- name: grafana
image: grafana/grafana
imagePullPolicy: IfNotPresent
env:
# The following env variables set up basic auth twith the default admin user and admin password.
- name: GF_AUTH_BASIC_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "false"
# - name: GF_AUTH_ANONYMOUS_ORG_ROLE
# value: Admin
# does not really work, because of template variables in exported dashboards:
# - name: GF_DASHBOARDS_JSON_ENABLED
# value: "true"
readinessProbe:
httpGet:
path: /login
port: 3000
volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-data-volume
ports:
- containerPort: 3000
protocol: TCP
volumes:
- name: grafana-data-volume
persistentVolumeClaim:
claimName: grafana-data-pvc
---
kind: Service
apiVersion: v1
metadata:
labels:
app: grafana
name: grafana-service
namespace: ns-monitor
spec:
ports:
- port: 3000
targetPort: 3000
selector:
app: grafana
type: NodePort
```
{% endhint %}
```
kubectl get svc -n ns-monitor
```
```
mkdir /nfs/grafana/data -p
```
```
vim /etc/exports
```
{% hint style="info" %}
#### /etc/exports
```
/data/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/var/nfsshare/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/nfs/prometheus/ 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
/nfs/grafana/data 192.168.8.0/24(rw,sync,no_root_squash,no_all_squash)
```
{% endhint %}
```
systemctl restart rpcbind
```
```
systemctl restart nfs
```
```
kubectl apply -f grafana.yaml
```
```
kubectl get pod -n ns-monitor
```
```
kubectl get svc -n ns-monitor
```
{% hint style="info" %}
change password `admin` to `root1234`
{% endhint %}
{% endtab %}
{% tab title="vm2" %}
```
docker login
```
```
docker images | grep xiaoji850312
```
{% endtab %}
{% tab title="vm3" %}
```
docker login
```
```
docker images | grep xiaoji850312
```
{% endtab %}
{% endtabs %}
## 課堂練習
### 鳶尾花
{% tabs %}
{% tab title="vm1" %}
```
cd harbor
```
### 執行安裝程序
```
sh install.sh
```
```
docker tag 514 xiaoji850312/iris:1.0
```
```
docker push xiaoji850312/iris:1.0
```
```
docker pull xiaoji850312/iris:1.0
```
```
cd
```
```
ls
```
```
cd iris
```
> #### 撰寫 dockerfile
```
gedit Dockerfile main.py requirements.txt &
```
{% hint style="info" %}
#### Dockerfile
```
FROM python:3.7.2-stretch
WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt
CMD python main.py
```
#### main.py
```
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
return 'Flask Dockerized'
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8888)
```
#### requirements.txt
```
Click==7.0
Flask==1.1.1
itsdangerous==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
Werkzeug==0.16.0
```
{% endhint %}
```
pip install --upgrade pip
```
```
docker images | grep iris
```
```
kubectl create deployment iris --image=xiaoji850312/iris:1.0 --dry-run -o yaml > iris-deployment.yaml
```
```
gedit iris-deployment.yaml &
```
```
kubectl apply -f iris-deployment.yaml
```
```
kubectl get deployment
```
```
kubectl get pod
```
```
kubectl get pod -o wide
```
```
kubectl expose deployment iris --port=8888 --target-port=8888 --type=NodePort
```
```
kubectl get svc
```
{% endtab %}
{% tab title="vm2" %}
```
docker login 192.168.8.129
```
```
docker pull xiaoji850312/iris:1.0
```
```
docker tag 514 192.168.8.129/xiaoji850312/iris:1.0
```
```
docker push 192.168.8.129/xiaoji850312/iris:1.0
```
{% endtab %}
{% tab title="Harbor" %}
{% hint style="info" %}
#### 新增倉庫
{% endhint %}
{% endtab %}
{% endtabs %}