一、Kubernetes监控概述Kubernetes监控是保障集群稳定性的关键监控层次基础设施层Node、Pod应用层业务指标集群层K8s组件二、Prometheus架构1. 架构图┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Exporter │ │ Exporter │ │ Exporter │ │ (Node) │ │ (Kube) │ │ (App) │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ └───────────────────┼───────────────────┘ │ ┌──────┴──────┐ │ Prometheus │ │ Server │ └──────┬──────┘ │ ┌─────────────────┼─────────────────┐ │ │ │ ┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐ │ Alertmanager│ │ Grafana │ │ Storage │ └─────────────┘ └─────────────┘ └─────────────┘2. 核心概念指标类型Counter单调递增Gauge可增可减Histogram直方图Summary分位数三、部署Prometheus1. Operator部署# 安装Prometheus Operatorkubectl create namespace monitoring helm repoaddprometheus-community https://prometheus-community.github.io/helm-charts helminstallprometheus prometheus-community/kube-prometheus-stack2. ServiceMonitorapiVersion:monitoring.coreos.com/v1kind:ServiceMonitormetadata:name:myapp-monitornamespace:monitoringspec:selector:matchLabels:app:myappendpoints:-port:httppath:/metricsinterval:15s3. PrometheusRuleapiVersion:monitoring.coreos.com/v1kind:PrometheusRulemetadata:name:myapp-alertsspec:groups:-name:myapprules:-alert:HighMemoryUsageexpr:container_memory_usage_bytes1e9for:5mlabels:severity:warningannotations:summary:内存使用率过高四、Grafana配置1. 数据源配置apiVersion:v1kind:ConfigMapmetadata:name:grafana-datasourcesnamespace:monitoringdata:datasources.yaml:|apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus-server.monitoring.svc:9090 isDefault: true2. Dashboard{dashboard:{title:Kubernetes Cluster,panels:[{title:CPU Usage,type:graph,targets:[{expr:rate(container_cpu_usage_seconds_total[5m])}]}]}}3. 常用查询# Node资源使用 node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 # Pod资源使用 sum(rate(container_cpu_usage_seconds_total{pod!}[5m])) by (pod) # K8s组件状态 up{jobkube-apiserver} # 存储使用 kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes五、告警配置1. AlertManager配置apiVersion:monitoring.coreos.com/v1alpha1kind:AlertmanagerConfigmetadata:name:alert-configspec:route:groupBy:[namespace]groupWait:30sgroupInterval:5mrepeatInterval:4hreceiver:defaultreceivers:-name:defaultwebhookConfigs:-url:http://notification-service:8080/webhook2. 告警规则groups:-name:k8srules:-alert:PodNotReadyexpr:kube_pod_status_ready{conditiontrue} 0for:5mlabels:severity:criticalannotations:summary:Pod {{ $labels.namespace }}/{{ $labels.pod }} not ready-alert:HighMemoryexpr:sum(container_memory_working_set_bytes) by (pod) / 1024/10241024for:5mlabels:severity:warning六、总结Kubernetes监控最佳实践Prometheus指标采集和存储Grafana可视化展示AlertManager告警管理ServiceMonitor自动服务发现个人观点仅供参考