1. 手工部署K8s集群从零到生产级的完整指南在容器编排领域Kubernetes简称K8s已经成为事实上的标准。虽然现在有各种托管服务和一键部署工具但手工部署仍然是理解K8s架构最有效的方式。我经历过从单节点测试到生产级集群的完整部署过程今天就把这些实战经验系统化地分享出来。手工部署K8s集群不仅能让你深入理解各个组件的工作原理还能在出现问题时快速定位和解决。不同于使用kubeadm等工具的半自动化部署纯手工方式需要你亲自配置每一个组件包括etcd、API Server、Controller Manager、Scheduler、kubelet和kube-proxy等。这种方式虽然复杂但学到的知识是实实在在的。2. 环境准备与基础配置2.1 硬件资源规划生产环境的K8s集群通常需要至少三个节点1个master和2个worker但为了演示目的我们可以从两个节点开始。以下是我的推荐配置节点类型CPU内存磁盘网络Master2核4GB40GB千兆Worker4核8GB100GB千兆注意实际生产环境需要根据工作负载调整资源配置特别是Worker节点的CPU和内存2.2 操作系统准备我推荐使用CentOS 7或Ubuntu 18.04/20.04 LTS作为基础系统。以下是所有节点都需要执行的基础配置# 关闭SELinux生产环境需谨慎 setenforce 0 sed -i s/^SELINUXenforcing$/SELINUXpermissive/ /etc/selinux/config # 关闭swap swapoff -a sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab # 设置主机名 hostnamectl set-hostname master-node # 在master节点执行 hostnamectl set-hostname worker-node1 # 在worker节点执行 # 设置hosts解析 cat EOF /etc/hosts 192.168.1.100 master-node 192.168.1.101 worker-node1 EOF # 允许iptables检查桥接流量 cat EOF /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 EOF sysctl --system3. 核心组件安装与配置3.1 安装容器运行时K8s支持多种容器运行时我推荐使用containerd# 安装containerd yum install -y containerd.io # CentOS apt-get install -y containerd # Ubuntu # 配置containerd mkdir -p /etc/containerd containerd config default /etc/containerd/config.toml sed -i s/SystemdCgroup false/SystemdCgroup true/ /etc/containerd/config.toml # 启动containerd systemctl restart containerd systemctl enable containerd3.2 安装kubeadm、kubelet和kubectl虽然我们是手工部署但仍然需要这些基础工具# 添加K8s源 cat EOF /etc/yum.repos.d/kubernetes.repo [kubernetes] nameKubernetes baseurlhttps://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF # 安装组件 yum install -y kubelet kubeadm kubectl --disableexcludeskubernetes # 设置kubelet自启 systemctl enable --now kubelet4. 手工部署控制平面组件4.1 部署etcd集群etcd是K8s的分布式键值存储我们先在master节点上部署单节点etcd# 下载etcd二进制文件 ETCD_VERv3.4.13 wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz tar -xvf etcd-${ETCD_VER}-linux-amd64.tar.gz mv etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/ # 创建etcd配置文件 mkdir -p /etc/etcd /var/lib/etcd cat EOF /etc/etcd/etcd.conf name: master-node># 创建必要的目录和证书简化版生产环境需要正式CA mkdir -p /etc/kubernetes/pki openssl genrsa -out /etc/kubernetes/pki/ca.key 2048 openssl req -x509 -new -nodes -key /etc/kubernetes/pki/ca.key -days 3650 -out /etc/kubernetes/pki/ca.crt -subj /CNkube-ca # 创建kube-apiserver配置文件 cat EOF /etc/kubernetes/apiserver.conf apiVersion: v1 kind: Config clusters: - cluster: certificate-authority: /etc/kubernetes/pki/ca.crt server: https://192.168.1.100:6443 name: kubernetes contexts: - context: cluster: kubernetes user: system:admin name: system:adminkubernetes current-context: system:adminkubernetes users: - name: system:admin user: client-certificate: /etc/kubernetes/pki/admin.crt client-key: /etc/kubernetes/pki/admin.key EOF # 创建systemd服务 cat EOF /etc/systemd/system/kube-apiserver.service [Unit] DescriptionKubernetes API Server Documentationhttps://kubernetes.io/docs/concepts/overview/ [Service] ExecStart/usr/local/bin/kube-apiserver \\ --advertise-address192.168.1.100 \\ --allow-privilegedtrue \\ --authorization-modeNode,RBAC \\ --client-ca-file/etc/kubernetes/pki/ca.crt \\ --enable-admission-pluginsNodeRestriction \\ --enable-bootstrap-token-authtrue \\ --etcd-servershttp://127.0.0.1:2379 \\ --kubelet-client-certificate/etc/kubernetes/pki/apiserver-kubelet-client.crt \\ --kubelet-client-key/etc/kubernetes/pki/apiserver-kubelet-client.key \\ --service-account-key-file/etc/kubernetes/pki/sa.pub \\ --service-account-signing-key-file/etc/kubernetes/pki/sa.key \\ --service-account-issuerhttps://kubernetes.default.svc.cluster.local \\ --service-cluster-ip-range10.96.0.0/12 \\ --tls-cert-file/etc/kubernetes/pki/apiserver.crt \\ --tls-private-key-file/etc/kubernetes/pki/apiserver.key \\ --proxy-client-cert-file/etc/kubernetes/pki/front-proxy-client.crt \\ --proxy-client-key-file/etc/kubernetes/pki/front-proxy-client.key \\ --requestheader-allowed-namesfront-proxy-client \\ --requestheader-client-ca-file/etc/kubernetes/pki/front-proxy-ca.crt \\ --requestheader-extra-headers-prefixX-Remote-Extra- \\ --requestheader-group-headersX-Remote-Group \\ --requestheader-username-headersX-Remote-User \\ --secure-port6443 Restarton-failure RestartSec5 [Install] WantedBymulti-user.target EOF # 启动API Server systemctl daemon-reload systemctl enable kube-apiserver systemctl start kube-apiserver5. 部署Worker节点组件5.1 配置kubelet在Worker节点上配置kubelet# 创建kubelet配置 cat EOF /etc/kubernetes/kubelet.conf apiVersion: v1 clusters: - cluster: certificate-authority: /etc/kubernetes/pki/ca.crt server: https://192.168.1.100:6443 name: kubernetes contexts: - context: cluster: kubernetes user: system:node:worker-node1 name: system:node:worker-node1kubernetes current-context: system:node:worker-node1kubernetes kind: Config preferences: {} users: - name: system:node:worker-node1 user: client-certificate: /etc/kubernetes/pki/kubelet.crt client-key: /etc/kubernetes/pki/kubelet.key EOF # 创建kubelet systemd服务 cat EOF /etc/systemd/system/kubelet.service [Unit] DescriptionKubernetes Kubelet Documentationhttps://kubernetes.io/docs/concepts/overview/ [Service] ExecStart/usr/local/bin/kubelet \\ --config/var/lib/kubelet/config.yaml \\ --container-runtimeremote \\ --container-runtime-endpointunix:///run/containerd/containerd.sock \\ --kubeconfig/etc/kubernetes/kubelet.conf \\ --network-plugincni \\ --register-nodetrue \\ --v2 Restarton-failure RestartSec5 [Install] WantedBymulti-user.target EOF # 启动kubelet systemctl daemon-reload systemctl enable kubelet systemctl start kubelet5.2 部署kube-proxy# 创建kube-proxy配置 cat EOF /etc/kubernetes/kube-proxy.conf apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration clientConnection: kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig mode: iptables clusterCIDR: 10.244.0.0/16 EOF # 创建systemd服务 cat EOF /etc/systemd/system/kube-proxy.service [Unit] DescriptionKubernetes Kube Proxy Documentationhttps://kubernetes.io/docs/concepts/overview/ [Service] ExecStart/usr/local/bin/kube-proxy \\ --config/etc/kubernetes/kube-proxy.conf \\ --v2 Restarton-failure RestartSec5 [Install] WantedBymulti-user.target EOF # 启动kube-proxy systemctl daemon-reload systemctl enable kube-proxy systemctl start kube-proxy6. 网络插件部署K8s需要网络插件实现Pod间通信我推荐使用Calico# 在master节点上执行 kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml # 验证网络插件 kubectl get pods -n kube-system -w7. 集群验证与测试7.1 检查节点状态kubectl get nodes应该看到master和worker节点都处于Ready状态。7.2 部署测试应用kubectl create deployment nginx --imagenginx kubectl expose deployment nginx --port80 --typeNodePort kubectl get svc访问Worker节点的NodePort端口应该能看到Nginx欢迎页面。8. 生产环境进阶配置8.1 高可用etcd集群生产环境需要至少3个etcd节点组成集群# 在每个etcd节点上修改配置 initial-cluster: etcd1http://192.168.1.100:2380,etcd2http://192.168.1.101:2380,etcd3http://192.168.1.102:2380 initial-cluster-state: new listen-peer-urls: http://0.0.0.0:2380 listen-client-urls: http://0.0.0.0:2379 advertise-client-urls: http://当前节点IP:23798.2 API Server负载均衡使用HAProxy或Nginx为API Server做负载均衡# HAProxy配置示例 frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-api-servers backend k8s-api-servers mode tcp option tcp-check balance roundrobin server master1 192.168.1.100:6443 check server master2 192.168.1.101:6443 check server master3 192.168.1.102:6443 check9. 常见问题排查9.1 Pod一直处于Pending状态可能原因资源不足没有满足条件的节点PV/PVC问题排查步骤kubectl describe pod pod-name kubectl get events --sort-by.metadata.creationTimestamp9.2 节点NotReady可能原因kubelet未正常运行网络插件问题磁盘空间不足排查步骤journalctl -u kubelet -f # 查看kubelet日志 systemctl status kubelet # 检查kubelet状态 df -h # 检查磁盘空间9.3 网络不通可能原因CNI插件未正确安装网络策略阻止节点间防火墙排查步骤kubectl get pods -n kube-system # 检查网络插件Pod iptables -L -n -v # 检查iptables规则 ping 目标Pod IP # 测试基础连通性10. 性能优化建议etcd优化使用SSD磁盘适当增加etcd的heartbeat-interval和election-timeout定期进行碎片整理API Server优化启用API优先级和公平性APF适当增加--max-requests-inflight和--max-mutating-requests-inflightkubelet优化设置--image-gc-high-threshold和--image-gc-low-threshold调整--kube-api-qps和--kube-api-burst网络优化选择适合的网络插件Calico性能优于Flannel考虑使用IPVS模式替代iptables手工部署K8s集群虽然复杂但能让你真正理解K8s的运作机制。在实际操作中我发现很多问题都源于对基础原理的不了解。通过这种方式部署后再使用kubeadm或其他工具时会更加得心应手。对于生产环境建议在完全掌握手工部署后再考虑使用自动化工具提高效率。