声明写这个的主要目的想做个记录以便日后再折腾时查阅。内容适用于本人的使用环境不同环境仅作参考。文中出现的内容来自网络和本人浅薄的认知如果有侵犯他人内容请私信。环境说明masterCentOS 7.9hostnamemaster.test.k8s.comIP192.168.103.61Docker version 26.1.4Kubernetes version 1.28.2node1CentOS 7.9hostnamenode1.test.k8s.comIP192.168.103.62Docker version 26.1.4Kubernetes version 1.28.2环境初始化master和node1都要执行修改主机名永久修改hostnamectl命令hostnamectl set-hostname[your-hostname]重启或者重启网络# 重启reboot# 重启网络重启网络之后SSH终端显示暂时不会改变需要重新连接systemctl restart network临时修改hostname命令hostname[your-hostname]即时生效添加主机名解析vi/etc/hosts192.168.103.61 master.test.k8s.com192.168.103.62 node1.test.k8s.com关闭防火墙和selinux关闭防火墙并禁止自启动systemctl stop firewalld systemctl disable firewalld关闭selinuxvi/etc/selinux/configSELINUXdisabed禁用swap分区注释掉swap那一行vi/etc/fstab#/dev/mapper/centos-swap swap swap defaults 0 0关闭swapswapoff-a替换yum源阿里云cp/etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bakcurl-o/etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all yum makecache修改网络配置允许 iptables 检查桥接流量(所有节点)catEOF|sudotee/etc/modules-load.d/k8s.confbr_netfilter EOFcatEOF|sudotee/etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 EOFsysctl--system重启服务器reboot安装dockermaster和node1有docker的可以跳过安装步骤或者卸载之前的docker重新安装卸载dockersudoyum removedocker\docker-client\docker-client-latest\docker-common\docker-latest\docker-latest-logrotate\docker-logrotate\docker-engine安装yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yuminstall-ydocker-ce docker-ce-cli containerd.io验证dockerversion设置开机自启systemctl startdockersystemctlenabledocker更换docker镜像源docker国内镜像源配置及走代理设置_docker镜像代理-CSDN博客docker镜像加速源配置目前可用镜像源列举_docker可用的镜像源-CSDN博客安装Kubernetes组件master和node1参考阿里云Kubernetes安装步骤kubernetes镜像_kubernetes下载地址_kubernetes安装教程-阿里巴巴开源镜像站 (aliyun.com)catEOF/etc/yum.repos.d/kubernetes.repo[kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck0 repo_gpgcheck0 gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF安装kubeadm、kubelet和kubectl三大组件yuminstall-ykubelet kubeadm kubectl设置开机自启systemctl start kubelet systemctlenablekubelet验证kubectl version kubelet--versionkubeadm version安装cri-docker组件在 k8s v1.24 及更早版本中我们使用 docker 作为容器引擎在 k8s 上使用时依赖一个 dockershim 的内置 k8s 组件k8s v1.24发行版中将dockershim 组件给移除了取而代之的就是 cri-dockerd当然还有其它容器接口简单讲 CRI 就是容器运行时接口Container Runtime InterfaceCRI也就是说 cri-dockerd 就是以 docker 作为容器引擎而提供的容器运行时接口如果我们想要用 docker 作为 k8s 的容器运行引擎我们需要先部署好 cri-dockerd用 cri-dockerd 来与 kubelet 交互然后再由 cri-dockerd 和 docker api 交互使我们在 k8s 能够正常使用 docker 作为容器引擎。# 下载wgethttps://github.com/Mirantis/cri-dockerd/releases/download/v0.3.15/cri-dockerd-0.3.15.amd64.tgz# 解压cri-dockertarxvf cri-dockerd-*.amd64.tgzcp-rcri-dockerd/ /usr/bin/chmodx /usr/bin/cri-dockerd/cri-dockerd写入启动cri-docker配置文件# https://github.com/Mirantis/cri-dockerd/blob/master/packaging/systemd/cri-docker.service# 写入启动cri-docker配置文件cat/usr/lib/systemd/system/cri-docker.serviceEOF [Unit] DescriptionCRI Interface for Docker Application Container Engine Documentationhttps://docs.mirantis.com Afternetwork-online.target firewalld.service docker.service Wantsnetwork-online.target Requirescri-docker.socket [Service] Typenotify ExecStart/usr/bin/cri-dockerd/cri-dockerd --network-plugincni --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.9 ExecReload/bin/kill -s HUP$MAINPIDTimeoutSec0 RestartSec2 Restartalways StartLimitBurst3 StartLimitInterval60s LimitNOFILEinfinity LimitNPROCinfinity LimitCOREinfinity TasksMaxinfinity Delegateyes KillModeprocess [Install] WantedBymulti-user.target EOF写入cri-docker的socket配置文件# 写入cri-docker的socket配置文件cat/usr/lib/systemd/system/cri-docker.socketEOF [Unit] DescriptionCRI Docker Socket for the API PartOfcri-docker.service [Socket] ListenStream%t/cri-dockerd.sock SocketMode0660 SocketUserroot SocketGroupdocker [Install] WantedBysockets.target EOF刷新配置并启用cri-docker# 刷新配置systemctl daemon-reload# 设置开机自启systemctlenablecri-docker.service# 查看当前状态systemctl status cri-docker.service安装集群所需镜像查看所需镜像kubeadm config images list registry.k8s.io/kube-apiserver:v1.28.14 registry.k8s.io/kube-controller-manager:v1.28.14 registry.k8s.io/kube-scheduler:v1.28.14 registry.k8s.io/kube-proxy:v1.28.14 registry.k8s.io/pause:3.9 registry.k8s.io/etcd:3.5.9-0 registry.k8s.io/coredns/coredns:v1.10.1拉取镜像viimanges_k8s.shimages(kube-apiserver:v1.28.14 kube-controller-manager:v1.28.14 kube-scheduler:v1.28.14 kube-proxy:v1.28.14 pause:3.9 etcd:3.5.9-0 coredns:v1.10.1)forimageNamein${images[]};dodockerpull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageNamedockertag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageNamek8s.gcr.io/$imageNamedockerrmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageNamedonechmodx imanges_k8s.sh ./imanges_k8s.sh创建集群初始化master节点kubeadm init\--apiserver-advertise-address192.168.103.61\--control-plane-endpointmaster.test.k8s.com\--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers\--kubernetes-version v1.28.2\--service-cidr10.96.0.0/16\--pod-network-cidr10.244.0.0/16\--cri-socket unix:///var/run/cri-dockerd.sock说明apiserver-advertise-addressmaster主机的IP地址–control-plane-endpoint控制端master的主机名image-repository镜像地址由于国外地址无法访问此处使用阿里云仓库地址kubernetes-versionKubernetes版本使用kubectl version查看pod-network-cidrKubernetes内部pod节点之间网络可以使用的IP段不能和service-cidr一样可直接套用这个地方的参数貌似和IP也有点关系192.168.0.0/16和10.0.0.0/16输入的参数会有不同service-cidr直接套用–cri-socket unix:///var/run/cri-dockerd.sock由于当前版本不再默认支持docker,使用docker需要在命令后面加入参数–cri-socket unix:///var/run/cri-dockerd.sock。如果init失败可以通过kubeadm命令重置kubeadm reset --cri-socket unix:///var/run/cri-dockerd.sock命令执行完成之后可以看到操作提示Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir-p$HOME/.kubesudocp-i/etc/kubernetes/admin.conf$HOME/.kube/configsudochown$(id-u):$(id-g)$HOME/.kube/config Alternatively,ifyou are the root user, you can run:exportKUBECONFIG/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Runkubectl apply -f [podnetwork].yamlwith one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can nowjoinany number of control-plane nodes by copying certificate authorities andserviceaccount keys on eachnodeandthenrunning the following as root: kubeadmjoinmaster.test.k8s.com:6443--tokensyukz2.nyd4hppm29wwyu32\--discovery-token-ca-cert-hash sha256:63297b6049e34918715f512b7f434b02366377c73a3715864cf2866ffbcc318d\--control-plane Then you canjoinany number of worker nodes by running the following on each as root: kubeadmjoinmaster.test.k8s.com:6443--tokensyukz2.nyd4hppm29wwyu32\--discovery-token-ca-cert-hash sha256:63297b6049e34918715f512b7f434b02366377c73a3715864cf2866ffbcc318d在master中执行mkdir-p$HOME/.kubesudocp-i/etc/kubernetes/admin.conf$HOME/.kube/configsudochown$(id-u):$(id-g)$HOME/.kube/configexportKUBECONFIG/etc/kubernetes/admin.conf查看节点kubectl getnodeNAME STATUS ROLES AGE VERSION master.test.k8s.com NotReady control-plane 2m39s v1.28.2添加node节点在node1中执行初始化成功后的最后一段代码由于当前版本不再默认支持docker,如果服务器使用的docker需要在命令后面加入参数–cri-socket unix:///var/run/cri-dockerd.sock。kubeadmjoinmaster.test.k8s.com:6443--tokensyukz2.nyd4hppm29wwyu32\--discovery-token-ca-cert-hash sha256:63297b6049e34918715f512b7f434b02366377c73a3715864cf2866ffbcc318d\--cri-socket unix:///var/run/cri-dockerd.sock添加完成之后再次查看节点kubectl getnodeNAME STATUS ROLES AGE VERSION master.test.k8s.com NotReady control-plane 17m v1.28.2 node1.test.k8s.com NotReadynone4m20s v1.28.2可以看到节点已经成功添加上来了通过查看节点可以看到节点状态为NotReady这是因为没有安装网络插件无法进行通信需要部署一个 pod 网络插件安装 Pod 网络是 Pod 之间进行通信的必要条件k8s 支持众多网络方案这里选用 calico 方案。添加网络插件下载calico配置文件# 下载calico.ymlcurlhttps://docs.tigera.io/archive/v3.25/manifests/calico.yaml-Ocalico 默认会找 eth0 网卡如果当前机器网卡不是这个名字可能会无法启动需要手动修改calico.yaml配置文件vicalico.yaml# IP automatic detection- name: IP_AUTODETECTION_METHOD value:interfaceens*,eth0PS1这里加eth0是因为在看资料的时候看到说网卡名称为eth0的可以不用配置所以我直接把网卡名称改了 Linux之手把手教会修改网卡名称_linux修改网卡名称-CSDN博客但是后来发现还是需要修改配置文件的所以添加网卡名称就行了不用修改PS2需要把master节点和node节点的网卡名称都添加到这里修改镜像地址sed-is#calico/cni:master#swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/cni:master#calico.yamlsed-is#calico/node:master#swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/node:master#calico.yamlsed-is#calico/kube-controllers:master#swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/kube-controllers:master#calico.yaml应用配置kubectl apply-fcalico.yaml查看节点状态kubectl getnode查看podkubectl get pod-A错误排查因为每个人的环境不一样很难保证每个人都能顺利的部署而且也无法保证都是一样的错误。授人以鱼不如授人以渔可以按照下面的方法进行排查查看nodes状态kubectl getnode查看pods状态kubectl get pods-ANAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-5f9c988956-kb8b70/1 Pending017s kube-system calico-node-n7x2n0/1 Init:0/3017s kube-system calico-node-znd4l0/1 Init:0/3017s kube-system coredns-6554b8b87f-r7nq20/1 Pending018h kube-system coredns-6554b8b87f-zps2c0/1 Pending018h查看pods详细消息kubectl describe pod -n [NAMESPACE] [NAME]kubectl describe pod-nkube-system calico-node-n7x2n Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m33s default-scheduler Successfully assigned kube-system/calico-node-8mdff to node1.test.k8s.com Warning Failed 62s kubelet Failed to pull imagedocker.io/calico/cni:master:Error response from daemon: Gethttps://registry-1.docker.io/v2/:net/http: request canceledwhilewaitingforconnection(Client.Timeout exceededwhileawaiting headers)Warning Failed 62s kubelet Error: ErrImagePull Normal BackOff 61s kubelet Back-off pulling imagedocker.io/calico/cni:masterWarning Failed 61s kubelet Error: ImagePullBackOff Normal Pulling 48s(x2 over 2m33s)kubelet Pulling imagedocker.io/calico/cni:master查看日志kubectl logs -f -n [NAMESPACE] [NAME]kubectl logs-f-nkube-system calico-node-n7x2n最后通过报错消息使用百度进行精准查询部署服务安装k8s Dashboard可视化界面编辑配置文件virecommended.yamlapiVersion: v1 kind: Namespace metadata: name: kubernetes-dashboard --- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard --- apiVersion: v1 kind: ServiceAccount metadata: name: dashboard-admin namespace: kubernetes-dashboard --- kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port:443targetPort:8443nodePort:31443selector: k8s-app: kubernetes-dashboard --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboard type: Opaque data: csrf:--- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboard type: Opaque --- kind: ConfigMap apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: - apiGroups:[]resources:[secrets]resourceNames:[kubernetes-dashboard-key-holder,kubernetes-dashboard-certs,kubernetes-dashboard-csrf]verbs:[get,update,delete]- apiGroups:[]resources:[configmaps]resourceNames:[kubernetes-dashboard-settings]verbs:[get,update]- apiGroups:[]resources:[services]resourceNames:[heapster,dashboard-metrics-scraper]verbs:[proxy]- apiGroups:[]resources:[services/proxy]resourceNames:[heapster,http:heapster:,https:heapster:,dashboard-metrics-scraper,http:dashboard-metrics-scraper]verbs:[get]--- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: - apiGroups:[metrics.k8s.io]resources:[pods,nodes]verbs:[get,list,watch]--- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: dashboard-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: dashboard-admin namespace: kubernetes-dashboard --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas:1revisionHistoryLimit:10selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard# image: kubernetesui/dashboard:v2.0.0-rc7image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard:v2.7.0 imagePullPolicy: Always ports: - containerPort:8443protocol: TCP args: - --auto-generate-certificates ---namespacekubernetes-dashboard volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port:8443initialDelaySeconds:30timeoutSeconds:30securityContext: allowPrivilegeEscalation:falsereadOnlyRootFilesystem:truerunAsUser:1001runAsGroup:2001volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir:{}serviceAccountName: kubernetes-dashboard nodeSelector:beta.kubernetes.io/os:linux tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule --- kind: Service apiVersion: v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: ports: - port:8000targetPort:8000selector: k8s-app: dashboard-metrics-scraper --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas:1revisionHistoryLimit:10selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper annotations: seccomp.security.alpha.kubernetes.io/pod:runtime/defaultspec: containers: - name: dashboard-metrics-scraper# image: kubernetesui/metrics-scraper:v1.0.4image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/metrics-scraper:v1.0.8 ports: - containerPort:8000protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port:8000initialDelaySeconds:30timeoutSeconds:30volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation:falsereadOnlyRootFilesystem:truerunAsUser:1001runAsGroup:2001serviceAccountName: kubernetes-dashboard nodeSelector:beta.kubernetes.io/os:linux tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir:{}编辑完成执行配置kubectl apply-frecommended.yaml查看podkubectl get pods-nkubernetes-dashboard NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-85d6cb547f-jr62p1/1 Running071s kubernetes-dashboard-7f57989cfb-rczkf1/1 Running071s查看服务# 查看全部 -Akubectl get svc-nkubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE service/dashboard-metrics-scraper ClusterIP10.96.159.64none8000/TCP 4m16s service/kubernetes-dashboard NodePort10.96.163.37none443:31443/TCP 4m16s访问端口是31443内部端口是443在浏览器访问节点输入 https://集群中任意节点IP:31443https://192.168.103.61:31443/使用K8s Dashboard由于登录dashboard需要使用token先创建token创建并编辑配置文件vidash-token.yamlkind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: admin annotations: rbac.authorization.kubernetes.io/autoupdate:trueroleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: admin namespace: kubernetes-dashboard --- apiVersion: v1 kind: ServiceAccount metadata: name: admin namespace: kubernetes-dashboard labels: kubernetes.io/cluster-service:trueaddonmanager.kubernetes.io/mode: Reconcile执行配置文件kubectl apply-fdash-token.yaml获取令牌kubectl create token admin--namespacekubernetes-dashboardeyJhbGciOiJSUzI1NiIsImtpZCI6IllLU0g0azRUdVNXeVN3aWc4MWQ5dkJlNHYtT0ExU25jUmFCOFJ4UXJqck0ifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzI4OTg2ODI1LCJpYXQiOjE3Mjg5ODMyMjUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbiIsInVpZCI6ImRiZWQ5YjNhLTBiNmEtNDllOS05YTc4LTlmOGZkYzFkNWMzNyJ9fSwibmJmIjoxNzI4OTgzMjI1LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4ifQ.hnGxXU9nfKDVv46eI61j4NMEotGEXZNy3nESaFjiEMBGMyrFYPh1FUKyiHjgu1t7ahF3ToJGeSiIp3CamSFf3kAXKwdjpd3Sb8mW4gWTXcFyBwl7x_yz_HKXgTVbvakzvWBUrxdvWf99CVIx9CKUBU9SVPGw_n9IsoKIDb1Hc2Pn9I7cVjiAm-z7zc6v6YmWauMjeTFGbIlf5ABu0IpG6p89RFefBaRQAiXtEDfX23G_VVYULyrdu7waxPPqFwZZdr3a0PK2j6SPp7MBU-xM24trnOwUyTLPhuusCxKVE49JFqXq3PSWZgqYZovd2qPqdLBdtpz0PoEnF9X6g7YO3Q仅做演示用自己生成的tokentoken默认的有效期为24小时过期之后就不能用了需要重新创建token操作如下kubeadm token create --print-join-command短时间内生成多个token时生成新token后建议删除前一个旧的。# 查看命令kubeadm token list# 删除命令kubeadm token delete tokenid使用token登录参考资料Kubernetes超详细教程一篇文章帮助你从零开始学习k8s从入门到实战-CSDN博客搭建K8s集群_found multiple cri endpoints on the host. please d-CSDN博客coredns [ERROR] plugin/errors: 2 read udp : i/o timeout_coredns io timeout-CSDN博客