第一部分理解 K8s 原理1.1 应用部署发展过程传统物理机部署软件直接装在服务器。缺点资源没法限制一个程序吃满 CPU 内存会把别的程序搞崩。虚拟机部署一台物理机跑多台虚拟机每台虚拟机独立操作系统。缺点每台虚拟机完整装一套系统特别吃内存硬盘资源浪费大。容器部署 (Docker)多个容器共享宿主机操作系统内核每个容器打包自己的程序和依赖体积小、启动快。问题来了机器少手动启动容器还行几十上百台服务器成千上万个容器手动管不过来容器挂了怎么自动重启访问量变大怎么自动多开容器怎么把请求分发到多个容器 → 这就需要容器编排工具 K8s专门帮我们批量管理容器。答这就需要容器编排工具 K8s专门帮我们批量管理容器。1.2 K8s 集群两大角色控制节点 工作节点整个 K8s 集群分成两类机器Master 控制节点做决策不跑业务程序apiserver整个集群唯一入口所有操作创建 / 删除应用都找它做身份校验读写集群数据。etcd集群数据库存全部集群信息有多少机器、有多少容器、配置是什么。scheduler 调度器新的容器该放到哪台 worker 机器上跑它来选机器。controller‑manager 控制器盯着集群状态如果容器挂了就命令重新拉起保证符合你的预期。Node 工作节点干活真正跑业务容器kubelet每台 worker 机器上的代理接收 master 大脑下发指令管理本机容器生命周期。容器运行时真正创建、启动容器的软件本实验用 Docker因为 k8s1.24 之后不再原生支持 docker必须装cri‑dockerd适配器做中转。kube‑proxy网络代理负责服务发现、负载均衡外部访问业务的时候转发流量。最小单元叫PodPod 是 k8s 最小单元一个 Pod 里面可以放 1 个或者多个容器同一个 Pod 容器共享网络、共享存储。平时我们部署应用本质是部署 Pod。1.3 为什么我们要部署 Harbor 私有镜像仓库镜像就是容器安装包。默认镜像要去外网下载实验室环境外网慢甚至不通。Harbor 就是本地局域网的镜像仓库服务器把所有 k8s 需要的组件镜像下载下来传到 harbor所有 k8s 机器从内网拉镜像速度飞快。1.4 为什么要用 cri‑dockerdK8s 1.24 版本开始官方抛弃 dockershim不再原生对接 Docker。Docker 本身不能直接被 k8s 调用。cri‑dockerd就是一个 “翻译适配器”k8s 发标准 CRI 接口指令适配器转成 Docker 能听懂的指令让 Docker 继续作为容器运行时。1.5 网络插件 Flannel 是干嘛的刚初始化完 k8s 集群节点状态是NotReady没准备好。因为 k8s 需要一套网络不同机器上 Pod 之间要互相通信。Flannel 就是网络插件搭建集群内部网络装好之后节点状态变成 Ready。1.6 必做环境预处理为什么关闭 swap 交换分区swap 是硬盘虚拟内存。k8s 不允许开启 swap内存不够宁愿把容器杀掉也不用硬盘虚拟内存会影响调度性能。内核参数开启网桥转发容器之间网络互通需要开启内核 iptables 转发。主机解析 /etc/hosts各个机器通过域名能互相找到不用记复杂 IP。第二部分完整实验部署集群规划主机名ip配置角色harbor172.25.254.200cpu2、mem 2Gharbor仓库master172.25.254.150cpu4、mem4Gmasterk8s集群控制节点node1172.25.254.30cpu2、mem 2Gworkerk8s集群工作节点node2172.25.254.40cpu2、mem 2Gworkerk8s集群工作节点步骤一部署 Harbor 服务器1.1 制作本地 docker yum 源[rootharbor ~]# cat /etc/yum.repos.d/docker.repo EOF[docker]name dockerbaseurl https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/gpgcheck 0EOF[rootharbor ~]# dnf install httpd createrepo -y[rootharbor ~]# mkdir /var/www/html/docker/ -p[rootharbor ~]# vim /etc/httpd/conf/httpd.conf[rootharbor ~]# systemctl enable --now httpd[rootharbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/ -y[rootharbor ~]# mv /mnt/*.rpm /var/www/html/docker/[rootharbor ~]# createrepo -v /var/www/html/docker/[rootharbor ~]# cat /etc/yum.repos.d/docker.repo EOF[docker]name dockerbaseurl http://172.25.254.200:4444/dockergpgcheck 0EOF1.2 Harbor 节点配置 docker 和内核参数[rootharbor ~]# dnf install docker-ce -y[rootharbor ~]# echo br_netfilter /etc/modules-load.d/docker_mod.conf[rootharbor ~]# modprobe -a br_netfilter[rootharbor ~]# vim /etc/sysctl.d/docker.conf[rootharbor ~]# systemctl restart systemd-modules-load.service[rootharbor ~]# sysctl --system[rootharbor ~]# vim /lib/systemd/system/docker.service[rootharbor ~]# systemctl daemon-reload[rootharbor ~]# systemctl enable --now docker1.3 生成自签名证书[rootharbor ~]# mkdir /data/certs -p[rootharbor ~]# openssl req -newkey rsa:4096 \-nodes -sha256 -keyout /data/certs/timinglee.org.key \-addext subjectAltName DNS:reg.timinglee.org \-x509 -days 365 -out /data/certs/timinglee.org.crt1.4 解压配置 Harbor[rootharbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/[rootharbor ~]# cd /opt/harbor/[rootharbor harbor]# cp harbor.yml.tmpl harbor.yml[rootharbor harbor]# vim harbor.yml[rootharbor harbor]# ./install.sh --with-chartmuseum1.5 设置 harbor 为 systemd 服务管理[rootharbor harbor]# vim /lib/systemd/system/harbor.service[rootharbor harbor]# systemctl daemon-reload[rootharbor harbor]# systemctl enable --now harbor[rootharbor harbor]# docker compose up[rootharbor harbor]# docker compose down1.6 docker 信任 harbor 证书[rootharbor harbor]# mkdir /etc/docker/certs.d/reg.timinglee.org/ -p[rootharbor harbor]# cp /data/certs/timinglee.org.crt /etc/docker/certs.d/reg.timinglee.org/ca.crt[rootharbor harbor]# vim /etc/hosts[rootharbor harbor]# systemctl restart docker步骤二所有 k8s 节点统一环境配置2.1 所有主机彻底关闭 swap 分区以harbor为例[rootharbor harbor]# sed /swap/s/^/#/g -i /etc/fstab[rootharbor harbor]# reboot2.2 所有节点配置 /etc/hosts 主机解析[rootharbor ~]# vim /etc/hosts[rootharbor ~]# for i in 150 30 40 do scp /etc/hosts root172.25.254.$i:/etc/hosts done步骤三利用Ansible部署kubernetes运行环境3.1 安装ansible[rootharbor ~]# vim /etc/yum.repos.d/epel.repo[rootharbor ~]# dnf install ansible-core.x86_64 ansible.noarch -y3.2 配置所需用户[rootharbor ~]# useradd devops[rootharbor ~]# echo lee | passwd --stdin devops[rootharbor ~]# su - devops[devopsharbor ~]$ mkdir ansible[devopsharbor ~]$ cd ansible/[devopsharbor ansible]$3.3 生成ansible的临时配置文件并建立运行环境[devopsharbor ansible]$ vim inventory[devopsharbor ansible]$ vim ansible.cfg#建立用户[devopsharbor ansible]$ ansible all -m user -a namedevops statepresent[devopsharbor ansible]$ ansible all -m shell -a echo lee | passwd --stdin devops[devopsharbor ansible]$ ansible all -m lineinfile -a path/etc/sudoers linedevops ALL(ALL) NOPASSWD: ALL insertafterEOF3.4生成ansible的主配置文件并测试[devopsharbor ansible]$ vim ansible.cfg测试3.5 利用ansible剧本部署docker#关闭防火墙[rootharbor ~]# systemctl disable --now firewalld.serviceRemoved /etc/systemd/system/multi-user.target.wants/firewalld.service.Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.[devopsharbor ansible]$ su -密码[rootharbor ~]# cp /opt/harbor/common/config/registry/root.crt /home/devops/ansible/ca.crt[rootharbor ~]# chown devops:devops /home/devops/ansible/ca.crt[rootharbor ~]# exit注销[devopsharbor ansible]$ ls -l /home/devops/ansible/ca.crt-rw-r--r-- 1 devops devops 0 8月 22 19:55 /home/devops/ansible/ca.crt[devopsharbor ansible]$ vim kubernetes-devops.yml- name: devops k8s from ansiblehosts: allbecome: truegather_facts: truetasks:- name: setup docker repoyum_repository:name: dockerdescription: dockerbaseurl: http://172.25.254.200:4444/dockerfile: dockergpgcheck: no- name: install dockerdnf:name: docker-cestate: present- name: modify docker.service ExecStartreplace:path: /lib/systemd/system/docker.serviceregexp: ^(ExecStart/usr/bin/dockerd.*)$replace: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestruebackup: yes- name: systemctl daemon‑reload after docker.service changesystemd:daemon_reload: yes- name: setup docker registry daemon.jsoncopy:content: |{registry-mirrors:[https://reg.timinglee.org]}dest: /etc/docker/daemon.jsonmode: 0644- name: create docker cert directoryfile:path: /etc/docker/certs.d/reg.timinglee.orgstate: directorymode: 0755- name: copy harbor ca certificatecopy:src: ./ca.crtdest: {{ item.dest }}loop:- { dest: /etc/docker/certs.d/reg.timinglee.org/ca.crt }- { dest: /etc/pki/ca-trust/source/anchors/ca.crt }- name: load kernel module br_netfilter configcopy:dest: {{ item.dest }}content: {{ item.value }}loop:- dest: /etc/modules-load.d/docker_mod.confvalue: br_netfilter- dest: /etc/sysctl.d/docker.confvalue: |net.bridge.bridge-nf-call-iptables 1net.bridge.bridge-nf-call-ip6tables 1net.ipv4.ip_forward 1- name: load br_netfilter kernel modulecommunity.general.modprobe:name: br_netfilterstate: present- name: refresh ca trust reload sysctlshell: update-ca-trust extract; sysctl --systemchanged_when: false- name: start and enable servicesservice:name: {{ item }}state: restartedenabled: yesloop:- docker- systemd-modules-load.service[devopsharbor ansible]$ ansible-playbook kubernetes-devops.yml3.6 利用ansible安装cri-dockerd[devopsharbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz[devopsharbor ansible]$ cd cri-dockerd/[devopsharbor cri-dockerd]$ vim cri-docker.service[devopsharbor ansible]$ vim kubernetes-devops.yml- name: devops k8s from ansiblehosts: alltasks:- name: setup docker repoyum_repository:name: dockerdescription: dockerbaseurl: http://172.25.254.200:4444/dockerfile: dockergpgcheck: no- name: install dockerdnf:name: docker-cestate: present- name: setup docker.servicereplace:path: /lib/systemd/system/docker.serviceregexp: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sockreplace: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestrue- name: setup docker registrycopy:content: |{registry-mirrors:[https://reg.timinglee.org]}dest: /etc/docker/daemon.json- name: setup docker certsfile:path: /etc/docker/certs.d/reg.timinglee.orgstate: directory- name: cp certs filecopy:src: {{item.src}}dest: {{item.dest}}loop:- src: /etc/docker/certs.d/reg.timinglee.org/ca.crtdest: /etc/docker/certs.d/reg.timinglee.org/ca.crt- src: /etc/docker/certs.d/reg.timinglee.org/ca.crtdest: /etc/pki/ca-trust/source/anchors/ca.crt########################################################- name: stup cri-dockerdcopy:src: {{item.src}}dest: {{item.dest}}mode: 0755loop:- src: ./cri-dockerd/cri-dockerddest: /usr/bin/cri-dockerd- src: ./cri-dockerd/cri-docker.servicedest: /lib/systemd/system/cri-docker.service- src: ./cri-dockerd/cri-docker.socketdest: /lib/systemd/system/cri-docker.socket- name: update ca certshell: update-ca-trust extract;sysctl --system;systemctl daemon-reload- name: start servicesservice:name: {{item}}state: restartedenabled: yesloop:- docker- systemd-modules-load.service- cri-docker[devopsharbor ansible]$ ansible-playbook kubernetes-devops.yml3.7 利用ansible部署k8s软件[devopsharbor ansible]$ mv kubernetes-devops.yml docker-devops.yml[devopsharbor ansible]$ vim kubernetes-devops.yml- name: install kubernetes from ansiblehosts: alltasks:- name: setup k8s repoyum_repository:name: kubernetesdescription: kubernetesbaseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpmfile: kubernetesgpgcheck: no- name: install software in masterdnf:name: {{item}}state: presentloop:- kubeadm-1.35.7-150500.1.1- kubelet-1.35.7-150500.1.1- kubectl-1.35.7-150500.1.1when: inventory_hostname 172.25.254.150- name: install software in work nodednf:name: {{item}}state: presentloop:- kubeadm-1.35.7-150500.1.1- kubelet-1.35.7-150500.1.1when: inventory_hostname 172.25.254.30 or inventory_hostname 172.25.254.40- name: setup kubectl kubeadm complicationlineinfile:path: /root/.bashrcline: |source (kubectl completion bash)source (kubeadm completion bash)when: inventory_hostname 172.25.254.150- name: start kubeletservice:name: kubeletstate: startedenabled: yes[devopsharbor ansible]$ ansible-playbook kubernetes-devops.yml#让补齐功能生效[rootmaster ~]# source ~/.bashrc[rootmaster ~]# kubectl 命令后按2此tab键alpha (Commands for features in alpha)annotate (更新一个资源的注解)api-resources (Print the supported API resources on the server)api-versions (Print the supported API versions on the server, in the form of group/version)apply (Apply a configuration to a resource by file name or stdin)attach (挂接到一个运行中的容器)auth (Inspect authorization)autoscale (Auto-scale a deployment, replica set, stateful set, or replication controller)certificate (Modify certificate resources)[rootmaster ~]# kubeadm