- A+
所属分类：.NET技术
前期准备
环境规划
K8S 与Docker兼容问题
k8s v1.18.0 => Docker v18.x
k8s v1.19.0 => Docker v19.x
| 软件 | 版本 |
|---|---|
| Linux操作系统 | CentOS 7.9.2009 (Core) x64 |
| Kubernetes | 1.8.0 |
| Docker | 18.06.3-ce |
| 角色 | IP | 组件 | 推荐配置(最低) |
|---|---|---|---|
| master | 192.168.137.101 | kubelet kubeadm kubectl docker |
CUP 2 核 + 内存 2G + |
| node1 | 192.168.137.102 | kubelet kubeadm kubectl docker |
CUP 2 核 + 内存 2G + |
| node2 | 192.168.137.103 | kubelet kubeadm kubectl docker |
CUP 2 核 + 内存 2G + |
修改HostName
# 修改hostname # vi /etc/hostname # 192.168.137.101 hostnamectl set-hostname master # 192.168.137.102 hostnamectl set-hostname node1 # 192.168.137.103 hostnamectl set-hostname node2 配置主机和IP映射
# 将本机IP指向hostname vi /etc/hosts 192.168.137.101 master 192.168.137.102 node1 192.168.137.103 node2 reboot -h # 重启(可以做完全部前期准备后再重启) 放行需求端口(线上环境)
# Master节点端口放行 # Kubernetes API Server 6443 firewall-cmd --zone=public --add-port=6443/tcp --permanent # etcd server client api 2379~2380 firewall-cmd --zone=public --add-port=2379-2380/tcp --permanent # kubelet 10250, kube-scheduler 10251, kube-controller-manager 10252 firewall-cmd --zone=public --add-port=10250-10252/tcp --permanent # Node节点端口放行 # kubelet API 10250 firewall-cmd --zone=public --add-port=10250/tcp --permanent # NodePort Services 30000~32767 firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent firewall-cmd --reload firewall-cmd --list-ports 直接关闭防火墙(不推荐)
systemctl disable firewalld systemctl stop firewalld 安装Docker
# 安装 wget yum install -y wget # 下载 docker 镜像源 wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo # docker安装版本查看 yum list docker-ce --showduplicates | sort -r # 安装 docker yum -y install docker-ce # 指定版本 yum -y install docker-ce-18.06.3.ce-3.el7 # 设置开机自启动 systemctl enable docker && systemctl start docker # 版本检查 docker --version Docker version 18.06.3-ce, build d7080c1 修改配置文件
vi /etc/docker/daemon.json { "registry-mirrors": [ "https://1nj0zren.mirror.aliyuncs.com", "https://docker.mirrors.ustc.edu.cn", "http://f1361db2.m.daocloud.io", "https://registry.docker-cn.com" ], "exec-opts": [ "native.cgroupdriver=systemd" ], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } #重新加载配置文件 systemctl daemon-reload #重启Docker systemctl restart docker 安装Kubernetes工具
添加源
由于国内网络原因, 官方文档中的地址不可用, 本文替换为阿里云镜像地址, 执行以下代码即可:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg exclude=kube* EOF # 注意：gpgkey 后面的两个网址中间是空格，不是换行，复制后出现换行会导致安装出错 安装
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes # 指定版本 yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0 --disableexcludes=kubernetes # 如下出现错误 [Errno -1] repomd.xml signature could not be verified for kubernetes 则是 repo 的 gpg 验证不通过导致的，可以修改 /etc/yum.repos.d/kubernetes.repo 中的 repo_gpgcheck=0 跳过验证。 systemctl enable kubelet && systemctl start kubelet 修改网络配置
cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system 注意: 以上的全部操作, 在 Node 机器上也需要执行. 注意hostname等不能相同.
初始化Master
生成初始化文件
1. 配置文件方式
kubeadm config print init-defaults > kubeadm-init.yaml vi kubeadm-init.yaml ################################################################# localAPIEndpoint: #advertiseAddress: 1.2.3.4 advertiseAddress: 192.168.137.101 # 本机IP nodeRegistration: #name: localhost.localdomain name: master #imageRepository: k8s.gcr.io imageRepository: registry.aliyuncs.com/google_containers # 镜像仓库 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 # 新增Pod子网络 ################################################################# :wq 修改完毕后文件如下:
apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: #advertiseAddress: 1.2.3.4 advertiseAddress: 192.168.137.101 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock #name: localhost.localdomain name: master taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd #imageRepository: k8s.gcr.io imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.18.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 scheduler: {} 2.直接传参方式（推荐，老司机常用方式）
kubeadm init --apiserver-advertise-address=192.168.137.101 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.0 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16 下载镜像
kubeadm config images pull --config kubeadm-init.yaml 配置禁用Swap
# 注意不要重复执行 sed -i 's/KUBELET_EXTRA_ARGS=/KUBELET_EXTRA_ARGS="--fail-swap-on=false"/' /etc/sysconfig/kubelet # 临时关闭 swapoff -a 执行初始化
kubeadm init --config kubeadm-init.yaml # 出现端口被占用情况 kubeadm reset kubeadm init --config kubeadm-init.yaml --ignore-preflight-errors=Swap # reset后初始化提示文件已存在 rm -rf /etc/kubernetes/manifests rm -rf /var/lib/etcd 验证是否成功
# 出现下面文字表示初始化成功： Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.137.101:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:d126a8ec9cb47ac4bfae5a2d7501172da937d91b1ccf0eae093a9a3687c841f2 配置环境, 让当前用户可以执行kubectl命令
# 配置kubectl执行命令环境 mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config # 执行kubectl命令查看机器节点 kubectl get node ----------------------------------------- NAME STATUS ROLES AGE VERSION master NotReady master 48m v1.18.8 配置网络
使用以下命令安装 Calico
wget https://docs.projectcalico.org/manifests/calico.yaml # 获取网络信息 firewall-cmd --get-active-zones public interfaces: eth0 vi calico.yaml # 大概从 3639 行开始，有些改动没有则追加 ##################################################################### # Cluster type to identify the deployment type - name: CLUSTER_TYPE value: "k8s,bgp" # Auto-detect the BGP IP address. - name: IP value: "autodetect" # IP automatic detection. - name: IP_AUTODETECTION_METHOD value: "interface=eth.*" # Enable IPIP - name: CALICO_IPV4POOL_IPIP #value: "Always" value: "Never" ##################################################################### # 构建calico网络 kubectl apply -f calico.yaml # 检查结果 kubectl get po -n kube-system -o wide | grep calico 检查 master 的状态是否已经成为 Ready：
kubectl get node NAME STATUS ROLES AGE VERSION master Ready master 5m20s v1.18.0 安装Dashboard
安装文档: Web UI (Dashboard)
部署文档：Web UI (Dashboard)
解决GitHub的raw.githubusercontent.com无法连接问题
1、进入网址 https://site.ip138.com/raw.Githubusercontent.com/
2、输入 raw.githubusercontent.com，查询对应的IP地址：151.101.108.133
3、编辑/etc/hosts文件配置映射：151.101.108.133 raw.githubusercontent.com
# 下载配置文件 wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml # 创建 pod kubectl apply -f recommended.yaml # 查看 pods 状态 kubectl get pods --all-namespaces | grep dashboard # 使用 nodeport方式 将 dashboard服务 暴露在集群外，指定使用 30443 端口 kubectl patch svc kubernetes-dashboard -n kubernetes-dashboard -p '{"spec":{"type":"NodePort","ports":[{"port":443,"targetPort":8443,"nodePort":30443}]}}' # 查看暴露的service,已修改为nodeport类型 kubectl -n kubernetes-dashboard get svc # 此时我们可以访问登录面板: https://192.168.137.101:30443，但是暂时还无法登录 修改 Service
# 删除现有的dashboard服务 kubectl delete -f recommended.yaml # 重命名 recommended.yaml mv recommended.yaml dashboard-svc.yaml # 修改配置项 vi dashboard-svc.yaml ##################################################################### kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort # 服务类型改为 NodePort ports: - port: 443 targetPort: 8443 nodePort: 30443 # 暴露端口 30443 selector: k8s-app: kubernetes-dashboard ##################################################################### :wq # 重新创建 pod kubectl apply -f dashboard-svc.yaml 创建用户
文档地址: Creating sample user
vi dashboard-svc-account.yaml ##################################################################### apiVersion: v1 kind: ServiceAccount metadata: name: dashboard-admin namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: dashboard-admin roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: dashboard-admin namespace: kube-system ##################################################################### :wq # 执行 kubectl apply -f dashboard-svc-account.yaml 生成证书
官方文档中提供了登录 1.7.X 以上版本的登录方式，而且步骤很不清晰，我们自己按下面步骤操作即可：
grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key # 生成证书时会提示输入密码, 可以直接两次回车跳过. openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client" # kubecfg.p12 即需要导入客户端机器的证书. 将证书拷贝到客户端机器上: 若生成证书时跳过了密码, 导入时提示填写密码直接回车即可 scp [email protected]:/root/.kube/kubecfg.p12 ./ # 此时我们可以访问登录面板: https://192.168.137.101:30443 ,登录时会提示选择证书, 确认后会提示输入当前用户名密码(注意是电脑的用户名密码). 登录Dashboard（Token登录）
文档地址: Bearer Token
# 获取 Token: kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep dashboard-admin | awk '{print $1}') # 复制该Token到登录页, 点击登录即可 添加 Node 节点
# 关闭交换空间 swapoff -a # 如果前面执行 kubeadm init 命令后没有保留 kubeadm join 语句，需要执行如下命令重新生成： kubeadm token create --print-join-command kubeadm join 192.168.137.101:6443 --token ngqaor.ayhyq00qb3o0gxjk --discovery-token-ca-cert-hash sha256:4c18ecc6e9bd4457308b028123cbd16b2d3cbdefb14ec1e61b43a15e05ab63b3 # 执行如下命令将 Node 加入集群: kubeadm join 192.168.137.101:6443 --token ngqaor.ayhyq00qb3o0gxjk --discovery-token-ca-cert-hash sha256:4c18ecc6e9bd4457308b028123cbd16b2d3cbdefb14ec1e61b43a15e05ab63b3 添加完毕后, 在 master 上查看节点状态：
# 查看所有节点状态 kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 6h38m v1.18.0 node1 Ready <none> 32m v1.18.0 node2 Ready <none> 32m v1.18.0 # 查看所有 pod 状态 kubectl get po --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-65d7476764-zgfp2 1/1 Running 0 5h44m kube-system calico-node-dk6v2 0/1 Running 0 5h44m kube-system calico-node-rgt4x 0/1 PodInitializing 0 9m19s kube-system calico-node-tzvn2 0/1 Running 0 9m29s kube-system coredns-7ff77c879f-5hgb6 1/1 Running 0 6h15m kube-system coredns-7ff77c879f-l7wpq 1/1 Running 0 6h15m kube-system etcd-master 1/1 Running 0 6h15m kube-system kube-apiserver-master 1/1 Running 0 6h15m kube-system kube-controller-manager-master 1/1 Running 0 6h15m kube-system kube-proxy-6jf4p 1/1 Running 0 6h15m kube-system kube-proxy-nrsr2 1/1 Running 0 9m19s kube-system kube-proxy-sfh7l 1/1 Running 0 9m29s kube-system kube-scheduler-master 1/1 Running 0 6h15m kubernetes-dashboard dashboard-metrics-scraper-6b4884c9d5-kh88n 1/1 Running 0 124m kubernetes-dashboard kubernetes-dashboard-7b544877d5-csfkz 1/1 Running 0 124m 
