iSt0ne's Notes

使用Rancher部署Kubernetes集群

准备工作

在Openstack上准备6台虚拟机,1个Rancher Server,5个Rancher Node用来部署Kubernetes

Openstack VM

所有机器上部署Docker服务

tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF

setenforce 0
# 升级CentOS内核到最新版本,否则Kubernetes Dashborad会启动失败
# https://github.com/rancher/rancher/issues/7436
yum update -y kernel
# 安装Docker 1.12.6
yum install -y docker-engine-1.12.6 docker-engine-selinux-1.12.6
# 启动Docker服务
systemctl enable docker && systemctl start docker
# 添加自己的Docker仓库
vi /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --insecure-registry=d.qianyilc.com
# 重启Docker服务
systemctl daemon-reload && systemctl restart docker
# 重启各节点
reboot

所有Rancher Node节点pull Kubernetes镜像(在中国不能访问gcr.io,所以要手动从自己的docker仓库中pull下来再改名)

images=(pause-amd64:3.0 kubedns-amd64:1.9 dnsmasq-metrics-amd64:1.0 kube-dnsmasq-amd64:1.4 exechealthz-amd64:1.2 kubernetes-dashboard-amd64:v1.5.0)
for imageName in ${images[@]} ; do
  docker pull ist0ne/$imageName
  docker tag ist0ne/$imageName gcr.io/google_containers/$imageName
  docker rmi ist0ne/$imageName
done

docker pull ist0ne/tiller:v2.1.3
docker tag ist0ne/tiller:v2.1.3 gcr.io/kubernetes-helm/tiller:v2.1.3
docker rmi ist0ne/tiller:v2.1.3

Rancher Server部署

在Rancher Server节点上部署rancher v1.3.4

docker run -d -v /data1/mysql:/var/lib/mysql --restart=unless-stopped -p 8080:8080 rancher/server:v1.3.4

Rancher Server

在Rancher Server节点上部署rancher agent

点击添加Host节点

Add Host

Rancher Server节点上执行以下命令安装rancher agent

docker run -e CATTLE_AGENT_IP="10.101.1.160"  -e CATTLE_HOST_LABELS='env=cattle'  -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.1.3 http://10.101.1.160:8080/v1/scripts/97DCE18521C57C4ADB39:1486958400000:pVJWR0mHYlgvnhh6VxXbsfC6w

等待Cattle部署完成,Cattle用于部署Kubernetes环境

Rancher Cattle

创建Kubernetes环境

由于官网Kubernetes环境模板中kubelet服务未挂载/var/log/containers到宿主机上,会造成部署日志收集服务时收集不到容器的日志(kubelet会将个容器的日志/var/lib/docker/containers/*/*-json.log连接到/var/log/containers目录下),所以要手工修改模板。

获取Rancher Server的ID

[root@rancher-server ~]# docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                              NAMES
e128f8e58d46        rancher/server:stable            "/usr/bin/entry /usr/"   3 days ago          Up 3 days           3306/tcp, 0.0.0.0:8080->8080/tcp   rancher

进入Rancher Server

[root@rancher-server ~]# docker exec -ti e128 bash
root@e128f8e58d46:/# cd /var/lib/cattle/DATA/library/infra-templates
# 添加- /var/log/containers:/var/log/containers行;由于gcr.io在中国不能访问,可以通过--pod-infra-container-image指定其他仓库地址
root@e128f8e58d46:/var/lib/cattle/DATA/library/infra-templates# vi k8s/17/docker-compose.yml
kubelet:
    labels:
        io.rancher.container.dns: "true"
        io.rancher.container.create_agent: "true"
        io.rancher.container.agent.role: environmentAdmin
        io.rancher.scheduler.global: "true"
        io.rancher.scheduler.affinity:host_label_ne: nopods=true
    command:
        - kubelet
        - --kubeconfig=/etc/kubernetes/ssl/kubeconfig
        - --api_servers=https://kubernetes.kubernetes.rancher.internal:6443
        - --allow-privileged=true
        - --register-node=true
        - --cloud-provider=${CLOUD_PROVIDER}
        - --healthz-bind-address=0.0.0.0
        - --cluster-dns=10.43.0.10
        - --cluster-domain=cluster.local
        - --network-plugin=cni
        - --network-plugin-dir=/etc/cni/managed.d
    image: rancher/k8s:v1.5.1-rancher1-7
    volumes:
        - /run:/run
        - /var/run:/var/run
        - /var/lib/docker:/var/lib/docker
        - /var/lib/kubelet:/var/lib/kubelet:shared
        - /var/log/containers:/var/log/containers
        - rancher-cni-driver:/etc/cni:ro
        - rancher-cni-driver:/opt/cni:ro
        - /dev:/host/dev
    net: host
    pid: host
    ipc: host
    privileged: true

点击 管理环境,然后添加环境

Rancher Manage ENV

填写环境名称,选择Kubernetes,然后点击创建

Rancher ENV Kubernetes

添加Host,将5个节点全部添加上

Rancher Add Kubernetes Host

在Rancher Node 1上执行,其他类似,注意对应修改CATTLE_AGENT_IP

docker run -e CATTLE_AGENT_IP="10.101.1.161"  -e CATTLE_HOST_LABELS='env=k8s'  -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.1.3 http://10.101.1.160:8080/v1/scripts/7630883879B26E0386FE:1486962000000:olEZygVDYB9NpFr8P16Zi34jo

五个节点全部添加

Rancher Add Kubernetes Host

等待Kubernetes环境部署完成

Rancher ENV Kubernetes

点击Kubernetes菜单,然后点击Dashboard,然后点击Kubernetes UI

Kubernetes UI

安装kubectl

kubectl install

在用户home目录下创建.kube/config文件,并下载kubectl二进制文件到/usr/local/bin目录下,给执行权限

kubectl install

添加Catalog

点击ADMIN菜单,点击Settings,设置Catalog

Rancher Settings

关闭系统自带的社区Catalog,换成自己的地址,点击保存

Rancher Settings Close Catalog

打开My Catalog

Rancher My Catalog

添加监控服务

监控服务配置代码

https://github.com/ist0ne/community-catalog/tree/master/kubernetes-templates/monitor

点击View Details

Rancher Add Monitor

Rancher Add Monitor

部署完成,访问30804端口

Rancher Add Monitor

能够监控到每台主机信息

Rancher Monitor Hosts

由于Kubernetes 1.5.1有bug,不能获取到pod name,所以监控不到pods

[root@rancher-node1 ~]# curl -s http://localhost:10255/stats/summary
{
  "node": {
   "nodeName": "rancher-node1.novalocal",
   "startTime": "2017-02-09T08:41:27Z",
   "cpu": {
    "time": "2017-02-13T05:38:19Z",
    "usageNanoCores": 10317365,
    "usageCoreNanoSeconds": 3175712778188
   },
   "memory": {
    "time": "2017-02-13T05:38:19Z",
    "availableBytes": 8159453184,
    "usageBytes": 120680448,
    "workingSetBytes": 43360256,
    "rssBytes": 36728832,
    "pageFaults": 46345275,
    "majorPageFaults": 89
   },
   "network": {
    "time": "2017-02-13T05:38:19Z",
    "rxBytes": 25590500746,
    "rxErrors": 0,
    "txBytes": 6993550652,
    "txErrors": 0
   },
   "fs": {
    "availableBytes": 72684294144,
    "capacityBytes": 85885063168,
    "usedBytes": 13200769024,
    "inodesFree": 83845738,
    "inodes": 83882368,
    "inodesUsed": 36630
   },
   "runtime": {
    "imageFs": {
     "availableBytes": 72684294144,
     "capacityBytes": 85885063168,
     "usedBytes": 4494467941,
     "inodesFree": 83845738,
     "inodes": 83882368,
     "inodesUsed": 36630
    }
   }
  },
  "pods": []
}

添加日志收集服务

日志服务配置代码

https://github.com/ist0ne/community-catalog/tree/master/kubernetes-templates/logging

进入My Catalog,点击View Details

Rancher Add Logging

Rancher Add Logging

待日志收集服务部署完成

Rancher Add Logging

查看日志收集情况

Rancher Logging