k8s部署Rabbitmq集群

rabbitmq-peer-discovery-k8s是RabbitMQ官方基于第三方开源项目rabbitmq-autocluster开发，对3.7.X版本提供的Kubernetes下的对等发现插件，可实现rabbitmq集群在k8s中的自动化部署；低于3.7.X版本请使用rabbitmq-autocluster。

参考：

rabbitmq-peer-discovery-k8s

rabbitmq-peer-discovery-k8s example

Cluster Formation and Peer Discovery

在Kubernetes上搭建RabbitMQ Cluster

创建

1. 创建RBAC授权

假设namesapce为ingress

rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq 
  namespace: ingress 
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: ingress 
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: ingress
subjects:
- kind: ServiceAccount
  name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader

创建

kubectl create -f rbac.yaml

2. 创建configmap

configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: ingress
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
  rabbitmq.conf: |
      ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more.
      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address?
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
      ## Set to "hostname" to use pod hostnames.
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
      ## environment variable.
      cluster_formation.k8s.address_type = hostname
      ## How often should node cleanup checks run?
      cluster_formation.node_cleanup.interval = 30
      ## Set to false if automatic removal of unknown/absent nodes
      ## is desired. This can be dangerous, see
      ##  * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
      ##  * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality
      queue_master_locator=min-masters
      ## See https://www.rabbitmq.com/access-control.html#loopback-users
      loopback_users.guest = false

      cluster_formation.randomized_startup_delay_range.min = 0
      cluster_formation.randomized_startup_delay_range.max = 2
      # 必须设置service_name，否则Pod无法正常启动，这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量
      cluster_formation.k8s.service_name = rabbitmq-cluster
      # 必须设置hostname_suffix，否则节点不能成为集群，同时保证namespace正确
      cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.ingress.svc.cluster.local
      # 内存上限
      vm_memory_high_watermark.absolute = 1GB
      # 硬盘上限
      disk_free_limit.absolute = 2GB

rabbitmq_peer_discovery_k8s参数解析：

• cluster_formation.peer_discovery_backend：对待发现后台插件
• cluster_formation.k8s.host：k8s服务主机(默认即可)
• cluster_formation.k8s.address_type：基于IP/Hostname
• cluster_formation.node_cleanup.interval：清理节点检查时间间隔

注意：官方demo示例中，cluster_formation.k8s.address_type = ip，使用的是IP模式；也就是说RabbitMQ Node的命名和访问地址是以IP地址作为区分，如rabbit@172.0.5.1，但这样的配置会产生比较大的问题，如果我们使用pv和pvc去做数据的持久化，那么每个节点的配置和数据存储都会放在rabbit@172.0.5.1这样的文件夹下，而Kubernetes集群中，Pod的IP都是不稳定的，当有RabbitMQ Node的Pod挂掉后，重新创建的Pod IP可能会变，这就会使得节点的配置和数据全部丢失。所以我们更希望RabbitMQ Node的命名是以一定规则编写的相对稳定的名称，如rabbit@rabbit-0，这就需要修改 cluster_formation.k8s.address_type = hostname，以启用hostname模式。这里headless service就派上用场了。

创建

kubectl create -f configmap.yaml

3. 创建headless service

service.yaml

---
kind: Service
apiVersion: v1
metadata:
  namespace: ingress
  name: rabbitmq-cluster
  labels:
    app: rabbitmq-cluster
    type: LoadBalancer  
spec:
  clusterIP: None
  # 由于使用DNS访问Pod需Pod和Headless service启动之后才能访问，publishNotReadyAddresses设置成true，防止readinessProbe在服务没启动时找不到DNS
  publishNotReadyAddresses: true 
  ports:
   - name: http
     protocol: TCP
     port: 15672
     targetPort: 15672
   - name: amqp
     protocol: TCP
     port: 5672
     targetPort: 5672
  selector:
    app: rabbitmq

创建

kubectl create -f service.yaml

4. 创建statefulset

statefulset.yaml

---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: ingress
spec:
  serviceName: rabbitmq-cluster
  replicas: 3
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      containers:        
      - name: rabbitmq
        image: 192.168.100.100/library/rabbitmq:3.7.4
        volumeMounts:
          - name: config-volume
            mountPath: /etc/rabbitmq
          - name: rabbitmq-data
            mountPath: /var/lib/rabbitmq
        ports:
          - name: http
            protocol: TCP
            containerPort: 15672
          - name: amqp
            protocol: TCP
            containerPort: 5672
        livenessProbe:
          exec:
            command: ["rabbitmqctl", "status"]
          initialDelaySeconds: 60
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
          periodSeconds: 60
          timeoutSeconds: 15
        readinessProbe:
          exec:
            command: ["rabbitmqctl", "status"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        imagePullPolicy: Always
        env:
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: RABBITMQ_USE_LONGNAME
            value: "true"
          # See a note on cluster_formation.k8s.address_type in the config file section
          - name: RABBITMQ_NODENAME
            value: "rabbit@$(HOSTNAME).rabbitmq-cluster.$(NAMESPACE).svc.cluster.local"
        #   若在ConfigMap中设置了service_name，则此处无需再次设置
        #   - name: K8S_SERVICE_NAME
        #     value: "rabbitmq-cluster"
          - name: RABBITMQ_ERLANG_COOKIE
            value: "mycookie" 
      volumes:
        - name: config-volume
          configMap:
            name: rabbitmq-config
            items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
            - key: enabled_plugins
              path: enabled_plugins
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-data
      labels:
        name: rabbitmq-data
    spec:
      storageClassName: managed-nfs-storage
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi

创建

kubectl create -f statefulset.yaml

等待所有Pod全是Running状态后，登陆到pod内部查询集群信息

[root@k8s-node1 services]# kubectl -n ingress exec -it rabbitmq-0 bash
root@rabbitmq-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-0.rabbitmq-cluster.ingress.svc.cluster.local ...
[{nodes,
     [{disc,
          ['rabbit@rabbitmq-0.rabbitmq-cluster.ingress.svc.cluster.local',
           'rabbit@rabbitmq-1.rabbitmq-cluster.ingress.svc.cluster.local',
           'rabbit@rabbitmq-2.rabbitmq-cluster.ingress.svc.cluster.local']}]},
 {running_nodes,
     ['rabbit@rabbitmq-2.rabbitmq-cluster.ingress.svc.cluster.local',
      'rabbit@rabbitmq-1.rabbitmq-cluster.ingress.svc.cluster.local',
      'rabbit@rabbitmq-0.rabbitmq-cluster.ingress.svc.cluster.local']},
 {cluster_name,
     <<"rabbit@rabbitmq-0.rabbitmq-cluster.ingress.svc.cluster.local">>},
 {partitions,[]},
 {alarms,
     [{'rabbit@rabbitmq-2.rabbitmq-cluster.ingress.svc.cluster.local',[]},
      {'rabbit@rabbitmq-1.rabbitmq-cluster.ingress.svc.cluster.local',[]},
      {'rabbit@rabbitmq-0.rabbitmq-cluster.ingress.svc.cluster.local',[]}]}]

5. 创建ingress

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "ingress"
  name: rabbitmq-ingress
  namespace: ingress
spec:
  rules:
  - host: rabbitmq-ingress.test.com
    http:
      paths:
      - backend:
          serviceName: rabbitmq-cluster
          servicePort: 15672
        path: /

创建

kubectl create -f ingress.yaml

登录http://rabbitmq-ingress.test.com:15672(保证域名解析正确)即可访问web管理页面（默认用户密码均为guest）

6. 节点伸缩

可以直接使用scale命令行来伸缩工作节点：

kubectl scale  statefulset rabbitmq --replicas=4 -n ingress