經(jīng)典wordpress網(wǎng)站寧波網(wǎng)站推廣代運(yùn)營(yíng)
?💝💝💝歡迎來(lái)到我的博客,很高興能夠在這里和您見(jiàn)面!希望您在這里可以感受到一份輕松愉快的氛圍,不僅可以獲得有趣的內(nèi)容和知識(shí),也可以暢所欲言、分享您的想法和見(jiàn)解。
推薦:Linux運(yùn)維老紀(jì)的首頁(yè),持續(xù)學(xué)習(xí),不斷總結(jié),共同進(jìn)步,活到老學(xué)到老
導(dǎo)航劍指大廠系列:全面總結(jié) 運(yùn)維核心技術(shù):系統(tǒng)基礎(chǔ)、數(shù)據(jù)庫(kù)、網(wǎng)路技術(shù)、系統(tǒng)安全、自動(dòng)化運(yùn)維、容器技術(shù)、監(jiān)控工具、腳本編程、云服務(wù)等。
常用運(yùn)維工具系列:常用的運(yùn)維開(kāi)發(fā)工具, zabbix、nagios、docker、k8s、puppet、ansible等
數(shù)據(jù)庫(kù)系列:詳細(xì)總結(jié)了常用數(shù)據(jù)庫(kù) mysql、Redis、MongoDB、oracle 技術(shù)點(diǎn),以及工作中遇到的 mysql 問(wèn)題等
懶人運(yùn)維系列:總結(jié)好用的命令,解放雙手不香嗎?能用一個(gè)命令完成絕不用兩個(gè)操作
數(shù)據(jù)結(jié)構(gòu)與算法系列:總結(jié)數(shù)據(jù)結(jié)構(gòu)和算法,不同類(lèi)型針對(duì)性訓(xùn)練,提升編程思維,劍指大廠
非常期待和您一起在這個(gè)小小的網(wǎng)絡(luò)世界里共同探索、學(xué)習(xí)和成長(zhǎng)。💝💝💝 ?? 歡迎訂閱本專(zhuān)欄 ??
基于k8s手動(dòng)部署rabbitmq集群
1、RabbitMQ介紹
RabbitMQ
是實(shí)現(xiàn)了高級(jí)消息隊(duì)列協(xié)議AMQP
的開(kāi)源消息代理軟件(亦稱(chēng)面向消息的中間件)。RabbitMQ
服務(wù)器是用Erlang
語(yǔ)言編寫(xiě)的,而集群和故障轉(zhuǎn)移是構(gòu)建在開(kāi)放電信平臺(tái)框架上的。AMQP
:Advanced Message Queue
,高級(jí)消息隊(duì)列協(xié)議。它是應(yīng)用層協(xié)議的一個(gè)開(kāi)放標(biāo)準(zhǔn),為面向消息的中間件設(shè)計(jì),基于此協(xié)議的客戶端與消息中間件可傳遞消息,并不受產(chǎn)品、開(kāi)發(fā)語(yǔ)言燈條件的限制
AMQP
具有如下的特性:
- 可靠性
Reliablity
:使用了一些機(jī)制來(lái)保證可靠性,比如持久化、傳輸確認(rèn)、發(fā)布確認(rèn) - 靈活的路由
Flexible Routing
:在消息進(jìn)入隊(duì)列之前,通過(guò)Exchange
來(lái)路由消息。對(duì)于典型的路由功能,Rabbit
已經(jīng)提供了一些內(nèi)置的Exchange
來(lái)實(shí)現(xiàn)。針對(duì)更復(fù)雜的路由功能,可以將多個(gè)Exchange
綁定在一起,也通過(guò)插件機(jī)制實(shí)現(xiàn)自己的Exchange
- 消息集群
Clustering
:多個(gè)RabbitMQ
服務(wù)器可以組成一個(gè)集群,形成一個(gè)邏輯Broker
- 高可用
Highly Avaliable Queues
:隊(duì)列可以在集群中的機(jī)器上進(jìn)行鏡像,使得在部分節(jié)點(diǎn)出問(wèn)題的情況下隊(duì)列仍然可用 - 多種協(xié)議
Multi-protocol
:支持多種消息隊(duì)列協(xié)議,如STOMP
、MQTT
等 - 多種語(yǔ)言客戶端
Many Clients
:幾乎支持所有常用語(yǔ)言,比如Java
、.NET
、Ruby
等 - 管理界面
Management UI
:提供了易用的用戶界面,使得用戶可以監(jiān)控和管理消息Broker
的許多方面 - 跟蹤機(jī)制
Tracing
:如果消息異常,RabbitMQ
提供了消息的跟蹤機(jī)制,使用者可以找出發(fā)生了什么 - 插件機(jī)制
Plugin System
:提供了許多插件,來(lái)從多方面進(jìn)行擴(kuò)展,也可以編輯自己的插件
2、RabbitMQ的持久化和鏡像隊(duì)列
RabbitMQ
持久化分為Exchange
、Queue
、Message
Exchange
和Queue
持久化:指持久化Exchange
、Queue
元數(shù)據(jù),持久化的是自身,服務(wù)宕機(jī)Exchange
和Queue
自身就沒(méi)有了Message
持久化:顧名思義就是把每一條消息體持久化,服務(wù)宕機(jī),消息不丟失
RabbitMQ
的隊(duì)列Queue
鏡像,指master node
在接受到請(qǐng)求后,會(huì)同步到其他節(jié)點(diǎn)上,以此來(lái)保證高可用。在confirm
模式下,具體過(guò)程如下
clientpublisher發(fā)送消息 –> master node接到消息 –> master node將消息持久化到磁盤(pán) –> 將消息異步發(fā)送給其他節(jié)點(diǎn) –> master將ack返回給client publisher
3、RabbitMQ集群在k8s中的部署
將RabbitMQ
以集群的方式部署在k8s
中,前提是RabbitMQ
的每個(gè)節(jié)點(diǎn)都能像傳統(tǒng)方式一樣進(jìn)行相互的服務(wù)發(fā)現(xiàn)。因此RabbitMQ
在k8s
集群中通過(guò)rabbitmq_peer_discovery_k8s plugin
與k8s apiserver
進(jìn)行交互,獲取各個(gè)服務(wù)的URL
,且RabbitMQ
在k8s
集群中必須用statefulset
和headless service
進(jìn)行匹配
需要注意的是,rabbitmq_peer_discovery_k8s
是RabbitMQ
官方基于第三方開(kāi)源項(xiàng)目rabbitmq-autocluster
開(kāi)發(fā),對(duì)3.7.X
及以上版本提供的Kubernetes
下的對(duì)等發(fā)現(xiàn)插件,可實(shí)現(xiàn)rabbitmq
集群在k8s
中的自動(dòng)化部署,因此低于3.7.X版本請(qǐng)使用rabbitmq-autocluster
3.1 環(huán)境介紹
本文部署的版本是3.8.3
默認(rèn)部署在default
命名空間下,
持久化存儲(chǔ)為storageclass
動(dòng)態(tài)存儲(chǔ),底層為nfs
提供
鏡像地址rabbitmq:3.8.3-management
以下yaml
參考自官方示例https://github.com/rabbitmq/diy-kubernetes-examples
3.2 創(chuàng)建configmap
01-rabbitmq-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:name: rabbitmq-cluster-confignamespace: defaultlabels:addonmanager.kubernetes.io/mode: Reconcile
data:enabled_plugins: |[rabbitmq_management,rabbitmq_peer_discovery_k8s].rabbitmq.conf: |default_user = admindefault_pass = 123!@### Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more.cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8scluster_formation.k8s.host = kubernetes.default.svc.cluster.local## Should RabbitMQ node name be computed from the pod's hostname or IP address?## IP addresses are not stable, so using [stable] hostnames is recommended when possible.## Set to "hostname" to use pod hostnames.## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME## environment variable.cluster_formation.k8s.address_type = hostname## How often should node cleanup checks run?cluster_formation.node_cleanup.interval = 30## Set to false if automatic removal of unknown/absent nodes## is desired. This can be dangerous, see## * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup## * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJcluster_formation.node_cleanup.only_log_warning = truecluster_partition_handling = autoheal## See https://www.rabbitmq.com/ha.html#master-migration-data-localityqueue_master_locator=min-masters## See https://www.rabbitmq.com/access-control.html#loopback-usersloopback_users.guest = falsecluster_formation.randomized_startup_delay_range.min = 0cluster_formation.randomized_startup_delay_range.max = 2# default is rabbitmq-cluster's namespace# hostname_suffixcluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.default.svc.cluster.local# memoryvm_memory_high_watermark.absolute = 1GB# diskdisk_free_limit.absolute = 2GB
部分參數(shù)說(shuō)明:
- enabled_plugins:聲明開(kāi)啟的插件名
- default_pass/default_pass:聲明用戶名和密碼(雖然有部分文章記錄可以通過(guò)環(huán)境變量的方式聲明,但是經(jīng)測(cè)試,針對(duì)此版本如果指定了
configmap
即rabbitmq
的配置文件,聲明的環(huán)境變量是沒(méi)有用的,都需要在配置文件中指定) - cluster_formation.k8s.address_type:從
k8s
返回的Pod
容器列表中計(jì)算對(duì)等節(jié)點(diǎn)列表,這里只能使用主機(jī)名,官方示例中是ip
,但是默認(rèn)情況下在k8s
中pod
的ip
都是不固定的,因此可能導(dǎo)致節(jié)點(diǎn)的配置和數(shù)據(jù)丟失,后面的yaml
中會(huì)通過(guò)引用元數(shù)據(jù)的方式固定pod
的主機(jī)名。
更多參數(shù)請(qǐng)參考官方文檔,這里就不贅述了
3.3 創(chuàng)建service
02-rabbitmq-service.yaml
kind: Service
apiVersion: v1
metadata:labels:app: rabbitmq-clustername: rabbitmq-clusternamespace: default
spec:clusterIP: Noneports:- name: rmqportport: 5672targetPort: 5672selector:app: rabbitmq-cluster---
kind: Service
apiVersion: v1
metadata:labels:app: rabbitmq-clustername: rabbitmq-cluster-managenamespace: default
spec:ports:- name: httpport: 15672protocol: TCPtargetPort: 15672selector:app: rabbitmq-clustertype: NodePort
上面定義了兩個(gè)Service
,一個(gè)是rabbitmq
的服務(wù)端口,一個(gè)是管理界面的端口,用戶外部訪問(wèn),這里通過(guò)NodePort
方式進(jìn)行暴露
3.4 創(chuàng)建rbac授權(quán)
前面的介紹中提到了RabbitMQ
通過(guò)插件與k8s apiserver交互獲得集群中節(jié)點(diǎn)相關(guān)信息,因此需要對(duì)其進(jìn)行RBAC
授權(quán)
03-rabbitmq-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: rabbitmq-clusternamespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: rabbitmq-clusternamespace: default
rules:
- apiGroups: [""]resources: ["endpoints"]verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: rabbitmq-clusternamespace: default
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: rabbitmq-cluster
subjects:
- kind: ServiceAccountname: rabbitmq-clusternamespace: default
3.5 創(chuàng)建statefulset
RabbitMQ
在k8s
中作為一個(gè)有狀態(tài)應(yīng)用進(jìn)行部署,因此控制器類(lèi)型為StatefulSet
,yaml
中還定義了pvc
相關(guān)內(nèi)容
04-rabbitmq-cluster-sts.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:labels:app: rabbitmq-clustername: rabbitmq-clusternamespace: default
spec:replicas: 3selector:matchLabels:app: rabbitmq-clusterserviceName: rabbitmq-clustertemplate:metadata:labels:app: rabbitmq-clusterspec:containers:- args:- -c- cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.shrabbitmq-servercommand:- shenv:- name: TZvalue: 'Asia/Shanghai'- name: RABBITMQ_ERLANG_COOKIEvalue: 'SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY='- name: K8S_SERVICE_NAMEvalue: rabbitmq-cluster- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local- name: RABBITMQ_CONFIG_FILEvalue: /var/lib/rabbitmq/rabbitmq.confimage: rabbitmq:3.8.3-managementimagePullPolicy: IfNotPresentlivenessProbe:exec:command:- rabbitmq-diagnostics- status# See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.initialDelaySeconds: 60periodSeconds: 60timeoutSeconds: 15name: rabbitmqports:- containerPort: 15672name: httpprotocol: TCP- containerPort: 5672name: amqpprotocol: TCPreadinessProbe:exec:command:- rabbitmq-diagnostics- statusinitialDelaySeconds: 20periodSeconds: 60timeoutSeconds: 10volumeMounts:- mountPath: /etc/rabbitmqname: config-volumereadOnly: false- mountPath: /var/lib/rabbitmqname: rabbitmq-storagereadOnly: false- name: timezonemountPath: /etc/localtimereadOnly: trueserviceAccountName: rabbitmq-clusterterminationGracePeriodSeconds: 30volumes:- name: config-volumeconfigMap:items:- key: rabbitmq.confpath: rabbitmq.conf- key: enabled_pluginspath: enabled_pluginsname: rabbitmq-cluster-config- name: timezonehostPath:path: /usr/share/zoneinfo/Asia/ShanghaivolumeClaimTemplates:- metadata:name: rabbitmq-storagespec:accessModes:- ReadWriteManystorageClassName: "managed-nfs-storage"resources:requests:storage: 2Gi
3.6 部署檢查
? rabbitmq-cluster pwd
/Users/ssgeek/Documents/k8s-manifests/rabbitmq-cluster
? rabbitmq-cluster ls
01-rabbitmq-configmap.yaml 02-rabbitmq-service.yaml 03-rabbitmq-rbac.yaml 04-rabbitmq-cluster-sts.yaml
? rabbitmq-cluster kubectl apply -f .
configmap/rabbitmq-cluster-config created
service/rabbitmq-cluster created
service/rabbitmq-cluster-manage created
serviceaccount/rabbitmq-cluster created
role.rbac.authorization.k8s.io/rabbitmq-cluster created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster created
statefulset.apps/rabbitmq-cluster created
等待一段時(shí)間,查看創(chuàng)建的相關(guān)資源
? rabbitmq-cluster kubectl get po,sts -l app=rabbitmq-cluster
NAME READY STATUS RESTARTS AGE
pod/rabbitmq-cluster-0 1/1 Running 0 3m22s
pod/rabbitmq-cluster-1 1/1 Running 0 2m18s
pod/rabbitmq-cluster-2 1/1 Running 0 2m23sNAME READY AGE
statefulset.apps/rabbitmq-cluster 3/3 3m25s
查看日志,從日志的最后部分觀察集群建立的狀態(tài)
? rabbitmq-cluster kubectl logs -f rabbitmq-cluster-0
'/etc/rabbitmq/rabbitmq.conf' -> '/var/lib/rabbitmq/rabbitmq.conf'
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: list of feature flags found:
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] drop_unroutable_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] empty_basic_get_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] implicit_default_bindings
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] quorum_queue
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] virtual_host_metadata
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: feature flag states written to disk: yes
2021-02-17 03:30:39.544 [info] <0.269.0> ra: meta data store initialised. 0 record(s) recovered
2021-02-17 03:30:39.547 [info] <0.274.0> WAL: recovering []
2021-02-17 03:31:10.676 [info] <0.313.0> Starting RabbitMQ 3.8.3 on Erlang 22.3.4.1Copyright (c) 2007-2020 Pivotal Software, Inc.Licensed under the MPL 1.1. Website: https://rabbitmq.com## ## RabbitMQ 3.8.3## ############ Copyright (c) 2007-2020 Pivotal Software, Inc.###### ############ Licensed under the MPL 1.1. Website: https://rabbitmq.comDoc guides: https://rabbitmq.com/documentation.htmlSupport: https://rabbitmq.com/contact.htmlTutorials: https://rabbitmq.com/getstarted.htmlMonitoring: https://rabbitmq.com/monitoring.htmlLogs: <stdout>Config file(s): /var/lib/rabbitmq/rabbitmq.confStarting broker...2021-02-17 03:31:10.678 [info] <0.313.0> node : rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.localhome dir : /var/lib/rabbitmqconfig file(s) : /var/lib/rabbitmq/rabbitmq.confcookie hash : H+IQL2spD4MDV4jPi7mMAg==log(s) : <stdout>database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
2021-02-17 03:31:10.695 [info] <0.313.0> Running boot step pre_boot defined by app rabbit...省略中間內(nèi)容2021-02-17 03:31:13.273 [info] <0.824.0> Statistics database started.
2021-02-17 03:31:13.273 [info] <0.823.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-02-17 03:31:13.467 [info] <0.9.0> Server startup complete; 5 plugins started.* rabbitmq_peer_discovery_k8s* rabbitmq_management* rabbitmq_web_dispatch* rabbitmq_management_agent* rabbitmq_peer_discovery_commoncompleted with 5 plugins.
2021-02-17 03:32:30.060 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:32:31.264 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:33:31.280 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:33:32.627 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up
進(jìn)入到pod
中通過(guò)客戶端查看集群狀態(tài)
? rabbitmq-cluster kubectl exec -it rabbitmq-cluster-0 bash
root@rabbitmq-cluster-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local ...
BasicsCluster name: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.localDisk Nodesrabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.localRunning Nodesrabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.localVersionsrabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1Alarms(none)Network Partitions(none)ListenersNode: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP APIFeature flagsFlag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled
通過(guò)NodePort
訪問(wèn)管理界面
kubectl get svc -l app=rabbitmq-cluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rabbitmq-cluster ClusterIP None <none> 5672/TCP 8m47s
rabbitmq-cluster-manage NodePort 10.1.239.191 <none> 15672:30888/TCP 8m47s
到這里,在
k8s
中手動(dòng)部署一個(gè)RabbitMQ
集群就完成啦~