做網(wǎng)站造假域名推薦
harbor（docker倉庫）倉庫部署 - 高可用
- 1. harbor高可用
- 1.1 方案說明
- 1. 雙主復(fù)制
- 2. 多harbor實(shí)例共享后端存儲(chǔ)
- 1.2 部署高可用（多harbor實(shí)例共享后端存儲(chǔ)）
- 1. 服務(wù)器劃分
- 2. 安裝harbor（先部署一套Harbor，用于將其所有表結(jié)構(gòu)導(dǎo)出）
- 3. 安裝Postgresql
- 4. 安裝nginx
- 5. 安裝nfs
- 6. 安裝redis
- 7. 部署harbor
- 8. 修改nginx配置
- 9. docker登錄harbor
- 10. harbor修改
- 11. 修改nginx
- 12. docker推送
1. harbor高可用
目前有兩種主流的方案來解決這個(gè)問題：
- 雙主復(fù)制
- 多harbor實(shí)例共享后端存儲(chǔ)
1.1 方案說明
1. 雙主復(fù)制
所謂的雙主復(fù)制其實(shí)就是復(fù)用主從同步實(shí)現(xiàn)兩個(gè)harbor節(jié)點(diǎn)之間的雙向同步，來保證數(shù)據(jù)的一致性，然后在兩臺(tái)harbor前端頂一個(gè)負(fù)載均衡器將進(jìn)來的請(qǐng)求分流到不同的實(shí)例中去，只要有一個(gè)實(shí)例中有了新的鏡像，就是自動(dòng)的同步復(fù)制到另外的的實(shí)例中去，這樣實(shí)現(xiàn)了負(fù)載均衡，也避免了單點(diǎn)故障，在一定程度上實(shí)現(xiàn)了Harbor的高可用性：
這個(gè)方案有一個(gè)問題就是有可能兩個(gè)Harbor實(shí)例中的數(shù)據(jù)不一致。假設(shè)如果一個(gè)實(shí)例A掛掉了，這個(gè)時(shí)候有新的鏡像進(jìn)來，那么新的鏡像就會(huì)在另外一個(gè)實(shí)例B中，后面即使恢復(fù)了掛掉的A實(shí)例，Harbor實(shí)例B也不會(huì)自動(dòng)去同步鏡像，這樣只能手動(dòng)的先關(guān)掉Harbor實(shí)例B的復(fù)制策略，然后再開啟復(fù)制策略，才能讓實(shí)例B數(shù)據(jù)同步，讓兩個(gè)實(shí)例的數(shù)據(jù)一致。
在實(shí)際生產(chǎn)使用中，主從復(fù)制十分的不靠譜，所以這里就不配置了。
2. 多harbor實(shí)例共享后端存儲(chǔ)
利用共享存儲(chǔ)和共享數(shù)據(jù)庫來實(shí)現(xiàn)服務(wù)的高可用性和數(shù)據(jù)的冗余
這個(gè)方案在實(shí)際生產(chǎn)環(huán)境中部署需要考慮三個(gè)問題：
- 共享存儲(chǔ)的選取，Harbor的后端存儲(chǔ)目前支持AWS S3、Openstack Swift, Ceph等，在我們的實(shí)驗(yàn)環(huán)境里，就直接使用nfs。
- Session在不同的實(shí)例上共享，這個(gè)現(xiàn)在其實(shí)已經(jīng)不是問題了，在最新的harbor中，默認(rèn)session會(huì)存放在redis中，我們只需要將redis獨(dú)立出來即可?？梢酝ㄟ^redis sentinel或者redis cluster等方式來保證redis的可用性。在我們的實(shí)驗(yàn)環(huán)境里，仍然使用單臺(tái)redis。
- Harbor多實(shí)例數(shù)據(jù)庫問題，這個(gè)也只需要將harbor中的數(shù)據(jù)庫拆出來獨(dú)立部署即可。讓多實(shí)例共用一個(gè)外部數(shù)據(jù)庫，數(shù)據(jù)庫的高可用也可以通過數(shù)據(jù)庫的高可用方案保證?？蛇x擇的數(shù)據(jù)庫包括PostgreSql，mysql等等。
1.2 部署高可用（多harbor實(shí)例共享后端存儲(chǔ)）
1. 服務(wù)器劃分
| 服務(wù)器 | IP | 說明 |
|---|---|---|
| k8s-harbor-01.xx.net | 192.168.17.220 | harbor1服務(wù)器 |
| k8s-harbor-02.xx.net | 192.168.17.221 | harbor2服務(wù)器 |
| k8s-harbor-lb-01.xx.net | 192.168.17.225 | nginx,redis,mysql,nfs |
我們將在k8s-harbor-lb-01.xx.net部署nginx，redis，mysql，nfs等服務(wù)，生產(chǎn)環(huán)境中應(yīng)該分開，并且配置成為高可用
2. 安裝harbor（先部署一套Harbor，用于將其所有表結(jié)構(gòu)導(dǎo)出）
下載安裝包并上傳到服務(wù)器
tar xvf harbor-offline-installer-v2.7.2.tgz
cd harbormkdir certs #證書
cd certs/
openssl genrsa -out ./harbor-ca.key #key
openssl req -x509 -new -nodes -key ./harbor-ca.key -subj "/CN=harbor.xx.net" -days 7120 -out ./harbor-ca.crt #認(rèn)證
配置harbor.yml
cp harbor.yml.tmpl harbor.yml
[root@k8s-harbor-01 harbor]# egrep -v '^$|^#|^ #' harbor.yml
hostname: harbor.xx.net
http:port: 80
https:port: 443certificate: /opt/harbor/certs/harbor-ca.crtprivate_key: /opt/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
...
啟動(dòng)harbor
./install.sh --with-trivy --with-chartmuseum
--with-trivy #鏡像漏洞檢測(cè)
--with-chartmuseum #Chart倉庫服務(wù)
本節(jié)搭建的目的是導(dǎo)出postgresql數(shù)據(jù)庫到其他服務(wù)器，接著導(dǎo)出數(shù)據(jù)庫
docker ps
docker exec -it harbor-db /bin/bash
進(jìn)入容器
## 執(zhí)行 psql 進(jìn)入數(shù)據(jù)庫
postgres [ / ]$ psql
psql (9.6.14)
Type "help" for help.## 查看當(dāng)前所有的數(shù)據(jù)庫，postgres、template0、template1為默認(rèn)數(shù)據(jù)庫
postgres=# \lList of databasesName | Owner | Encoding | Collate | Ctype | Access privileges
--------------+----------+----------+-------------+-------------+-----------------------notaryserver | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +| | | | | postgres=CTc/postgres+| | | | | server=CTc/postgresnotarysigner | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +| | | | | postgres=CTc/postgres+| | | | | signer=CTc/postgrespostgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | registry | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +| | | | | postgres=CTc/postgrestemplate1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +| | | | | postgres=CTc/postgres
(6 rows)postgres=# \q
## 導(dǎo)出表結(jié)構(gòu)及數(shù)據(jù)
postgres [ / ]$ pg_dump -U postgres registry > /tmp/registry.sql
postgres [ / ]$ pg_dump -U postgres notaryserver > /tmp/notaryserver.sql
postgres [ / ]$ pg_dump -U postgres notarysigner > /tmp/notarysigner.sql-U 數(shù)據(jù)庫用戶-p 訪問端口-f 指定文件，和 > 功能一樣-h 指定數(shù)據(jù)庫地址-s 表示只導(dǎo)出表結(jié)構(gòu)，不導(dǎo)數(shù)據(jù)
導(dǎo)出到宿主機(jī)
docker cp 8d69069a2cd7:/tmp/registry.sql ./
docker cp 8d69069a2cd7:/tmp/notaryserver.sql ./
docker cp 8d69069a2cd7:/tmp/notarysigner.sql ./
3. 安裝Postgresql
yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
yum install -y postgresql13-server/usr/pgsql-13/bin/postgresql-13-setup initdb #初始化數(shù)據(jù)庫# 修改遠(yuǎn)程訪問配置
vim /var/lib/pgsql/13/data/postgresql.conf
...
#將 listen_addresses = 'localhost' 修改為
listen_addresses = '*'# 添加信任的遠(yuǎn)程連接,生產(chǎn)中不要添加0.0.0.0
vim /var/lib/pgsql/13/data/pg_hba.conf
...
host all all 0.0.0.0/0 trust
# host all all 0.0.0.0/0 md5
# 最后一列如果是trust，則登錄pg不需要密碼，若為md5，則需要密碼# start and enable server
systemctl enable postgresql-13
systemctl start postgresql-13# 檢查服務(wù)是否啟動(dòng)成功
#ps看進(jìn)程 或 ss看端口號(hào)#給postgresql設(shè)置密碼，增強(qiáng)安全性
su - postgres
-bash-4.2$ psql
## 直接寫入新密碼
postgres=# \password
Enter new password for user "postgres":
Enter it again:
postgres=# \q#驗(yàn)證
psql -U postgres -h localhost
Password for user postgres:
psql (13.14)
Type "help" for help.postgres=# exit#把全部信任改為指定IP
[root@k8s-harbor-lb-01 ~]# tail -3 /var/lib/pgsql/13/data/pg_hba.conf
host all all 192.168.17.220/24 trust
host all all 192.168.17.221/24 trust
#host all all 0.0.0.0/0 trust
將備份的數(shù)據(jù)，導(dǎo)入進(jìn)單獨(dú)部署的postgresql中
## 創(chuàng)建數(shù)據(jù)庫
postgres=# CREATE DATABASE registry;
postgres=# CREATE DATABASE notaryserver;
postgres=# CREATE DATABASE notarysigner;
將harbor服務(wù)器的導(dǎo)出的SQL拷貝到本機(jī)
scp -r 192.168.17.220:/opt/harbor/postgresql_export /opt/postgresql_export
notaryserver.sql 100% 491 213.2KB/s 00:00
notarysigner.sql 100% 491 281.3KB/s 00:00
registry.sql 100% 101KB 19.7MB/s 00:00
導(dǎo)入數(shù)據(jù)
##
psql -h localhost -U postgres -p 5432 -d registry -f registry.sql
psql -h localhost -U postgres -p 5432 -d notaryserver -f notaryserver.sql
psql -h localhost -U postgres -p 5432 -d notarysigner -f notarysigner.sql -U 數(shù)據(jù)庫用戶-p 訪問端口-f 指定文件，和 < 功能一樣-h 指定數(shù)據(jù)庫地址-d 指定數(shù)據(jù)庫名
4. 安裝nginx
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
yum install -y nginx
配置nginx.conf
[root@k8s-harbor-lb-01 ~]# cat /etc/nginx/nginx.conf
...include /etc/nginx/conf.d/*.conf;upstream harborsvr {server 192.168.17.220:80 weight=2;server 192.168.17.221:80 weight=1;}server {listen 80;server_name 192.168.17.225;location / {proxy_pass http://harborsvr;}}
}
啟動(dòng)
systemctl start nginx
systemctl status nginx
systemctl enable nginx
5. 安裝nfs
yum install -y nfs-utils# 編輯/etc/exports文件
/data *(rw,no_root_squash)chmod 777 -R /datasystemctl start nfs-server
systemctl enable nfs-servermdkir /data/
mount -t nfs `hostname`:/data/ /data/
6. 安裝redis
yum install epel-release -y
yum install redis -y##
vim /etc/redis.conf
...
bind 0.0.0.0 # 設(shè)置所有主機(jī)可以連接
requirepass 123456 # 設(shè)置客戶端連接密碼
daemonize yes # 打開守護(hù)進(jìn)程模式
...## 啟動(dòng)redis
systemctl start redis
systemctl enable redis
7. 部署harbor
我們將第2段中，部署的harbor進(jìn)行修改配置文件
[root@k8s-harbor-01 harbor]# docker-compose down
編輯配置文件，需要更改的主要有以下幾點(diǎn)：
1.hostname 改為主機(jī)ip或完全限定域名，不要使用127.0.0.1或localhost
2.https選項(xiàng)，如需要，指定crt和key的路徑，若不需要，直接注釋掉
3.harbor_admin_password，默認(rèn)密碼，可以更改
4.data_volume，數(shù)據(jù)默認(rèn)存儲(chǔ)位置，設(shè)計(jì)為共享路徑
5.注釋掉database模塊 及 Clair模塊
6.開啟external_database 和 external_redis模塊及正確配置其中參數(shù)
7.集群內(nèi)所有harbor配置均一樣，改一下hostname值即可
修改配置文件（經(jīng)過后面掛掉之后，重新改的配置文件）
[root@k8s-harbor-01 harbor]# egrep -v '^$|^#|^ #' harbor.yml
hostname: harbor.xx.net
http:port: 80
https:port: 443certificate: /opt/harbor/certs/harbor-ca.crtprivate_key: /opt/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
data_volume: /data
trivy:ignore_unfixed: falseskip_update: falseoffline_scan: falsesecurity_check: vulninsecure: false
jobservice:max_job_workers: 10
notification:webhook_job_max_retry: 10
chart:absolute_url: disabled
log:level: infolocal:# Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.rotate_count: 50# Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.# If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G# are all valid.rotate_size: 200M# The directory on your host that store loglocation: /var/log/harbor
_version: 2.7.0
external_database:harbor:host: 192.168.17.225port: 5432db_name: registryusername: postgrespassword: 123456ssl_mode: disablemax_idle_conns: 2max_open_conns: 0notary_signer:host: 192.168.17.225port: 5432db_name: notarysignerusername: postgrespassword: 123456ssl_mode: disablenotary_server:host: 192.168.17.225port: 5432db_name: notaryserverusername: postgrespassword: 123456ssl_mode: disable
external_redis:host: 192.168.17.225:6379password: 123456registry_db_index: 1jobservice_db_index: 2chartmuseum_db_index: 3chair_db_index: 4trivy_db_index: 5idle_timeout_seconds: 30
proxy:http_proxy:https_proxy:no_proxy:components:- core- jobservice- trivy
metric:enabled: falseport: 9090path: /metrics
upload_purging:enabled: trueage: 168hinterval: 24hdryrun: false
cache:enabled: falseexpire_hours: 24啟動(dòng)harbor的過程中發(fā)現(xiàn)jobservice容器無法啟動(dòng)，懷疑是數(shù)據(jù)庫連接失敗
[root@k8s-harbor-01 harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
767ac8431315 goharbor/harbor-jobservice:v2.7.2 "/harbor/entrypoint.…" 18 hours ago Restarting (2) 17 seconds ago harbor-jobservice
...
日志
[root@k8s-harbor-01 harbor]# docker logs 767
Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
2024-02-21T17:00:15Z [ERROR] [/pkg/registry/client.go:82]: Failed to parse REGISTRY_HTTP_CLIENT_TIMEOUT: strconv.ParseInt: parsing "": invalid syntax, use default value: 30m0s
2024-02-21T17:00:15Z [INFO] [/controller/artifact/annotation/parser.go:71]: the annotation parser to parser artifact annotation version v1alpha1 registered
2024-02-21T17:00:15Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.wasm.config.v1+json registered
2024-02-21T17:00:15Z [ERROR] [/lib/config/config.go:81]: failed to get config manager
2024-02-21T17:00:15Z [ERROR] [/lib/config/config.go:81]: failed to get config manager
在harbor服務(wù)器安裝postgresql客戶端，
yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm -y
yum install postgresql13 -y[root@k8s-harbor-01 harbor]# psql -U postgres -h 192.168.17.225 -p 5432
psql (13.14)
Type "help" for help.postgres=# \lList of databasesName | Owner | Encoding | Collate | Ctype | Access privileges
--------------+----------+----------+-------------+-------------+-----------------------notaryserver | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |notarysigner | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |registry | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +| | | | | postgres=CTc/postgrestemplate1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +| | | | | postgres=CTc/postgres
(6 rows)postgres=#
發(fā)現(xiàn)可以連接，注釋掉緩存數(shù)據(jù)庫redis配置，發(fā)現(xiàn)可以正常啟動(dòng)，說明是redis配置有問題
[root@k8s-harbor-01 harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dccc714a7abe goharbor/harbor-jobservice:v2.7.2 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice
經(jīng)過調(diào)查，原來是redis的password修改過，和默認(rèn)的不匹配
harbor1服務(wù)器配置完成，接下來配置harbor2服務(wù)器，相同配置，改下域名就可以了
[root@k8s-harbor-02 harbor]# grep hostname harbor.yml
# The IP address or hostname to access admin UI and registry service.
hostname: harbor2.xx.net
8. 修改nginx配置
由于我們安裝的harbor通過http跳轉(zhuǎn)到https訪問，所以前面設(shè)置的nginx的負(fù)載均衡的配置需要進(jìn)行修改，否則無法訪問
[root@k8s-harbor-lb-01 nginx]# cat /etc/nginx/nginx.conf
...
http {include /etc/nginx/mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log /var/log/nginx/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;#gzip on;include /etc/nginx/conf.d/*.conf;upstream harborsvrs {server 192.168.17.220:443 weight=2;server 192.168.17.221:443 weight=1;}server {listen 443;server_name 192.168.17.225;location / {proxy_pass https://harborsvrs/;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-Proto $scheme;}}
}
修改后，重啟nginx
9. docker登錄harbor
[root@k8s-master-01 ansible]# docker login 192.168.17.225:443
Username: admin
Password:
Error response from daemon: Get "http://192.168.17.225:443/v2/": Get "https://harbor1.xx.net/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry": dial tcp: lookup harbor1.xx.net on 8.8.8.8:53: no such host
解析不了harbor的域名，在沒有DNS服務(wù)器的情況下，修改/etc/hosts文件
[root@k8s-master-01 ansible]# cat /etc/hosts
...
192.168.17.220 k8s-harbor-01.xx.net harbor1 harbor1.xx.net
192.168.17.221 k8s-harbor-02.xx.net harbor2 harbor2.xx.net
登錄
[root@k8s-master-01 ansible]# docker login 192.168.17.225:443
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
由于推送的時(shí)候出現(xiàn)問題，筆者一直未能解決，所以將harbor回退到使用http協(xié)議
[root@k8s-master-01 ansible]# docker push harbor2.xx.net/alpine/alpine:latest
The push refers to repository [harbor2.xx.net/alpine/alpine]
d4fc045c9e3a: Layer already exists
unauthorized: unauthorized to access repository: alpine/alpine, action: push: unauthorized to access repository: alpine/alpine, action: push
10. harbor修改
[root@k8s-harbor-01 harbor]# head -20 harbor.yml
# Configuration file of Harbor# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
#hostname: harbor1.xx.net
hostname: 192.168.17.220# http related config
http:# port for http, default is 80. If https enabled, this port will redirect to https portport: 80# https related config
#https:# https port for harbor, default is 443
# port: 443# The path of cert and key files for nginx
# certificate: /opt/harbor/certs/harbor-ca.crt
# private_key: /opt/harbor/certs/harbor-ca.key修改harbor1和harbor2服務(wù)器的配置文件，并重啟harbor
11. 修改nginx
http {include /etc/nginx/mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log /var/log/nginx/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;#gzip on;include /etc/nginx/conf.d/*.conf;upstream harborsvrs {server 192.168.17.220:80 weight=2;server 192.168.17.221:80 weight=1;}server {listen 80;server_name 192.168.17.225;autoindex on;location / {proxy_pass http://harborsvrs/;}}
}
通過負(fù)載均衡可以訪問harbor
12. docker推送
[root@k8s-master-01 ansible]# docker login 192.168.17.221:80 -u admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
[root@k8s-master-01 ansible]# docker tag alpine:latest 192.168.17.221:80/alpine/alpine:v1
[root@k8s-master-01 ansible]# docker push 192.168.17.221:80/alpine/alpine:v1
The push refers to repository [192.168.17.221:80/alpine/alpine]
d4fc045c9e3a: Layer already exists
v1: digest: sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 size: 528
[root@k8s-master-01 ansible]# docker login 192.168.17.220:80 -u admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
[root@k8s-master-01 ansible]# docker push 192.168.17.220:80/alpine/alpine:latest
The push refers to repository [192.168.17.220:80/alpine/alpine]
d4fc045c9e3a: Layer already exists
latest: digest: sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 size: 528
在其他服務(wù)器登錄harbor后，也可以正常推送
