目錄

前言

核心流程函數(shù)調(diào)用路徑

GetReplicationAnalysis

故障類型和對應(yīng)的處理函數(shù)

?編輯

拓?fù)浣Y(jié)構(gòu)警告類型

核心流程總結(jié)

與MHA相比

---

前言

Orchestrator另外一個(gè)重要的功能是監(jiān)控集群，發(fā)現(xiàn)故障。根據(jù)從復(fù)制拓?fù)浔旧慝@得的信息，它可以識別各種故障場景。

核心流程函數(shù)調(diào)用路徑

ContinuousDiscovery
--> CheckAndRecover // 檢查恢復(fù)的入口函數(shù)
--> GetReplicationAnalysis  // 查詢SQL，根據(jù)實(shí)例的狀態(tài)確定故障或者警告類型。檢查復(fù)制問題  (dead master; unreachable master; 等)
--> executeCheckAndRecoverFunction  // 根據(jù)分析結(jié)果選擇正確的檢查和恢復(fù)函數(shù)。然后會同步執(zhí)行該函數(shù)。
--> getCheckAndRecoverFunction // 根據(jù)分析結(jié)果選擇正確的檢查和恢復(fù)函數(shù)。然后會同步執(zhí)行該函數(shù)。
--> runEmergentOperations // 
--> checkAndExecuteFailureDetectionProcesses //   嘗試往數(shù)據(jù)庫中插入這個(gè)故障發(fā)現(xiàn)記錄，然后執(zhí)行故障發(fā)現(xiàn)階段所需的操作，執(zhí)行 OnFailureDetectionProcesses （故障發(fā)現(xiàn)階段的）鉤子腳本
--> AttemptFailureDetectionRegistration //  嘗試往數(shù)據(jù)庫中插入故障發(fā)現(xiàn)記錄 ，如果失敗 意味著這個(gè)問題可能已經(jīng)被發(fā)現(xiàn)了，
--> checkAndRecoverDeadMaster  // 根據(jù)故障情況 執(zhí)行恢復(fù)，這里選擇DeadMaster 故障類型舉例 
--> AttemptRecoveryRegistration // 嘗試往數(shù)據(jù)庫中插入一條恢復(fù)記錄；如果這一嘗試失敗，那么意味著恢復(fù)已經(jīng)在進(jìn)行中。
--> recoverDeadMaster //  用于恢復(fù)DeadMaster故障類型的函數(shù)，其中包含了完整的邏輯。會執(zhí)行PreFailoverProcesses 鉤子腳本

GetReplicationAnalysis

該函數(shù)將檢查復(fù)制問題 (dead master; unreachable master; 等），根據(jù)實(shí)例的狀態(tài)確定故障或者警告類型。

該函數(shù)會運(yùn)行一個(gè)復(fù)雜SQL ,該SQL會每秒執(zhí)行一次 ，執(zhí)行間隔是由定時(shí)器 recoveryTick決定 ，SQL中已經(jīng)進(jìn)行了一部分邏輯處理，并將數(shù)據(jù)存儲到?ReplicationAnalysis 結(jié)構(gòu)體中，

SELECTmaster_instance.hostname,master_instance.port,master_instance.read_only AS read_only,MIN(master_instance.data_center) AS data_center,MIN(master_instance.region) AS region,MIN(master_instance.physical_environment) AS physical_environment,MIN(master_instance.master_host) AS master_host,MIN(master_instance.master_port) AS master_port,MIN(master_instance.cluster_name) AS cluster_name,MIN(master_instance.binary_log_file) AS binary_log_file,MIN(master_instance.binary_log_pos) AS binary_log_pos,MIN(IFNULL(master_instance.binary_log_file = database_instance_stale_binlog_coordinates.binary_log_fileAND master_instance.binary_log_pos = database_instance_stale_binlog_coordinates.binary_log_posAND database_instance_stale_binlog_coordinates.first_seen < NOW() - interval 10 second,0)) AS is_stale_binlog_coordinates,MIN(IFNULL(cluster_alias.alias,master_instance.cluster_name)) AS cluster_alias,MIN(IFNULL(cluster_domain_name.domain_name,master_instance.cluster_name)) AS cluster_domain,MIN(master_instance.last_checked <= master_instance.last_seenand master_instance.last_attempted_check <= master_instance.last_seen + interval 6 second) = 1 AS is_last_check_valid,/* To be considered a master, traditional async replication must not be present/valid AND the host should either *//* not be a replication group member OR be the primary of the replication group */MIN(master_instance.last_check_partial_success) as last_check_partial_success,MIN((master_instance.master_host IN ('', '_')OR master_instance.master_port = 0OR substr(master_instance.master_host, 1, 2) = '//')AND (master_instance.replication_group_name = ''OR master_instance.replication_group_member_role = 'PRIMARY')) AS is_master,MIN(master_instance.replication_group_name != ''AND master_instance.replication_group_member_state != 'OFFLINE') AS is_replication_group_member,MIN(master_instance.is_co_master) AS is_co_master,MIN(CONCAT(master_instance.hostname,':',master_instance.port) = master_instance.cluster_name) AS is_cluster_master,MIN(master_instance.gtid_mode) AS gtid_mode,COUNT(replica_instance.server_id) AS count_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seen),0) AS count_valid_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running != 0AND replica_instance.slave_sql_running != 0),0) AS count_valid_replicating_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running = 0AND replica_instance.last_io_error like '%error %connecting to master%'AND replica_instance.slave_sql_running = 1),0) AS count_replicas_failing_to_connect_to_master,MIN(master_instance.replication_depth) AS replication_depth,GROUP_CONCAT(concat(replica_instance.Hostname,':',replica_instance.Port)) as slave_hosts,MIN(master_instance.slave_sql_running = 1AND master_instance.slave_io_running = 0AND master_instance.last_io_error like '%error %connecting to master%') AS is_failing_to_connect_to_master,MIN(master_downtime.downtime_active is not nulland ifnull(master_downtime.end_timestamp, now()) > now()) AS is_downtimed,MIN(IFNULL(master_downtime.end_timestamp, '')) AS downtime_end_timestamp,MIN(IFNULL(unix_timestamp() - unix_timestamp(master_downtime.end_timestamp),0)) AS downtime_remaining_seconds,MIN(master_instance.binlog_server) AS is_binlog_server,MIN(master_instance.pseudo_gtid) AS is_pseudo_gtid,MIN(master_instance.supports_oracle_gtid) AS supports_oracle_gtid,MIN(master_instance.semi_sync_master_enabled) AS semi_sync_master_enabled,MIN(master_instance.semi_sync_master_wait_for_slave_count) AS semi_sync_master_wait_for_slave_count,MIN(master_instance.semi_sync_master_clients) AS semi_sync_master_clients,MIN(master_instance.semi_sync_master_status) AS semi_sync_master_status,SUM(replica_instance.is_co_master) AS count_co_master_replicas,SUM(replica_instance.oracle_gtid) AS count_oracle_gtid_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.oracle_gtid != 0),0) AS count_valid_oracle_gtid_replicas,SUM(replica_instance.binlog_server) AS count_binlog_server_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.binlog_server != 0),0) AS count_valid_binlog_server_replicas,SUM(replica_instance.semi_sync_replica_enabled) AS count_semi_sync_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.semi_sync_replica_enabled != 0),0) AS count_valid_semi_sync_replicas,MIN(master_instance.mariadb_gtid) AS is_mariadb_gtid,SUM(replica_instance.mariadb_gtid) AS count_mariadb_gtid_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.mariadb_gtid != 0),0) AS count_valid_mariadb_gtid_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updates),0) AS count_logging_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updatesAND replica_instance.binlog_format = 'STATEMENT'),0) AS count_statement_based_logging_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updatesAND replica_instance.binlog_format = 'MIXED'),0) AS count_mixed_based_logging_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updatesAND replica_instance.binlog_format = 'ROW'),0) AS count_row_based_logging_replicas,IFNULL(SUM(replica_instance.sql_delay > 0),0) AS count_delayed_replicas,IFNULL(SUM(replica_instance.slave_lag_seconds > 10),0) AS count_lagging_replicas,IFNULL(MIN(replica_instance.gtid_mode), '') AS min_replica_gtid_mode,IFNULL(MAX(replica_instance.gtid_mode), '') AS max_replica_gtid_mode,IFNULL(MAX(case when replica_downtime.downtime_active is not nulland ifnull(replica_downtime.end_timestamp, now()) > now() then '' else replica_instance.gtid_errant end),'') AS max_replica_gtid_errant,IFNULL(SUM(replica_downtime.downtime_active is not nulland ifnull(replica_downtime.end_timestamp, now()) > now()),0) AS count_downtimed_replicas,COUNT(DISTINCT case when replica_instance.log_binAND replica_instance.log_slave_updates then replica_instance.major_version else NULL end) AS count_distinct_logging_major_versionsFROMdatabase_instance master_instanceLEFT JOIN hostname_resolve ON (master_instance.hostname = hostname_resolve.hostname)LEFT JOIN database_instance replica_instance ON (COALESCE(hostname_resolve.resolved_hostname,master_instance.hostname) = replica_instance.master_hostAND master_instance.port = replica_instance.master_port)LEFT JOIN database_instance_maintenance ON (master_instance.hostname = database_instance_maintenance.hostnameAND master_instance.port = database_instance_maintenance.portAND database_instance_maintenance.maintenance_active = 1)LEFT JOIN database_instance_stale_binlog_coordinates ON (master_instance.hostname = database_instance_stale_binlog_coordinates.hostnameAND master_instance.port = database_instance_stale_binlog_coordinates.port)LEFT JOIN database_instance_downtime as master_downtime ON (master_instance.hostname = master_downtime.hostnameAND master_instance.port = master_downtime.portAND master_downtime.downtime_active = 1)LEFT JOIN database_instance_downtime as replica_downtime ON (replica_instance.hostname = replica_downtime.hostnameAND replica_instance.port = replica_downtime.portAND replica_downtime.downtime_active = 1)LEFT JOIN cluster_alias ON (cluster_alias.cluster_name = master_instance.cluster_name)LEFT JOIN cluster_domain_name ON (cluster_domain_name.cluster_name = master_instance.cluster_name)WHEREdatabase_instance_maintenance.database_instance_maintenance_id IS NULLAND '' IN ('', master_instance.cluster_name)GROUP BYmaster_instance.hostname,master_instance.portHAVING(MIN(master_instance.last_checked <= master_instance.last_seenand master_instance.last_attempted_check <= master_instance.last_seen + interval 6 second) = 1/* AS is_last_check_valid */) = 0OR (IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running = 0AND replica_instance.last_io_error like '%error %connecting to master%'AND replica_instance.slave_sql_running = 1),0)/* AS count_replicas_failing_to_connect_to_master */> 0)OR (IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seen),0)/* AS count_valid_replicas */< COUNT(replica_instance.server_id)/* AS count_replicas */)OR (IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running != 0AND replica_instance.slave_sql_running != 0),0)/* AS count_valid_replicating_replicas */< COUNT(replica_instance.server_id)/* AS count_replicas */)OR (MIN(master_instance.slave_sql_running = 1AND master_instance.slave_io_running = 0AND master_instance.last_io_error like '%error %connecting to master%')/* AS is_failing_to_connect_to_master */)OR (COUNT(replica_instance.server_id)/* AS count_replicas */> 0)ORDER BYis_master DESC,is_cluster_master DESC,count_replicas DESC\G

故障類型和對應(yīng)的處理函數(shù)

主要根據(jù)結(jié)構(gòu)體?ReplicationAnalysis 中值判斷故障類型。

故障類型	處理函數(shù)
NoProblem	沒有
DeadMasterWithoutReplicas	沒有
DeadMaster	checkAndRecoverDeadMaster
DeadMasterAndReplicas	checkAndRecoverGenericProblem
DeadMasterAndSomeReplicas	checkAndRecoverDeadMaster
UnreachableMasterWithLaggingReplicas	checkAndRecoverGenericProblem
UnreachableMaster	checkAndRecoverGenericProblem
MasterSingleReplicaNotReplicating
MasterSingleReplicaDead
AllMasterReplicasNotReplicating	checkAndRecoverGenericProblem
AllMasterReplicasNotReplicatingOrDead	checkAndRecoverGenericProblem
LockedSemiSyncMasterHypothesis
LockedSemiSyncMaster	checkAndRecoverLockedSemiSyncMaster
MasterWithTooManySemiSyncReplicas	checkAndRecoverMasterWithTooManySemiSyncReplicas
MasterWithoutReplicas
DeadCoMaster	checkAndRecoverDeadCoMaster
DeadCoMasterAndSomeReplicas	checkAndRecoverDeadCoMaster
UnreachableCoMaster
AllCoMasterReplicasNotReplicating
DeadIntermediateMaster	checkAndRecoverDeadIntermediateMaster
DeadIntermediateMasterWithSingleReplica	checkAndRecoverDeadIntermediateMaster
DeadIntermediateMasterWithSingleReplicaFailingToConnect	checkAndRecoverDeadIntermediateMaster
DeadIntermediateMasterAndSomeReplicas	checkAndRecoverDeadIntermediateMaster
DeadIntermediateMasterAndReplicas	checkAndRecoverGenericProblem
UnreachableIntermediateMasterWithLaggingReplicas	checkAndRecoverGenericProblem
UnreachableIntermediateMaster
AllIntermediateMasterReplicasFailingToConnectOrDead	checkAndRecoverDeadIntermediateMaster
AllIntermediateMasterReplicasNotReplicating
FirstTierReplicaFailingToConnectToMaster
BinlogServerFailingToConnectToMaster
// Group replication problems
DeadReplicationGroupMemberWithReplicas	checkAndRecoverDeadGroupMemberWithReplicas

?不知道為什么從表格中粘貼到博客中每一列的寬度不能調(diào)整，這里粘貼為圖片，大家湊合看吧

拓?fù)浣Y(jié)構(gòu)警告類型

• StatementAndMixedLoggingReplicasStructureWarning
• StatementAndRowLoggingReplicasStructureWarning
• MixedAndRowLoggingReplicasStructureWarning
• MultipleMajorVersionsLoggingReplicasStructureWarning
• NoLoggingReplicasStructureWarning
• DifferentGTIDModesStructureWarning
• ErrantGTIDStructureWarning
• NoFailoverSupportStructureWarning
• NoWriteableMasterStructureWarning
• NotEnoughValidSemiSyncReplicasStructureWarning

核心流程總結(jié)

在實(shí)例發(fā)現(xiàn)階段已經(jīng)將實(shí)例的一些基礎(chǔ)信息存儲到了后臺管理數(shù)據(jù)庫的表中，故障發(fā)現(xiàn)階段核心流程：

1. 執(zhí)行一個(gè)復(fù)雜SQL，查詢實(shí)例和拓?fù)涞臓顟B(tài)，在SQL中已經(jīng)進(jìn)行了很多邏輯處理，狀態(tài)對比。
2. 將該SQL的查詢結(jié)果存儲到結(jié)構(gòu)體ReplicationAnalysis中。
3. 根據(jù)結(jié)構(gòu)體ReplicationAnalysis中的字段進(jìn)行條件判斷，確定故障/失敗類型或警告類型。
4. 在恢復(fù)節(jié)點(diǎn)就會根據(jù)故障類型選擇對應(yīng)的故障處理函數(shù)

與MHA相比

MHA探活

核心功能主要有 MHA::HealthCheck::wait_until_unreachable?實(shí)現(xiàn)：

1. 該函數(shù)通過一個(gè)死循環(huán)，檢測 4 次，每次 sleep ping_interval 秒（這個(gè)值在配置文件指定,參數(shù)是 ping_interval），持續(xù)四次失敗，就認(rèn)為數(shù)據(jù)已經(jīng)宕機(jī)；

2. 如果有二路檢測腳本，需要二路檢測腳本檢測主庫宕機(jī)，才是真正的宕機(jī)，否則只是推出死循環(huán)，結(jié)束檢測，不切換

3. 通過添加鎖來保護(hù)數(shù)據(jù)庫的訪問，防止腳本多次啟動；

4. 該函數(shù)可調(diào)用三種檢測方法：ping_select、ping_insert、ping_connect

MHA探活流程圖

	MHA	OC
故障類型	故障類型單一， 主要就是主庫是否存活	故障類型豐富 有多種故障與警告類型 見上面的總結(jié)
探活節(jié)點(diǎn)	支持多個(gè)節(jié)點(diǎn)探活 即manager服務(wù)器與二次檢查腳本中定義的服務(wù)器	多個(gè)OC節(jié)點(diǎn) 以及 配合該實(shí)例的從庫復(fù)制狀態(tài)是否正常進(jìn)行檢測
探活方式	ping_select、 ping_insert、 ping_connect	查詢被管理的數(shù)據(jù)庫，將數(shù)據(jù)寫入到OC的后臺管理數(shù)據(jù)庫中，進(jìn)行時(shí)間戳的比較或狀態(tài)值比較等方式
探活間隔	默認(rèn)3秒	默認(rèn)每1秒
探活次數(shù)	4次	1次
故障探測速度	慢，需要多次探活。	快
探活理念	多次多個(gè)節(jié)點(diǎn)避免網(wǎng)絡(luò)抖動等	配合從副本的主從狀態(tài)以及OC節(jié)點(diǎn)的探活