最近總結一些診斷OCeanBase的一些經驗，出一個【OceanBase診斷調優(yōu)】專題，也歡迎大家貢獻自己的診斷OceanBase的方法。

1. 前言

obdiag定位為OceanBase敏捷診斷工具。1.2版本的obdiag支持診斷信息的一鍵收集，光有收集信息的能力，沒有分析能力怎么行，所以我們在obdiag的1.3.0版本加上了OB集群的日志分析能力。你可以一鍵去分析你的集群的OB日志，看看有沒有一些異常情況。

2. obdiag 日志分析設計

2.1 ?架構設計

主體架構還是依托于obdiag的集中式采集模式，當用戶發(fā)起obdiag 的分析的時候需要去各個節(jié)點上進行采集，將采集回來的數據集中進行分析處理。

2.2 obdiag執(zhí)行在線日志分析的時序圖

1. 用戶設置配置文件，配置文件的路徑在obdiag安裝目錄的config/config.yml中，主要是設置所要分析的OceanBase集群的ssh登陸信息，因為obdiag需要通過ssh方式去集群拉取日志到obdiag的節(jié)點上進行分析

2. 執(zhí)行obdiag analyze log <option> 命令

3. obdiag 接收到用戶的analyze命令后會去解析<option> 內的參數

4. obdiag解析完analyze參數后會啟動日志拉取的環(huán)節(jié)，拉取的節(jié)點是步驟一中用戶配置的，拉取的日志的時間范圍、過濾條件等都是步驟三<option>設定的

5. obdiag 發(fā)送遠程主機的執(zhí)行指令

6. 遠程執(zhí)行日志的grep或者cp命令來獲取日志

7. 符合條件的日志會統一放到臨時文件中，便于后續(xù)的回傳

8. 下載遠程主機上篩選出來的符合條件的日志

9. 下載完畢后，發(fā)送臨時文件清理指令

10. 遠程主機臨時文件會被清理

11. obdiag 對遠程主機拉取回來的日志文件進行分析，對于日志分析，主要規(guī)則是針對日志中的retcode進行分析，統計各retcode出現的次數、最早開始時間、最晚出現的時間以及其對應的trace_id的等信息

12. obdiag分析完日志后會在黑屏上打印出總覽的日志分析信息

13. obdiag分析日志的詳細信息會輸出到文件中

14. 用戶可以通過obdiag 輸出的文件地址查看詳細的日志分析報告

3. obdiag日志分析實踐

obdiag analyze <analyze type> [options]

analyze type?包含如下：

• log：一鍵分析 OceanBase 的日志。

3.1 obdiag analyze log

使用該命令可以一鍵在線分析 OceanBase 集群的日志，或者通過?--files?開啟離線分析模式。

• 本文所指的在線分析指的是 OceanBase 集群在線運行狀態(tài)，日志分布在各個 OBServer 節(jié)點上。
• 本文所指的離線分析模式是?--files?參數傳遞下，可以分析已經收集到機 obdiag 部署機器上的 OBServer 節(jié)點日志。
• 需要確保已經在 obdiag 配置文件?config.yml?中配置好需要收集節(jié)點的登錄信息。相關的詳細配置介紹，參見?obdiag 配置。

例子：

obdiag analyze log --scope observer --from 2023-10-08 10:25:00 --to 2023-10-08 11:30:00...
FileListInfo:
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node           | LogList                                                                                                                                                                                                               |
+================+=======================================================================================================================================================================================================================+
| xx.xx.xx.xx   | ['observer.log.20231008104204260', 'observer.log.20231008111305072', 'observer.log.20231008114410668', 'observer.log.wf.20231008104204260', 'observer.log.wf.20231008111305072', 'observer.log.wf.20231008114410668'] |
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
...Analyze OceanBase Online Log Summary:
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| Node           | Status    | FileName                                                                     |   ErrorCode | Message                                                                                                                       |   Count |
+================+===========+==============================================================================+=============+===============================================================================================================================+=========+
| xx.xx.xx.xx    | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008104204260       |       -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use |       2 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| xx.xx.xx.xx    | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008111305072       |       -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use |       8 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| xx.xx.xx.xx    | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008114410668       |       -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use |      10 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| xx.xx.xx.xx    | Completed | analyze_pack_20231008171201/xx_xx_xx_xx/observer.log.20231008114410668       |       -4009 | IO error                                                                                                                      |      20 |
+----------------+-----------+------------------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
For more details, please run cmd 'cat analyze_pack_20231008171201/result_details.txt'

快捷分析最近一段時間的日志：

在線分析最近一小時的日志，該指令執(zhí)行的時候會從遠程主機上拉取最近一小時的日志進行分析，診斷出出現過的錯誤
obdiag gather log --scope observer --since 1h# 在線分析最近 30 分鐘的日志，該指令執(zhí)行的時候會從遠程主機上拉取最近30分鐘的日志進行分析，診斷出出現過的錯誤
obdiag analyze log --scope observer --since 30m

離線分析日志：

ls -lh test/
-rw-r--r--  1 admin  staff   256M Oct  8 17:24 observer.log.20231008104204260
-rw-r--r--  1 admin  staff   256M Oct  8 17:24 observer.log.20231008111305072
-rw-r--r--  1 admin  staff   256M Oct  8 17:24 observer.log.20231008114410668
-rw-r--r--  1 admin  staff    18K Oct  8 17:24 observer.log.wf.20231008104204260
-rw-r--r--  1 admin  staff    19K Oct  8 17:24 observer.log.wf.20231008111305072
-rw-r--r--  1 admin  staff    18K Oct  8 17:24 observer.log.wf.20231008114410668obdiag analyze log --files test/Analyze OceanBase Offline Log Summary:
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| Node      | Status    | FileName                                                              |   ErrorCode | Message                                                                                                                       |   Count |
+===========+===========+=======================================================================+=============+===============================================================================================================================+=========+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008104204260 |       -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use |       2 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008111305072 |       -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use |       8 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008114410668 |       -5006 | You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use |      10 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
| 127.0.0.1 | Completed | analyze_pack_20231008172144/127_0_0_1_/observer.log.20231008114410668 |       -4009 | IO error                                                                                                                      |      20 |
+-----------+-----------+-----------------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+---------+
For more details, please run cmd 'cat analyze_pack_20231008172144/result_details.txt'

4. 附錄

• obdiag 下載地址：?OceanBase分布式數據庫-海量數據 筆筆算數
• obdiag 官方文檔：?OceanBase分布式數據庫-海量數據 筆筆算數
• obdiag github地址：?GitHub - oceanbase/oceanbase-diagnostic-tool: OceanBase Diagnostic Tool is designed to help OceanBase users quickly gather necessary information and analyze the cause of the problem.