同步數(shù)據(jù)需要注意的事項 前提條件

1 要同步的mysql 表必須包含主鍵?

2 mysql binlog 必須是row 模式

3 不支持程序運行過程中修改表結(jié)構(gòu)

4 要賦予連接mysql 賬號的權限 reload, replication super 權限? ?如果是root 權限則不需要?

安裝 go-mysql-transfer?

?
git clone??https://github.com/wj596/go-mysql-transfer.gitcd go-mysql-transfer
?go get
go build

1 設置mysql binlog模式為row?

同步數(shù)據(jù)位置? 查看mysql binlog 日志位置

確保binlog 日志為 on?

數(shù)據(jù)庫結(jié)構(gòu) 表 test?

app.yml 配置?

# mysql
addr: 192.168.217.134:3306
user: root
pass: 123456
charset : utf8
slave_id: 1001 #slave ID
flavor: mysql #mysql or mariadb,榛???mysqltarget: elasticsearch ##elasticsearch
es_addrs: 127.0.0.1:9200
es_version: 6 # 
#es_password:  #
#es_version:  #rule:- schema: gametable: lottery order_by_column: idcolumn_underscore_to_camel: false value_encoder: jsones_index: lottery es_mappings: - column: idfield: idtype: long- column: losefield: lose #estype: keyword- column: winfield: wintype: keyword- column: stakefield: staketype: double- column: oddsfield: oddstype: double

全局數(shù)據(jù)初始化

./go-mysql-transfer -stock

增量數(shù)據(jù)同步

修改mysql 的個別數(shù)據(jù)? 同步binlog到最新位置 。。。執(zhí)行同步操作

同步數(shù)據(jù)到最新位置，修改部分數(shù)據(jù)。。。

./go-mysql-transfer -position binlog.000015 2457?

[root@localhost data]# ./go-mysql-transfer -position binlog.000015 2457 
2024-04-27 10:14:36.564695 I | process id: 46615
2024-04-27 10:14:36.564905 I | GOMAXPROCS :2
2024-04-27 10:14:36.564909 I | source  mysql(192.168.217.134:3306)
2024-04-27 10:14:36.564912 I | destination elasticsearch(http://127.0.0.1:9200)
The current dump position is : binlog.000015 2457

執(zhí)行增量同步

./go-mysql-transfer -config app.yml

數(shù)據(jù)類型映射表

mysql數(shù)據(jù)類型	es數(shù)據(jù)類型
int	long?
varchar	keyword
decimal?	double?

??

報錯?

同步數(shù)據(jù)報錯

在? go-mysql-transfer 中執(zhí)行?

go get github.com/json-iterator/go@v1.1.12

報錯:? document missing??

2024-05-01 17:15:16.081082 I | dumper stopped
2024-05-01 17:15:16.083295 I | transfer run from position(binlog.000017 6375)
2024-05-01 17:15:16.289692 I | [_doc][6]: document missing
2024-05-01 17:15:17.086846 I | transfer stop
2024-05-01 17:15:17.088804 I | dumper stopped
2024-05-01 17:15:17.091393 I | transfer run from position(binlog.000017 6375)
2024-05-01 17:15:17.294877 I | [_doc][6]: document missing
2024-05-01 17:15:18.095177 I | transfer stop
2024-05-01 17:15:18.097077 I | dumper stopped
2024-05-01 17:15:18.099320 I | transfer run from position(binlog.000017 6375)
2024-05-01 17:15:18.302411 I | [_doc][6]: document missing
^C2024-05-01 17:15:18.704188 I | application stoped錛𸱳ignal: interrupt 
2024-05-01 17:15:19.102070 I | transfer stop
2024-05-01 17:15:19.103859 I | dumper stopped

_doc 默認的索引類型? 一個索引index 可以包含多個type

類型 type 包含多個文檔(document)

原因： es 中沒有數(shù)據(jù)，更新導致找不到對應的數(shù)據(jù) 因此需要先初始化數(shù)據(jù)到es 中?

./go-mysql-transfer -stock?