Elasticsearch生产级平滑迁移切换主节点操作记录

163次阅读

共计 3502 个字符，预计需要花费 9 分钟才能阅读完成。

服务器信息

角色	服务器
主	server-es-1
node	server-es-2
node	server-es-3
node	server-newes-1
node	server-newes-2
node	server-newes-3

elasticsearch 版本: 7.6.2

服务器版本: Centos7.6

es-[1-3]是老的 es 服务器，newes-[1-3]是新的 es 服务器，这里需要把老的 es 平滑迁移到新的 es 服务器上面

Es 同步

curl -u 'elastic:XXXXXXX'  -X GET 'http://localhost:9200/_cluster/health'

首先需要新建目录并初始化环境 new-es 为 server-newes-1,2,3 服务器

ansible new-es -m shell -a 'echo"DefaultLimitNOFILE=65535">> /etc/systemd/system.conf && systemctl daemon-reexec'

ansible new-es -m shell -a 'echo"vm.max_map_count = 262144">> /etc/sysctl.conf && sysctl -p'
ansible new-es -m shell -a 'useradd elastic -s /sbin/nologin'

ansible new-es -m shell -a 'mkdir -p /data/elasticsearch-7.6.2/{data,logs}'

ansible new-es -m shell -a 'chown -R elastic:elastic  /data/elasticsearch-7.6.2'

在 server-es- 1 上面 rsync 同步数据到 server-newes-1,2,3，这里排除了数据目录和日志目录。

注意: 新扩容的 es 数据目录和 log 目录一定要为空，这里方便排查问题。

rsync -avz --exclude='data' --exclude='logs' /data/elasticsearch-7.6.2/ root@server-newes-1:/data/elasticsearch-7.6.2/ 
rsync -avz --exclude='data' --exclude='logs' /data/elasticsearch-7.6.2/ root@server-newes-2:/data/elasticsearch-7.6.2/ 
rsync -avz --exclude='data' --exclude='logs' /data/elasticsearch-7.6.2/ root@server-newes-3:/data/elasticsearch-7.6.2/

新 ES 配置文件写入

操作主机: server-newes-1,2,3

这里的配置文件需要根据自己主机的配置去调整。

cat > /data/elasticsearch-7.6.2/config/elasticsearch.yml <<EOF
cluster.name: es-cluster2
node.name: $(hostname)
cluster.max_shards_per_node: 10000
network.host: 0.0.0.0
path:
  data: "/data/elasticsearch-7.6.2/data"
  logs: "/data/elasticsearch-7.6.2/logs"
cluster.initial_master_nodes:
  - "service-es-1"
  - "server-newes-1"
  - "server-newes-2"
  - "server-newes-3"
discovery.seed_hosts:
  - service-es-1:9300
  - service-es-2:9300
  - service-es-3:9300
  - server-newes-1:9300
  - server-newes-2:9300
  - server-newes-3:9300
transport.profiles.default.port: 9300
bootstrap.system_call_filter: false
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: /.*/
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
EOF

cat > /usr/lib/systemd/system/elasticsearch.service << EOF
[Unit]
Description=elasticsearch
After=network.target
[Service]
Type=simple
User=elastic
Group=elastic
LimitNOFILE=100000
LimitNPROC=100000
Restart=no
ExecStart=/data/elasticsearch-7.6.2/bin/elasticsearch
PrivateTmp=true
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
EOF
chown -R elastic:elastic  /data/elasticsearch-7.6.2
systemctl daemon-reload
systemctl start elasticsearch
systemctl status elasticsearch

依次启动 server-newes-1,2,3 后查看集群状态和分片信息，等待集群副本同步以及集群状态为green

等待集群同步完成

# 查看健康状态
curl -u 'elastic:XXXXXXX'  -X GET 'http://localhost:9200/_cluster/health?pretty'
# 查看分片
curl -u 'elastic:XXXXXXX'  -X GET 'http://localhost:9200/_cat/shards'

依次排除老的 node 节点并等待分片转移完成

curl -u 'elastic:XXXXXXX' -XPUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d '{"transient": {"cluster.routing.allocation.exclude._name":"service-es-3"}
}'
# es-2
curl -u 'elastic:XXXXXXX' -XPUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d '{"transient": {"cluster.routing.allocation.exclude._name":"service-es-2"}
}'
# es-1
curl -u 'elastic:XXXXXXX' -XPUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d '{"transient": {"cluster.routing.allocation.exclude._name":"service-es-1"}
}'