linux 学习笔记-081-NoSQL-redis 集群介绍,redis 集群搭建配置,redis 集群操作

发布于 2018-05-07  209 次阅读


redis 集群介绍

多个 redis 节点网络互联,数据共享

所有的节点都是一主一从(可以是多个从),其中从不提供服务,仅作为备用

不支持同时处理多个键(如 mset/mget),因为 redis 需要把键均匀分布在各个节点上,并发量很高的情况下同时创建键值会降低性能并导致不可预测的行为。

支持在线增加、删除节点

客户端可以连任何一个主节点进行读写

linux 学习笔记-081-NoSQL-redis 集群介绍,redis 集群搭建配置,redis 集群操作

redis 集群搭建配置

场景设置:

两台机器,分别开启三个 Redis 服务(端口)

A 机器上三个端口 7000,7002,7004

B 机器上三个端口 7001,7003,7005

两台机器上都要编译安装 redis,然后编辑并复制 3 个不同的 redis.conf,分别设置不同的端口号、dir 等参数,还需要增加 cluster 相关参数,然后分别启动 6 个 redis 服务

具体 redis 配置文件可以参考:

https://coding.net/u/aminglinux/p/yuanke_centos7/git/tree/master/21NOSQL

01

先设置好 A 机器和 B 机器的共 6 个配置文件

[root@am-01:~#] vim /etc/redis_7000.conf

port 7000

bind 172.17.1.240

daemonize yes

pidfile /var/run/redis_7000.pid

dir /data/redis_data/7000

cluster-enabled yes

cluster-config-file nodes_7000.conf

cluster-node-timeout 10100

appendonly yes

[root@am-01:~#] cp /etc/redis_7000.conf /etc/redis_7002.conf

[root@am-01:~#] vim /etc/redis_7002.conf

port 7002

bind 172.17.1.240

daemonize yes

pidfile /var/run/redis_7002.pid

dir /data/redis_data/7002

cluster-enabled yes

cluster-config-file nodes_7002.conf

cluster-node-timeout 10100

#定义开启 cluster,"nodes_7002.conf"会在 dir 目录下生成

appendonly yes

[root@am-01:~#] cp /etc/redis_7002.conf /etc/redis_7004.conf

[root@am-01:~#] vim /etc/redis_7004.conf

port 7004

bind 172.17.1.240

daemonize yes

pidfile /var/run/redis_7004.pid

dir /data/redis_data/7004

cluster-enabled yes

cluster-config-file nodes_7004.conf

cluster-node-timeout 10100

appendonly yes
[root@am-02:~#] vim /etc/redis_7001.conf

port 7001

bind 172.17.1.242

daemonize yes

pidfile /var/run/redis_7001.pid

dir /data/redis_data/7001

cluster-enabled yes

cluster-config-file nodes_7001.conf

cluster-node-timeout 10100

appendonly yes

[root@am-02:~#] cp /etc/redis_7001.conf /etc/redis_7003.conf

[root@am-02:~#] cp /etc/redis_7001.conf /etc/redis_7005.conf

[root@am-02:~#] vim /etc/redis_7003.conf

port 7003

bind 172.17.1.242

daemonize yes

pidfile /var/run/redis_7003.pid

dir /data/redis_data/7003

cluster-enabled yes

cluster-config-file nodes_7003.conf

cluster-node-timeout 10100

appendonly yes

[root@am-02:~#] vim /etc/redis_7005.conf

port 7005

bind 172.17.1.242

daemonize yes

pidfile /var/run/redis_7005.pid

dir /data/redis_data/7005

cluster-enabled yes

cluster-config-file nodes_7005.conf

cluster-node-timeout 10100

appendonly yes

02

分别创建 dir 目录

[root@am-01:~#] mkdir -p /data/redis_data/{7000,7002,7004}

[root@am-01:~#] ls /data/redis_data/

7000  7002  7004
[root@am-02:~#] mkdir -p /data/redis_data/{7001,7003,7005}

[root@am-02:~#] ls /data/redis_data/

7001  7003  7005

03

启动 A 的 redis 服务

[root@am-01:~#] redis-server /etc/redis_7000.conf

45916:C 07 May 19:33:44.328 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

45916:C 07 May 19:33:44.328 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=45916, just started

45916:C 07 May 19:33:44.328 # Configuration loaded

[root@am-01:~#] redis-server /etc/redis_7002.conf

45922:C 07 May 19:33:55.831 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

45922:C 07 May 19:33:55.831 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=45922, just started

45922:C 07 May 19:33:55.831 # Configuration loaded

[root@am-01:~#] redis-server /etc/redis_7004.conf

45927:C 07 May 19:33:59.559 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

45927:C 07 May 19:33:59.559 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=45927, just started

45927:C 07 May 19:33:59.559 # Configuration loaded

[root@am-01:~#] ps aux | grep redis

root      45938  0.1  0.2 149364  2664 ?        Ssl  19:34   0:00 redis-server 172.17.1.240:7000 [cluster]

root      45943  0.2  0.2 149364  2660 ?        Ssl  19:34   0:00 redis-server 172.17.1.240:7002 [cluster]

root      45948  0.0  0.2 149364  2660 ?        Ssl  19:34   0:00 redis-server 172.17.1.240:7004 [cluster]

root      45953  0.0  0.0 112676   976 pts/0    S+   19:34   0:00 grep --color=auto redis

04

在 B 上编译安装 redis 并启动 redis 服务

[root@am-01:~#] scp -r /usr/local/src/redis-4.0.9 172.17.1.242:/usr/local/src/

#在 A 机器上把 redis 的源码包拷贝到 B 机器上
[root@am-02:~#] cd /usr/local/src/

[root@am-02:/usr/local/src#] cd redis-4.0.9/

[root@am-02:/usr/local/src/redis-4.0.9#] make install

cd src && make install

make[1]: 进入目录“/usr/local/src/redis-4.0.9/src”



Hint: It's a good idea to run 'make test' ; )



    INSTALL install

    INSTALL install

    INSTALL install

    INSTALL install

    INSTALL install

make[1]: 离开目录“/usr/local/src/redis-4.0.9/src”

#因为 redis 源码包已经在 A 机器上编译过,所以拷贝到 B 机器后只需要 make install

[root@am-02:/usr/local/src/redis-4.0.9#] redis-server /etc/redis_7001.conf

23384:C 07 May 22:43:55.438 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

23384:C 07 May 22:43:55.438 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=23384, just started

23384:C 07 May 22:43:55.438 # Configuration loaded

[root@am-02:/usr/local/src/redis-4.0.9#] redis-server /etc/redis_7003.conf

23389:C 07 May 22:43:58.785 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

23389:C 07 May 22:43:58.785 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=23389, just started

23389:C 07 May 22:43:58.785 # Configuration loaded

[root@am-02:/usr/local/src/redis-4.0.9#] redis-server /etc/redis_7005.conf

23394:C 07 May 22:44:01.510 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

23394:C 07 May 22:44:01.510 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=23394, just started

23394:C 07 May 22:44:01.510 # Configuration loaded

[root@am-02:/usr/local/src/redis-4.0.9#] ps aux | grep redis

root      23385  0.3  0.7 145268  7560 ?        Ssl  22:43   0:00 redis-server 172.17.1.242:7001 [cluster]

root      23390  0.3  0.7 145268  7556 ?        Ssl  22:43   0:00 redis-server 172.17.1.242:7003 [cluster]

root      23395  0.3  0.7 145268  7556 ?        Ssl  22:44   0:00 redis-server 172.17.1.242:7005 [cluster]

root      23400  0.0  0.0 112676   976 pts/0    S+   22:44   0:00 grep --color=auto redis

05

合理设置防火墙规则(我这里是清空了),关闭 selinux

[root@am-01:~#] iptables -nvL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination        



Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination        



Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination      

[root@am-01:~#] getenforce

Disabled
[root@am-02:~#] iptables -nvL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination        



Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination        



Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination        

[root@am-02:~#] getenforce

Disabled

06

受 redis 版本所限,这里需要安装 ruby2.2(选一台机器安装即可,这里选择 A)

[root@am-01:~#] yum -y groupinstall "Development Tools"

[root@am-01:~#] yum -y install gdbm-devel libdb4-devel libffi-devel libyaml libyaml-devel ncurses-devel openssl-devel readline-devel tcl-deve

#安装一些库

[root@am-01:~#] cd /root/

[root@am-01:~#] mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}

[root@am-01:~#] wget http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.3.tar.gz -P rpmbuild/SOURCES

#下载 ruby 源码包

[root@am-01:~#] wget https://raw.githubusercontent.com/tjinjin/automate-ruby-rpm/master/ruby22x.spec -P rpmbuild/SPECS

#下载 SPEC 文件,这个主要用来吧 ruby 的源码包变成 rpm 包

[root@am-01:~#] rpmbuild -bb rpmbuild/SPECS/ruby22x.spec

#编译 ruby 并打包成 rpm 包

[root@am-01:~#] yum -y localinstall rpmbuild/RPMS/x86_64/ruby-2.2.3-1.el7.centos.x86_64.rpm

#使用 yum 安装 ruby pm 包,这样能顺便解决一些依赖

[root@am-01:~#] gem install redis

#安装 redis 中与 ruby 相关的包

07

配置 redis 集群

[root@am-01:~#] cp /usr/local/src/redis-4.0.9/src/redis-trib.rb /usr/bin/

[root@am-01:~#] redis-trib.rb create --replicas 1 172.17.1.240:7000 172.17.1.240:7002 172.17.1.240:7004 172.17.1.242:7001 172.17.1.242:7003 172.17.1.242:7005

>>> Creating cluster

>>> Performing hash slots allocation on 6 nodes...

Using 3 masters:

172.17.1.240:7000

172.17.1.242:7001

172.17.1.240:7002

Adding replica 172.17.1.242:7005 to 172.17.1.240:7000

Adding replica 172.17.1.240:7004 to 172.17.1.242:7001

Adding replica 172.17.1.242:7003 to 172.17.1.240:7002

M: 7f638f3646df80d6a1e503464fbf41c2ad032252 172.17.1.240:7000

   slots:0-5460 (5461 slots) master

M: 3c140721fba6a6e0a273bc5c8d42702206cf3e80 172.17.1.240:7002

   slots:10923-16383 (5461 slots) master

S: b7cb7728962bd27f2db3ed7a7d1adb75175d6731 172.17.1.240:7004

   replicates acfe177f0df7677163e91834dab498783b31b88a

M: acfe177f0df7677163e91834dab498783b31b88a 172.17.1.242:7001

   slots:5461-10922 (5462 slots) master

S: fc871f25690fa2bcea9a8bd849a8c03ed447e28c 172.17.1.242:7003

   replicates 3c140721fba6a6e0a273bc5c8d42702206cf3e80

S: fdbfcbf5995684c1171558e900a8f71755b42339 172.17.1.242:7005

   replicates 7f638f3646df80d6a1e503464fbf41c2ad032252

Can I set the above configuration? (type 'yes' to accept): yes

>>> Nodes configuration updated

>>> Assign a different config epoch to each node

>>> Sending CLUSTER MEET messages to join the cluster

Waiting for the cluster to join..

>>> Performing Cluster Check (using node 172.17.1.240:7000)

M: 7f638f3646df80d6a1e503464fbf41c2ad032252 172.17.1.240:7000

   slots:0-5460 (5461 slots) master

   1 additional replica(s)

M: acfe177f0df7677163e91834dab498783b31b88a 172.17.1.242:7001

   slots:5461-10922 (5462 slots) master

   1 additional replica(s)

S: fdbfcbf5995684c1171558e900a8f71755b42339 172.17.1.242:7005

   slots: (0 slots) slave

   replicates 7f638f3646df80d6a1e503464fbf41c2ad032252

S: fc871f25690fa2bcea9a8bd849a8c03ed447e28c 172.17.1.242:7003

   slots: (0 slots) slave

   replicates 3c140721fba6a6e0a273bc5c8d42702206cf3e80

S: b7cb7728962bd27f2db3ed7a7d1adb75175d6731 172.17.1.240:7004

   slots: (0 slots) slave

   replicates acfe177f0df7677163e91834dab498783b31b88a

M: 3c140721fba6a6e0a273bc5c8d42702206cf3e80 172.17.1.240:7002

   slots:10923-16383 (5461 slots) master

   1 additional replica(s)

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

[root@am-01:~#] echo $?

0

#使用 redis-trib.rb 工具创建集群,可以见到会随机分配 3 个做主,3 个做从,最后见到两个 OK 关键字,即证明集群已经配置好

redis 集群操作

对 redis 集群做操作的时候,指定的端口号可以是主的其中一个端口,也可以是从的其中一个端口

01

以集群方式连接 redis,并创建一些 key 来体验一下 redis 集群

[root@am-01:~#] redis-cli -c -h 172.17.1.240 -p 7000

#-c 的意思是使用集群的方式去登陆 redis,-h 指定 IP,-p 指定端口

172.17.1.240:7000> set key1 123

-> Redirected to slot [9189] located at 172.17.1.242:7001

OK

172.17.1.242:7001> set key2 abc

-> Redirected to slot [4998] located at 172.17.1.240:7000

OK

172.17.1.240:7000> set key3 xyz

OK

172.17.1.240:7000> set key4 qqq

-> Redirected to slot [13120] located at 172.17.1.240:7002

OK

#可以见到,创建的 key 会随机的分布在 3 台主服务器上

172.17.1.240:7002> get key1

-> Redirected to slot [9189] located at 172.17.1.242:7001

"123"

172.17.1.242:7001> get key2

-> Redirected to slot [4998] located at 172.17.1.240:7000

"abc"

#get 也是一样使用的,只不过会到存放对应 key 的主上读取数据

02

查看 redis 集群的状态

[root@am-01:~#] redis-trib.rb check  172.17.1.240:7000

>>> Performing Cluster Check (using node 172.17.1.240:7000)

M: 7f638f3646df80d6a1e503464fbf41c2ad032252 172.17.1.240:7000

   slots:0-5460 (5461 slots) master

   1 additional replica(s)

M: acfe177f0df7677163e91834dab498783b31b88a 172.17.1.242:7001

   slots:5461-10922 (5462 slots) master

   1 additional replica(s)

S: fdbfcbf5995684c1171558e900a8f71755b42339 172.17.1.242:7005

   slots: (0 slots) slave

   replicates 7f638f3646df80d6a1e503464fbf41c2ad032252

S: fc871f25690fa2bcea9a8bd849a8c03ed447e28c 172.17.1.242:7003

   slots: (0 slots) slave

   replicates 3c140721fba6a6e0a273bc5c8d42702206cf3e80

S: b7cb7728962bd27f2db3ed7a7d1adb75175d6731 172.17.1.240:7004

   slots: (0 slots) slave

   replicates acfe177f0df7677163e91834dab498783b31b88a

M: 3c140721fba6a6e0a273bc5c8d42702206cf3e80 172.17.1.240:7002

   slots:10923-16383 (5461 slots) master

   1 additional replica(s)

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

03

列出节点

[root@am-01:~#] redis-cli -c -h 172.17.1.240 -p 7000

172.17.1.240:7000> CLUSTER NODES

acfe177f0df7677163e91834dab498783b31b88a 172.17.1.242:7001@17001 master - 0 1525707555132 4 connected 5461-10922

fdbfcbf5995684c1171558e900a8f71755b42339 172.17.1.242:7005@17005 slave 7f638f3646df80d6a1e503464fbf41c2ad032252 0 1525707556140 6 connected

fc871f25690fa2bcea9a8bd849a8c03ed447e28c 172.17.1.242:7003@17003 slave 3c140721fba6a6e0a273bc5c8d42702206cf3e80 0 1525707554126 5 connected

b7cb7728962bd27f2db3ed7a7d1adb75175d6731 172.17.1.240:7004@17004 slave acfe177f0df7677163e91834dab498783b31b88a 0 1525707552111 4 connected

7f638f3646df80d6a1e503464fbf41c2ad032252 172.17.1.240:7000@17000 myself,master - 0 1525707555000 1 connected 0-5460

3c140721fba6a6e0a273bc5c8d42702206cf3e80 172.17.1.240:7002@17002 master - 0 1525707553117 2 connected 10923-16383

04

查看集群信息

172.17.1.240:7000> CLUSTER INFO

cluster_state:ok

cluster_slots_assigned:16384

cluster_slots_ok:16384

cluster_slots_pfail:0

cluster_slots_fail:0

cluster_known_nodes:6

cluster_size:3

cluster_current_epoch:6

cluster_my_epoch:1

cluster_stats_messages_ping_sent:766

cluster_stats_messages_pong_sent:769

cluster_stats_messages_sent:1535

cluster_stats_messages_ping_received:764

cluster_stats_messages_pong_received:766

cluster_stats_messages_meet_received:5

cluster_stats_messages_received:1535

05

添加新节点

[root@am-02:~#] cp /etc/redis_7001.conf /etc/redis_7006.conf

[root@am-02:~#] vim /etc/redis_7006.conf

port 7006

bind 172.17.1.242

daemonize yes

pidfile /var/run/redis_7006.pid

dir /data/redis_data/7006

cluster-enabled yes

cluster-config-file nodes_7006.conf

cluster-node-timeout 10100

appendonly yes

[root@am-02:~#] mkdir /data/redis_data/7006

[root@am-02:~#] redis-server /etc/redis_7006.conf

23504:C 07 May 23:42:28.960 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

23504:C 07 May 23:42:28.961 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=23504, just started

23504:C 07 May 23:42:28.961 # Configuration loaded

#在 B 机器上新建一个 7006 节点
[root@am-01:~#] redis-cli -c -h 172.17.1.240 -p 7000

172.17.1.240:7000> CLUSTER MEET 172.17.1.242 7006

OK

172.17.1.240:7000> CLUSTER NODES

4ca21410eda8e961b4a1eb0c87703ff87cecb731 172.17.1.242:7006@17006 master - 0 1525707807000 0 connected

#把 7006 节点添加到集群后发现是主
[root@am-01:~#] cp /etc/redis_7000.conf /etc/redis_7008.conf

[root@am-01:~#] vim /etc/redis_7008.conf

port 7008

bind 172.17.1.240

daemonize yes

pidfile /var/run/redis_7008.pid

dir /data/redis_data/7008

cluster-enabled yes

cluster-config-file nodes_7008.conf

cluster-node-timeout 10100

appendonly yes

[root@am-01:~#] mkdir /data/redis_data/7008

[root@am-01:~#] redis-server /etc/redis_7008.conf

61413:C 07 May 23:48:29.758 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

61413:C 07 May 23:48:29.759 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=61413, just started

61413:C 07 May 23:48:29.759 # Configuration loaded

[root@am-01:~#] redis-cli -c -h 172.17.1.240 -p 7000

172.17.1.240:7000> CLUSTER MEET 172.17.1.240 7008

OK

172.17.1.240:7000> CLUSTER NODES

b811ac53a57d593ed5b5388c1027d97766a56344 172.17.1.240:7008@17008 master - 0 1525708139625 0 connected

#在 A 机器上新建 7008 节点,添加到集群后发现也是主

06

把当前登陆节点设置为指定节点的从(cluster replicate node_id)

[root@am-01:~#] redis-cli -c -h 172.17.1.242 -p 7006

172.17.1.242:7006> CLUSTER REPLICATE b811ac53a57d593ed5b5388c1027d97766a56344

OK

172.17.1.242:7006> CLUSTER NODES

b811ac53a57d593ed5b5388c1027d97766a56344 172.17.1.240:7008@17008 master - 0 1525708308000 0 connected

4ca21410eda8e961b4a1eb0c87703ff87cecb731 172.17.1.242:7006@17006 myself,slave b811ac53a57d593ed5b5388c1027d97766a56344 0 1525708306000 7 connected

07

移除某个节点(cluster forget node_id),主的不能移除,需要先把其设置为一个从才可以移除

[root@am-01:~#] redis-cli -c -h 172.17.1.240 -p 7000

172.17.1.240:7000> CLUSTER FORGET 4ca21410eda8e961b4a1eb0c87703ff87cecb731

OK

172.17.1.240:7000> CLUSTER NODES

acfe177f0df7677163e91834dab498783b31b88a 172.17.1.242:7001@17001 master - 0 1525708564855 4 connected 5461-10922

fdbfcbf5995684c1171558e900a8f71755b42339 172.17.1.242:7005@17005 slave 7f638f3646df80d6a1e503464fbf41c2ad032252 0 1525708566000 6 connected

fc871f25690fa2bcea9a8bd849a8c03ed447e28c 172.17.1.242:7003@17003 slave 3c140721fba6a6e0a273bc5c8d42702206cf3e80 0 1525708564000 5 connected

b7cb7728962bd27f2db3ed7a7d1adb75175d6731 172.17.1.240:7004@17004 slave acfe177f0df7677163e91834dab498783b31b88a 0 1525708566872 4 connected

7f638f3646df80d6a1e503464fbf41c2ad032252 172.17.1.240:7000@17000 myself,master - 0 1525708563000 1 connected 0-5460

b811ac53a57d593ed5b5388c1027d97766a56344 172.17.1.240:7008@17008 master - 0 1525708565863 0 connected

3c140721fba6a6e0a273bc5c8d42702206cf3e80 172.17.1.240:7002@17002 master - 0 1525708564000 2 connected 10923-16383

08

保存配置文件,保存后可以见到,"nodes_7000.conf"的时间是可以见到有所变动的,证明对其做了一些操作

[root@am-01:~#] ls -lt /data/redis_data/7000/

总用量 12

-rw-r--r-- 1 root root 1041 5 月   7 23:57 nodes_7000.conf

-rw-r--r-- 1 root root   87 5 月   7 23:32 appendonly.aof

-rw-r--r-- 1 root root  175 5 月   7 23:28 dump.rdb

172.17.1.240:7000> CLUSTER SAVECONFIG

OK

[root@am-01:~#] ls -lt /data/redis_data/7000/

总用量 12

-rw-r--r-- 1 root root 1041 5 月   7 23:59 nodes_7000.conf

-rw-r--r-- 1 root root   87 5 月   7 23:32 appendonly.aof

-rw-r--r-- 1 root root  175 5 月   7 23:28 dump.rdb

扩展

redis 哨兵:

https://segmentfault.com/a/1190000002680804

http://www.cnblogs.com/jaycekon/p/6237562.html