[root@redis01 ~]# tail -f /var/log/redis/redis_6379.log 36561:signal-handler (1549951235) Received SIGTERM scheduling shutdown... 36561:M 12 Feb 2019 01:00:35.534 # User requested shutdown... 36561:M 12 Feb 2019 01:00:35.534 * Calling fsync() on the AOF file. 36561:M 12 Feb 2019 01:00:35.534 * Removing the pid file. 36561:M 12 Feb 2019 01:00:35.534 # Redis is now ready to exit, bye bye...
当其它节点均标识此master节点离线时,即表示此master为客观下线状态,如下日志
1 2
[root@redis02 ~]# tail -f /var/log/redis/redis_6380.log 36641:S 12 Feb 2019 01:00:54.274 * Marking node c876e5e99902ad9f11c714eca6770dac7626ffa0 as failing (quorum reached).
[root@redis03 ~]# tail -f /var/log/redis/redis_6380.log 36623:S 12 Feb 2019 01:00:35.550 # Connection with master lost. 36623:S 12 Feb 2019 01:00:35.550 * Caching the disconnected master state. 36623:S 12 Feb 2019 01:00:35.558 * Connecting to MASTER 192.168.100.1:6379 36623:S 12 Feb 2019 01:00:35.558 * MASTER <-> REPLICA sync started ... 36623:S 12 Feb 2019 01:00:51.481 * Marking node c876e5e99902ad9f11c714eca6770dac7626ffa0 as failing (quorum reached). 36623:S 12 Feb 2019 01:00:51.504 # Start of election delayed for 684 milliseconds (rank #0, offset 3488450). ... 36623:S 12 Feb 2019 01:01:11.556 # Currently unable to failover: Waiting for votes, but majority still not reached. ... 36623:S 12 Feb 2019 01:01:22.286 # Currently unable to failover: Failover attempt expired. ... 36623:S 12 Feb 2019 01:01:52.274 # Start of election delayed for 766 milliseconds (rank #0, offset 3488450). 36623:S 12 Feb 2019 01:01:52.374 # Currently unable to failover: Waiting the delay before I can start a new failover. 36623:S 12 Feb 2019 01:01:52.774 * Connecting to MASTER 192.168.100.1:6379 36623:S 12 Feb 2019 01:01:52.774 * MASTER <-> REPLICA sync started 36623:S 12 Feb 2019 01:01:52.775 # Error condition on socket for SYNC: Connection refused 36623:S 12 Feb 2019 01:01:53.074 # Starting a failover election for epoch 8. 36623:S 12 Feb 2019 01:01:53.077 # Currently unable to failover: Waiting for votes, but majority still not reached. 36623:S 12 Feb 2019 01:01:53.077 # Failover election won: I'm the new master. 36623:S 12 Feb 2019 01:01:53.077 # configEpoch set to 8 after successful failover 36623:M 12 Feb 2019 01:01:53.077 # Setting secondary replication ID to ec3a9236f6a0b677f35aacc271b12d034b06b3be, valid up to offset: 3488451. New replication ID is aa66a8d82de43ac2dd3da8126f9f54182d5ac77f 36623:M 12 Feb 2019 01:01:53.077 * Discarding previously cached master state.
[root@redis02 ~]# tail -f /var/log/redis/redis_6380.log 36641:S 12 Feb 2019 01:14:34.772 * Clear FAIL state for node c876e5e99902ad9f11c714eca6770dac7626ffa0: master without slots is reachable again.
此节点会自动识别并将自己变成新master节点的slave,并同步数据
1 2 3 4 5 6
[root@redis01 ~]# tail -f /var/log/redis/redis_6379.log 48088:M 12 Feb 2019 01:14:34.707 * Ready to accept connections 48088:M 12 Feb 2019 01:14:34.709 # Configuration change detected. Reconfiguring myself as a replica of fa1fa2a7c281115192f1caec30857921dadf0941 48088:S 12 Feb 2019 01:14:34.709 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 48088:S 12 Feb 2019 01:14:34.709 # Cluster state changed: ok 48088:S 12 Feb 2019 01:14:35.709 * Connecting to MASTER 192.168.100.3:6380
在新的master节点上,也能看到此节点的数据同步请求日志
1 2 3 4 5 6 7 8 9 10
[root@redis03 ~]# tail -f /var/log/redis/redis_6380.log 36623:M 12 Feb 2019 01:14:34.768 * Clear FAIL state for node c876e5e99902ad9f11c714eca6770dac7626ffa0: master without slots is reachable again. 36623:M 12 Feb 2019 01:14:35.724 * Replica 192.168.100.1:6379 asks for synchronization 36623:M 12 Feb 2019 01:14:35.724 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '0e2cbfecff38048e5d0fcb15fb7d5b16e44df9b4', my replication IDs are 'aa66a8d82de43ac2dd3da8126f9f54182d5ac77f' and 'ec3a9236f6a0b677f35aacc271b12d034b06b3be') 36623:M 12 Feb 2019 01:14:35.724 * Starting BGSAVE for SYNC with target: disk 36623:M 12 Feb 2019 01:14:35.728 * Background saving started by pid 48336 48336:C 12 Feb 2019 01:14:35.730 * DB saved on disk 48336:C 12 Feb 2019 01:14:35.731 * RDB: 8 MB of memory used by copy-on-write 36623:M 12 Feb 2019 01:14:35.769 * Background saving terminated with success 36623:M 12 Feb 2019 01:14:35.769 * Synchronization with replica 192.168.100.1:6379 succeeded
[root@redis01 ~]# redis-cli 127.0.0.1:6379> cluster failover OK
此节点由slave变更为master
1 2 3 4 5 6 7 8 9 10 11 12 13 14
[root@redis01 ~]# tail -f /var/log/redis/redis_6379.log 48088:S 12 Feb 2019 01:35:35.180 # Manual failover user request accepted. 48088:S 12 Feb 2019 01:35:35.208 # Received replication offset for paused master manual failover: 3490214 48088:S 12 Feb 2019 01:35:35.308 # All master replication stream processed, manual failover can start. 48088:S 12 Feb 2019 01:35:35.308 # Start of election delayed for 0 milliseconds (rank #0, offset 3490214). 48088:S 12 Feb 2019 01:35:35.408 # Starting a failover election for epoch 9. 48088:S 12 Feb 2019 01:35:35.412 # Failover election won: I'm the new master. 48088:S 12 Feb 2019 01:35:35.412 # configEpoch set to 9 after successful failover 48088:M 12 Feb 2019 01:35:35.412 # Setting secondary replication ID to aa66a8d82de43ac2dd3da8126f9f54182d5ac77f, valid up to offset: 3490215. New replication ID is 04057341f176ca0a1490fbe619d951bf28c8495e 48088:M 12 Feb 2019 01:35:35.412 # Connection with master lost. 48088:M 12 Feb 2019 01:35:35.412 * Caching the disconnected master state. 48088:M 12 Feb 2019 01:35:35.412 * Discarding previously cached master state. 48088:M 12 Feb 2019 01:35:36.412 * Replica 192.168.100.3:6380 asks for synchronization 48088:M 12 Feb 2019 01:35:36.412 * Partial resynchronization request from 192.168.100.3:6380 accepted. Sending 0 bytes of backlog starting from offset 3490215.
redis03:6380由master变回slave
1 2 3 4 5 6 7 8 9 10 11 12 13 14
[root@redis03 ~]# tail -f /var/log/redis/redis_6380.log 36623:M 12 Feb 2019 01:35:35.194 # Manual failover requested by replica c876e5e99902ad9f11c714eca6770dac7626ffa0. 36623:M 12 Feb 2019 01:35:35.424 # Failover auth granted to c876e5e99902ad9f11c714eca6770dac7626ffa0 for epoch 9 36623:M 12 Feb 2019 01:35:35.426 # Connection with replica 192.168.100.1:6379 lost. 36623:M 12 Feb 2019 01:35:35.428 # Configuration change detected. Reconfiguring myself as a replica of c876e5e99902ad9f11c714eca6770dac7626ffa0 36623:S 12 Feb 2019 01:35:35.428 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 36623:S 12 Feb 2019 01:35:36.425 * Connecting to MASTER 192.168.100.1:6379 36623:S 12 Feb 2019 01:35:36.425 * MASTER <-> REPLICA sync started 36623:S 12 Feb 2019 01:35:36.426 * Non blocking connect for SYNC fired the event. 36623:S 12 Feb 2019 01:35:36.426 * Master replied to PING, replication can continue... 36623:S 12 Feb 2019 01:35:36.426 * Trying a partial resynchronization (request aa66a8d82de43ac2dd3da8126f9f54182d5ac77f:3490215). 36623:S 12 Feb 2019 01:35:36.426 * Successful partial resynchronization with master. 36623:S 12 Feb 2019 01:35:36.426 # Master replication ID changed to 04057341f176ca0a1490fbe619d951bf28c8495e 36623:S 12 Feb 2019 01:35:36.427 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.