Moving Ceph Monitor Interface

Problem

A deployed ceph monitor is bound to the wrong interface and needs to be reconfigured. In this case, the ceph monitor on ceph2 is bound to localhost when it needs to be bound to the publically accesible interface.

The ceph status shows the IP address and socket the monitor is bound to

[root@ceph2 ~]# ceph -s
    cluster 7200aea0-2ddd-4a32-aa2a-d49f66ab554c
     health HEALTH_WARN
            too many PGs per OSD (320 > max 300)
     monmap e1: 1 mons at {ceph2=127.0.0.1:6789/0}
            election epoch 4, quorum 0 ceph2
     osdmap e9: 1 osds: 1 up, 1 in
            flags sortbitwise
      pgmap v29432: 320 pgs, 5 pools, 0 bytes data, 0 objects
            2577 MB used, 10692 MB / 13270 MB avail
                 320 active+clean

Initial Conditions

• Ceph monitor is bound to localhost
• External IP address is 192.168.122.7
• CentOS 7 is the base OS
• Ceph Jewel is installed

Reconfiguration Process

• Obtain and print current monitor map

[root@ceph2 ~]# ceph mon getmap -o monmap
got monmap epoch 1
[root@ceph2 ~]# monmaptool --print monmap
monmaptool: monmap file monmap
epoch 1
fsid 7200aea0-2ddd-4a32-aa2a-d49f66ab554c
last_changed 2016-07-22 00:14:22.139218
created 2016-07-22 00:14:22.139218
0: 127.0.0.1:6789/0 mon.ceph2

• Shut down all ceph services. Shutting down the monitor is show below.

[root@ceph2 ~]# systemctl stop ceph-mon@ceph2
[root@ceph2 ~]# systemctl status ceph-mon@ceph2
● ceph-mon@ceph2.service - Ceph cluster monitor daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Sun 2016-09-11 10:01:15 EDT; 5s ago
  Process: 8535 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
 Main PID: 8535 (code=exited, status=0/SUCCESS)

Sep 11 10:00:45 ceph2.example.com systemd[1]: Started Ceph cluster monitor daemon.
Sep 11 10:00:45 ceph2.example.com systemd[1]: Starting Ceph cluster monitor daemon...
Sep 11 10:00:45 ceph2.example.com ceph-mon[8535]: starting mon.ceph2 rank 0 at 127.0.0.1:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph2 fsid 7200aea0-2ddd-4a32-aa2a-d49f66ab554c
Sep 11 10:01:15 ceph2.example.com systemd[1]: Stopping Ceph cluster monitor daemon...
Sep 11 10:01:15 ceph2.example.com ceph-mon[8535]: 2016-09-11 10:01:15.784773 7fbd7acc2700 -1 mon.ceph2@0(leader) e1 *** Got Signal Terminated ***
Sep 11 10:01:15 ceph2.example.com systemd[1]: Stopped Ceph cluster monitor daemon.

• Update the ceph.conf file to list the new IP address

[root@ceph2 ~]# diff -u /etc/ceph/ceph.conf.ORIG /etc/ceph/ceph.conf
--- /etc/ceph/ceph.conf.ORIG    2016-09-11 10:03:42.612569161 -0400
+++ /etc/ceph/ceph.conf    2016-09-11 10:04:27.634243638 -0400
@@ -9,7 +9,7 @@
 auth_supported = cephx
 auth_cluster_required = cephx
 osd_journal_size = 100
-mon_host = 127.0.0.1
+mon_host = 192.168.122.7
 auth_client_required = cephx
 osd_pool_default_size = 1

• Remove old entry from the monitor map

[root@ceph2 ~]# monmaptool --rm ceph2 monmap
monmaptool: monmap file monmap
monmaptool: removing ceph2
monmaptool: writing epoch 1 to monmap (0 monitors)
[root@ceph2 ~]# monmaptool --print monmap
monmaptool: monmap file monmap
epoch 1
fsid 7200aea0-2ddd-4a32-aa2a-d49f66ab554c
last_changed 2016-07-22 00:14:22.139218
created 2016-07-22 00:14:22.139218

• Add new entry to the monitor map

[root@ceph2 ~]# monmaptool --add ceph2 192.168.122.7:6789 monmap
monmaptool: monmap file monmap
monmaptool: writing epoch 1 to monmap (1 monitors)
[root@ceph2 ~]# monmaptool --print monmap
monmaptool: monmap file monmap
epoch 1
fsid 7200aea0-2ddd-4a32-aa2a-d49f66ab554c
last_changed 2016-07-22 00:14:22.139218
created 2016-07-22 00:14:22.139218
0: 192.168.122.7:6789/0 mon.ceph2

• Add the new monitor map to the cluster

[root@ceph2 ~]# ceph-mon -i ceph2 --inject-monmap monmap

• Copy the new monmap to any other ceph monitors and perform the same remove and add process. 
• Copy ceph.conf file to all ceph members and clients
• Restart and verify the ceph services. The ceph monitor is show below. 

[root@ceph2 ~]# systemctl start ceph-mon@ceph2
[root@ceph2 ~]# ceph -s
    cluster 7200aea0-2ddd-4a32-aa2a-d49f66ab554c
     health HEALTH_WARN
            too many PGs per OSD (320 > max 300)
     monmap e3: 1 mons at {ceph2=192.168.122.7:6789/0}
            election epoch 7, quorum 0 ceph2
     osdmap e9: 1 osds: 1 up, 1 in
            flags sortbitwise
      pgmap v29432: 320 pgs, 5 pools, 0 bytes data, 0 objects
            2577 MB used, 10692 MB / 13270 MB avail
                 320 active+clean
[root@ceph2 ~]# netstat -tulpn | grep 6789
tcp        0      0 192.168.122.7:6789      0.0.0.0:*               LISTEN      9081/ceph-mon