Jump to: navigation, search
MithiWiki Home > ConnectXf Home > ConnectXf Administration > Configuration > After reboot the DR server has become as primary and is live now


Troubleshooting Icon.png
Troubleshooting
Product ConnectXf
Version All
Applies to Administrators
Level Advanced




After reboot the DR server has become as primary and is live now.

Now I want to switch back to the primary server.

  • Check and note the following things:

On DR server which is live now

- Check if heartbeat is running on the server

/etc/init.d/heartbeat status

- DRBD status. This should show the server as Primary

cat /proc/drbd

- Check the partitions. mailstore and mcsdata should be mounted on drbd

df -h

- Check the virtual. IP should be assigned to eth0:1

ifconfig

On PR server which is currently not live check the following

- Check heartbeat service. It should be running on the server

/etc/init.d/heartbeat status

- DRBD status. This should show the server as Secondary

cat /proc/drbd

- Check the partitions. mailstore and mcsdata should not be mounted

df -h

- Check if the virtual. It should only show eth0 and eth1. No virtua ip asigned to eth0:1

ifconfig


  • Steps to switch back to the primary:

NOTE: The assumption here is before reboot both the servers were in sync (consistent) and the data on both the servers is identical.

- Stop heartbeat service on the DR server which is currently live

/etc/init.d/heartbeat stop &

- Tail the log to know the status

tail -f /var/log/ha-log

- On the primary server which is not live tail the log

tail -f /var/log/ha-log

- Please note that this process may take a very long time approx. 15 to 20 min.

- In between the access to the primary server may get lossed and will get the connection again.

- During this process please do not do any changes on any of the servers. Just keep monitoring both the servers:


  • After heartbeat process is completed following should be the status on both the servers:

On Primary

- This server should be live now

- Check the drbd status. The server should be Primary

cat /proc/drbd

- Check the heartbeat status

/etc/init.d/heartbeat status

- Check the partitions. mailstore and mcsdata should be mounted

df -h

- All required services should be running

/mithi/mcs/bin/checkservices.sh

- Check the entire mailflow and mail access

- Check if the data on the server is intact (No mail loss)


On DR server

- This server should not be live now

- Check the drbd status. The server should be Secondary

cat /proc/drbd

- Check the heartbeat status. The service should be stopped

/etc/init.d/heartbeat status

- Check the partitions. mailstore and mcsdata should not be mounted

df -h

- Check the services. No services should be running on the server

/mithi/mcs/bin/checkservices.sh


  • Final steps to connect both the servers:

- On primary server run the following command

drbdadm connect all

- On dr server run the following command

drbdadm connect all

- Check the drbd status on both the servers.

cat /proc/drbd

On Primary the output will be

 cat /proc/drbd
 version: 0.7.24 (api:79/proto:74)
SVN Revision: 2875 build by bachbuilder@, 2007-07-31 15:55:43
0: cs:SyncSource st:Primary/Secondary ld:Consistent
   ns:460428 nr:0 dw:527932 dr:3115633 al:2141 bm:2176 lo:0 pe:122 ua:202 ap:0
       [>...................] sync'ed:  3.7% (11943/12392)M
       finish: 0:21:37 speed: 9,392 (9,368) K/sec
1: cs:PausedSyncS st:Primary/Secondary ld:Consistent
   ns:15316 nr:0 dw:786176 dr:1127957 al:2779 bm:2622 lo:1 pe:0 ua:0 ap:1

On Secondary the output will be

cat /proc/drbd
version: 0.7.24 (api:79/proto:74)
SVN Revision: 2875 build by bachbuilder@, 2007-07-31 15:55:43
0: cs:SyncTarget st:Secondary/Primary ld:Inconsistent
   ns:0 nr:85220 dw:122212 dr:1326837 al:258 bm:520 lo:517 pe:117 ua:517 ap:0
       [>...................] sync'ed:  0.8% (12311/12392)M
       finish: 0:20:12 speed: 10,376 (10,376) K/sec
1: cs:PausedSyncT st:Secondary/Primary ld:Inconsistent
   ns:0 nr:2668 dw:381192 dr:914509 al:87 bm:214 lo:0 pe:0 ua:0 ap:0

Wait till it becomes consistent and finally the output on the dr should be

- On Primary server it should connect and start syncing to the dr server

- After the sync is completed both the servers should show consistent

- Start the heartbeat service on the secondary server:

/etc/init.d/heartbeat start