============================================================
status case : network at 3 down and network 4 up then lb2 start to master instead!!  
network 1 up
network 2 up
network 3 down
network 4 up
service 1 on 
service 2 on
lb0 = 10.0.1.149 (Resource take to lb2)  old_status = master , net_status = slave 
lb2 = 10.0.1.146  old_status = slave , new_status = master
downtime take to lb2 = 32 sec
downtime to failback = 5 sec
======================log at lb0 when network at 3 down===========================
heartbeat: 2007/12/13_15:45:52 WARN: node 192.168.0.6: is dead
heartbeat: 2007/12/13_15:45:52 info: Link 192.168.0.6:192.168.0.6 dead.
heartbeat: 2007/12/13_15:45:52 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2007/12/13_15:45:54 info: lb0 wants to go standby [all]
heartbeat: 2007/12/13_15:45:54 info: standby: lb2 can take our all resources
heartbeat: 2007/12/13_15:45:54 info: give up all HA resources (standby).
heartbeat: 2007/12/13_15:45:54 info: Releasing resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:45:54 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stop
heartbeat: 2007/12/13_15:45:54 info: Releasing resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:45:54 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master stop
heartbeat: 2007/12/13_15:45:55 info: ipvs_syncmaster down
heartbeat: 2007/12/13_15:45:55 info: ipvs_syncbackup up
heartbeat: 2007/12/13_15:45:55 info: ipvs_syncmaster released
heartbeat: 2007/12/13_15:45:55 info: Releasing resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:45:55 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 stop
heartbeat: 2007/12/13_15:45:55 info: /sbin/ip -f inet addr delete 10.0.1.152 dev eth0
heartbeat: 2007/12/13_15:45:55 info: /sbin/ip -o -f inet addr show eth0
heartbeat: 2007/12/13_15:45:55 info: IP Address 10.0.1.152 released
heartbeat: 2007/12/13_15:45:55 info: Releasing resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:45:55 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 stop
heartbeat: 2007/12/13_15:45:55 info: /sbin/ip -f inet addr delete 192.168.0.254 dev eth1
heartbeat: 2007/12/13_15:45:55 info: /sbin/ip -o -f inet addr show eth1
heartbeat: 2007/12/13_15:45:55 info: IP Address 192.168.0.254 released
heartbeat: 2007/12/13_15:45:55 info: all HA resource release completed (standby).
heartbeat: 2007/12/13_15:45:55 info: Local standby process completed [all].
heartbeat: 2007/12/13_15:45:57 WARN: 1 lost packet(s) for [lb2] [1314:1316]
heartbeat: 2007/12/13_15:45:57 info: remote resource transition completed.
heartbeat: 2007/12/13_15:45:57 info: No pkts missing from lb2!
heartbeat: 2007/12/13_15:45:57 info: Other node completed standby takeover of all resources.

===========================log in lb2 when take resource to be master =======================
heartbeat: 2007/12/13_15:45:53 info: lb0 wants to go standby [all]
heartbeat: 2007/12/13_15:45:55 info: standby: acquire [all] resources from lb0
heartbeat: 2007/12/13_15:45:55 info: acquire all HA resources (standby).
heartbeat: 2007/12/13_15:45:55 info: Acquiring resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:45:55 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
heartbeat: 2007/12/13_15:45:55 info: Acquiring resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:45:56 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
heartbeat: 2007/12/13_15:45:56 info: ipvs_syncbackup down
heartbeat: 2007/12/13_15:45:56 info: ipvs_syncmaster up
heartbeat: 2007/12/13_15:45:56 info: ipvs_syncmaster obtained
heartbeat: 2007/12/13_15:45:56 info: Acquiring resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:45:56 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 start
heartbeat: 2007/12/13_15:45:56 info: /sbin/ip -f inet addr add 10.0.1.152/23 brd 10.0.1.255 dev eth0
heartbeat: 2007/12/13_15:45:56 info: /sbin/ip link set eth0 up
heartbeat: 2007/12/13_15:45:56 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-10.0.1.152 eth0 10.0.1.152 auto 10.0.1.152 ffffffffffff
heartbeat: 2007/12/13_15:45:56 info: Acquiring resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:45:56 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 start
heartbeat: 2007/12/13_15:45:56 info: /sbin/ip -f inet addr add 192.168.0.254/24 brd 192.168.0.255 dev eth1
heartbeat: 2007/12/13_15:45:56 info: /sbin/ip link set eth1 up
heartbeat: 2007/12/13_15:45:56 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.0.254 eth1 192.168.0.254 auto 192.168.0.254 ffffffffffff
heartbeat: 2007/12/13_15:45:56 info: all HA resource acquisition completed (standby).
heartbeat: 2007/12/13_15:45:56 info: Standby resource acquisition done [all].
heartbeat: 2007/12/13_15:45:56 info: remote resource transition completed.

====================== when network 3 is back to up , log in lb0 ===========================
heartbeat: 2007/12/13_15:48:36 WARN: Late heartbeat: Node 192.168.0.6: interval 195070 ms
heartbeat: 2007/12/13_15:48:36 info: Status update for node 192.168.0.6: status ping
heartbeat: 2007/12/13_15:48:36 info: Link 192.168.0.6:192.168.0.6 up.
heartbeat: 2007/12/13_15:48:38 info: lb2 wants to go standby [foreign]
heartbeat: 2007/12/13_15:48:39 info: standby: acquire [foreign] resources from lb2
heartbeat: 2007/12/13_15:48:39 info: acquire local HA resources (standby).
heartbeat: 2007/12/13_15:48:39 info: Acquiring resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:48:40 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
heartbeat: 2007/12/13_15:48:40 info: Acquiring resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:48:40 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
heartbeat: 2007/12/13_15:48:41 info: ipvs_syncbackup down
heartbeat: 2007/12/13_15:48:41 info: ipvs_syncmaster up
heartbeat: 2007/12/13_15:48:41 info: ipvs_syncmaster obtained
heartbeat: 2007/12/13_15:48:41 info: Acquiring resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:48:41 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 start
heartbeat: 2007/12/13_15:48:41 info: /sbin/ip -f inet addr add 10.0.1.152/23 brd 10.0.1.255 dev eth0
heartbeat: 2007/12/13_15:48:41 info: /sbin/ip link set eth0 up
heartbeat: 2007/12/13_15:48:41 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-10.0.1.152 eth0 10.0.1.152 auto 10.0.1.152 ffffffffffff
heartbeat: 2007/12/13_15:48:41 info: Acquiring resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:48:41 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 start
heartbeat: 2007/12/13_15:48:41 info: /sbin/ip -f inet addr add 192.168.0.254/24 brd 192.168.0.255 dev eth1
heartbeat: 2007/12/13_15:48:41 info: /sbin/ip link set eth1 up
heartbeat: 2007/12/13_15:48:41 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.0.254 eth1 192.168.0.254 auto 192.168.0.254 ffffffffffff
heartbeat: 2007/12/13_15:48:41 info: local HA resource acquisition completed (standby).
heartbeat: 2007/12/13_15:48:41 info: Standby resource acquisition done [foreign].
heartbeat: 2007/12/13_15:48:41 info: remote resource transition completed.


====================== log in lb2 , when back to slave when network 3 up again =============

heartbeat: 2007/12/13_15:48:37 info: lb2 wants to go standby [foreign]
heartbeat: 2007/12/13_15:48:37 info: standby: lb0 can take our foreign resources
heartbeat: 2007/12/13_15:48:37 info: give up foreign HA resources (standby).
heartbeat: 2007/12/13_15:48:37 info: Releasing resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:48:37 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stop
heartbeat: 2007/12/13_15:48:38 info: Releasing resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:48:38 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master stop
heartbeat: 2007/12/13_15:48:39 info: ipvs_syncmaster down
heartbeat: 2007/12/13_15:48:39 info: ipvs_syncbackup up
heartbeat: 2007/12/13_15:48:39 info: ipvs_syncmaster released
heartbeat: 2007/12/13_15:48:39 info: Releasing resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:48:39 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 stop
heartbeat: 2007/12/13_15:48:39 info: /sbin/ip -f inet addr delete 10.0.1.152 dev eth0
heartbeat: 2007/12/13_15:48:39 info: /sbin/ip -o -f inet addr show eth0
heartbeat: 2007/12/13_15:48:39 info: IP Address 10.0.1.152 released
heartbeat: 2007/12/13_15:48:39 info: Releasing resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:48:39 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 stop
heartbeat: 2007/12/13_15:48:39 info: /sbin/ip -f inet addr delete 192.168.0.254 dev eth1
heartbeat: 2007/12/13_15:48:39 info: /sbin/ip -o -f inet addr show eth1
heartbeat: 2007/12/13_15:48:39 info: IP Address 192.168.0.254 released
heartbeat: 2007/12/13_15:48:39 info: foreign HA resource release completed (standby).
heartbeat: 2007/12/13_15:48:39 info: Local standby process completed [foreign].
heartbeat: 2007/12/13_15:48:40 WARN: 1 lost packet(s) for [lb0] [1387:1389]
heartbeat: 2007/12/13_15:48:40 info: remote resource transition completed.
heartbeat: 2007/12/13_15:48:40 info: No pkts missing from lb0!
heartbeat: 2007/12/13_15:48:40 info: Other node completed standby takeover of foreign resources.
