
============================================================
status case : network at 1 down and network 2 up then lb2 start to master instead!!  
network 1 down
network 2 up
network 3 up
network 4 up
service 1 on 
service 2 on
lb0 = 10.0.1.149 (Resource take to lb2)  old_status = master , new_status = slave 
lb2 = 10.0.1.146  old_status = slave , new_status = master
downtime take resource to lb2 = 33 sec
downtime to failback on lb0 = 5 sec

======================log at lb0 when network at 1 down =============================
heartbeat: 2007/12/13_15:27:50 WARN: node 10.0.0.254: is dead
heartbeat: 2007/12/13_15:27:50 info: Link 10.0.0.254:10.0.0.254 dead.
heartbeat: 2007/12/13_15:27:50 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2007/12/13_15:27:52 info: lb0 wants to go standby [all]
heartbeat: 2007/12/13_15:27:52 info: standby: lb2 can take our all resources
heartbeat: 2007/12/13_15:27:52 info: give up all HA resources (standby).
heartbeat: 2007/12/13_15:27:52 info: Releasing resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:27:52 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stop
heartbeat: 2007/12/13_15:27:53 info: Releasing resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:27:53 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master stop
heartbeat: 2007/12/13_15:27:53 info: ipvs_syncmaster down
heartbeat: 2007/12/13_15:27:53 info: ipvs_syncbackup up
heartbeat: 2007/12/13_15:27:53 info: ipvs_syncmaster released
heartbeat: 2007/12/13_15:27:53 info: Releasing resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:27:53 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 stop
heartbeat: 2007/12/13_15:27:53 info: /sbin/ip -f inet addr delete 10.0.1.152 dev eth0
heartbeat: 2007/12/13_15:27:53 info: /sbin/ip -o -f inet addr show eth0
heartbeat: 2007/12/13_15:27:53 info: IP Address 10.0.1.152 released
heartbeat: 2007/12/13_15:27:53 info: Releasing resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:27:53 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 stop
heartbeat: 2007/12/13_15:27:54 info: /sbin/ip -f inet addr delete 192.168.0.254 dev eth1
heartbeat: 2007/12/13_15:27:54 info: /sbin/ip -o -f inet addr show eth1
heartbeat: 2007/12/13_15:27:54 info: IP Address 192.168.0.254 released
heartbeat: 2007/12/13_15:27:54 info: all HA resource release completed (standby).
heartbeat: 2007/12/13_15:27:54 info: Local standby process completed [all].
heartbeat: 2007/12/13_15:27:55 WARN: 1 lost packet(s) for [lb2] [210:212]
heartbeat: 2007/12/13_15:27:55 info: remote resource transition completed.
heartbeat: 2007/12/13_15:27:55 info: No pkts missing from lb2!
heartbeat: 2007/12/13_15:27:55 info: Other node completed standby takeover of all resources.





===================== log at lb2 take resource to master node , when nw at 1 down===============
heartbeat: 2007/12/13_15:27:52 info: lb0 wants to go standby [all]
heartbeat: 2007/12/13_15:27:53 info: standby: acquire [all] resources from lb0
heartbeat: 2007/12/13_15:27:53 info: acquire all HA resources (standby).
heartbeat: 2007/12/13_15:27:53 info: Acquiring resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:27:54 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
heartbeat: 2007/12/13_15:27:54 info: Acquiring resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:27:54 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
heartbeat: 2007/12/13_15:27:54 info: ipvs_syncbackup down
heartbeat: 2007/12/13_15:27:54 info: ipvs_syncmaster up
heartbeat: 2007/12/13_15:27:54 info: ipvs_syncmaster obtained
heartbeat: 2007/12/13_15:27:55 info: Acquiring resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:27:55 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 start
heartbeat: 2007/12/13_15:27:55 info: /sbin/ip -f inet addr add 10.0.1.152/23 brd 10.0.1.255 dev eth0
heartbeat: 2007/12/13_15:27:55 info: /sbin/ip link set eth0 up
heartbeat: 2007/12/13_15:27:55 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-10.0.1.152 eth0 10.0.1.152 auto 10.0.1.152 ffffffffffff
heartbeat: 2007/12/13_15:27:55 info: Acquiring resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:27:55 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 start
heartbeat: 2007/12/13_15:27:55 info: /sbin/ip -f inet addr add 192.168.0.254/24 brd 192.168.0.255 dev eth1
heartbeat: 2007/12/13_15:27:55 info: /sbin/ip link set eth1 up
heartbeat: 2007/12/13_15:27:55 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.0.254 eth1 192.168.0.254 auto 192.168.0.254 ffffffffffff
heartbeat: 2007/12/13_15:27:55 info: all HA resource acquisition completed (standby).
heartbeat: 2007/12/13_15:27:55 info: Standby resource acquisition done [all].
heartbeat: 2007/12/13_15:27:55 info: remote resource transition completed.

=================== when network at 1 up  , and take resource back to master , log in lb0 =========
heartbeat: 2007/12/13_15:28:33 info: Link 10.0.0.254:10.0.0.254 up.
heartbeat: 2007/12/13_15:28:33 WARN: Late heartbeat: Node 10.0.0.254: interval 73030 ms
heartbeat: 2007/12/13_15:28:33 info: Status update for node 10.0.0.254: status ping
heartbeat: 2007/12/13_15:28:35 info: lb2 wants to go standby [foreign]
heartbeat: 2007/12/13_15:28:37 info: standby: acquire [foreign] resources from lb2
heartbeat: 2007/12/13_15:28:37 info: acquire local HA resources (standby).
heartbeat: 2007/12/13_15:28:37 info: Acquiring resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:28:38 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
heartbeat: 2007/12/13_15:28:38 info: Acquiring resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:28:38 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
heartbeat: 2007/12/13_15:28:39 info: ipvs_syncbackup down
heartbeat: 2007/12/13_15:28:39 info: ipvs_syncmaster up
heartbeat: 2007/12/13_15:28:39 info: ipvs_syncmaster obtained
heartbeat: 2007/12/13_15:28:39 info: Acquiring resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:28:39 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 start
heartbeat: 2007/12/13_15:28:39 info: /sbin/ip -f inet addr add 10.0.1.152/23 brd 10.0.1.255 dev eth0
heartbeat: 2007/12/13_15:28:39 info: /sbin/ip link set eth0 up
heartbeat: 2007/12/13_15:28:39 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-10.0.1.152 eth0 10.0.1.152 auto 10.0.1.152 ffffffffffff
heartbeat: 2007/12/13_15:28:39 info: Acquiring resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:28:39 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 start
heartbeat: 2007/12/13_15:28:39 info: /sbin/ip -f inet addr add 192.168.0.254/24 brd 192.168.0.255 dev eth1
heartbeat: 2007/12/13_15:28:39 info: /sbin/ip link set eth1 up
heartbeat: 2007/12/13_15:28:39 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.0.254 eth1 192.168.0.254 auto 192.168.0.254 ffffffffffff
heartbeat: 2007/12/13_15:28:39 info: local HA resource acquisition completed (standby).
heartbeat: 2007/12/13_15:28:39 info: Standby resource acquisition done [foreign].
heartbeat: 2007/12/13_15:28:39 info: remote resource transition completed.


=========================== lb2 back to slave again , when network at 1 up===============================
heartbeat: 2007/12/13_15:28:35 info: lb2 wants to go standby [foreign]
heartbeat: 2007/12/13_15:28:35 info: standby: lb0 can take our foreign resources
heartbeat: 2007/12/13_15:28:35 info: give up foreign HA resources (standby).
heartbeat: 2007/12/13_15:28:35 info: Releasing resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:28:35 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stop
heartbeat: 2007/12/13_15:28:36 info: Releasing resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:28:36 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master stop
heartbeat: 2007/12/13_15:28:37 info: ipvs_syncmaster down
heartbeat: 2007/12/13_15:28:37 info: ipvs_syncbackup up
heartbeat: 2007/12/13_15:28:37 info: ipvs_syncmaster released
heartbeat: 2007/12/13_15:28:37 info: Releasing resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:28:37 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 stop
heartbeat: 2007/12/13_15:28:37 info: /sbin/ip -f inet addr delete 10.0.1.152 dev eth0
heartbeat: 2007/12/13_15:28:37 info: /sbin/ip -o -f inet addr show eth0
heartbeat: 2007/12/13_15:28:37 info: IP Address 10.0.1.152 released
heartbeat: 2007/12/13_15:28:37 info: Releasing resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:28:37 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 stop
heartbeat: 2007/12/13_15:28:37 info: /sbin/ip -f inet addr delete 192.168.0.254 dev eth1
heartbeat: 2007/12/13_15:28:37 info: /sbin/ip -o -f inet addr show eth1
heartbeat: 2007/12/13_15:28:37 info: IP Address 192.168.0.254 released
heartbeat: 2007/12/13_15:28:37 info: foreign HA resource release completed (standby).
heartbeat: 2007/12/13_15:28:37 info: Local standby process completed [foreign].
heartbeat: 2007/12/13_15:28:38 WARN: 1 lost packet(s) for [lb0] [163:165]
heartbeat: 2007/12/13_15:28:38 info: remote resource transition completed.
heartbeat: 2007/12/13_15:28:38 info: No pkts missing from lb0!
heartbeat: 2007/12/13_15:28:38 info: Other node completed standby takeover of foreign resources.
===========================================================================