============================================================
status case : service at lb0 down and service lb2 up then lb2 start to master instead!!  
network 1 up
network 2 up
service 1 down 
service 2 on
lb0 = 10.0.1.149 (Resource take to lb2) old_status = master , current_status = slave
lb2 = 10.0.1.146  old_status = slave , current_status = master
downtime take to lb2 = 5 sec
downtime failback to lb0 when service lb0 on again = 5 sec
======================log at lb0===========================
heartbeat: 2007/12/13_15:02:04 info: Heartbeat shutdown in progress. (20363)
heartbeat: 2007/12/13_15:02:04 info: Giving up all HA resources.
heartbeat: 2007/12/13_15:02:04 info: Releasing resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:02:04 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stop
heartbeat: 2007/12/13_15:02:04 info: Releasing resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:02:04 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master stop
heartbeat: 2007/12/13_15:02:05 info: ipvs_syncmaster down
heartbeat: 2007/12/13_15:02:05 info: ipvs_syncbackup up
heartbeat: 2007/12/13_15:02:05 info: ipvs_syncmaster released
heartbeat: 2007/12/13_15:02:05 info: Releasing resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:02:05 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 stop
heartbeat: 2007/12/13_15:02:05 info: /sbin/ip -f inet addr delete 10.0.1.152 dev eth0
heartbeat: 2007/12/13_15:02:05 info: /sbin/ip -o -f inet addr show eth0
heartbeat: 2007/12/13_15:02:05 info: IP Address 10.0.1.152 released
heartbeat: 2007/12/13_15:02:05 info: Releasing resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:02:05 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 stop
heartbeat: 2007/12/13_15:02:05 info: /sbin/ip -f inet addr delete 192.168.0.254 dev eth1
heartbeat: 2007/12/13_15:02:05 info: /sbin/ip -o -f inet addr show eth1
heartbeat: 2007/12/13_15:02:05 info: IP Address 192.168.0.254 released
heartbeat: 2007/12/13_15:02:05 info: All HA resources relinquished.
heartbeat: 2007/12/13_15:02:05 WARN: 1 lost packet(s) for [lb2] [2610:2612]
heartbeat: 2007/12/13_15:02:05 info: No pkts missing from lb2!
heartbeat: 2007/12/13_15:02:06 info: killing HBFIFO process 20372 with signal 15
heartbeat: 2007/12/13_15:02:06 info: killing HBWRITE process 20373 with signal 15
heartbeat: 2007/12/13_15:02:06 info: killing HBREAD process 20374 with signal 15
heartbeat: 2007/12/13_15:02:06 info: killing HBWRITE process 20375 with signal 15
heartbeat: 2007/12/13_15:02:06 info: killing HBREAD process 20376 with signal 15
heartbeat: 2007/12/13_15:02:06 info: killing HBWRITE process 20377 with signal 15
heartbeat: 2007/12/13_15:02:06 info: killing HBREAD process 20378 with signal 15
heartbeat: 2007/12/13_15:02:06 info: Core process 20378 exited. 7 remaining
heartbeat: 2007/12/13_15:02:06 info: Core process 20377 exited. 6 remaining
heartbeat: 2007/12/13_15:02:06 info: Core process 20376 exited. 5 remaining
heartbeat: 2007/12/13_15:02:06 info: Core process 20375 exited. 4 remaining
heartbeat: 2007/12/13_15:02:06 info: Core process 20374 exited. 3 remaining
heartbeat: 2007/12/13_15:02:06 info: Core process 20373 exited. 2 remaining
heartbeat: 2007/12/13_15:02:06 info: Core process 20372 exited. 1 remaining
heartbeat: 2007/12/13_15:02:06 info: Heartbeat shutdown complete.
=========================== log at lb2 when take resource ==========================
heartbeat: 2007/12/13_15:02:05 info: Received shutdown notice from 'lb0'.
heartbeat: 2007/12/13_15:02:05 info: Resources being acquired from lb0.
heartbeat: 2007/12/13_15:02:05 info: acquire local HA resources (standby).
heartbeat: 2007/12/13_15:02:05 info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys lb2] to acquire.
heartbeat: 2007/12/13_15:02:05 info: local HA resource acquisition completed (standby).
heartbeat: 2007/12/13_15:02:05 info: Standby resource acquisition done [foreign].
heartbeat: 2007/12/13_15:02:05 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2007/12/13_15:02:05 info: Taking over resource group ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:02:05 info: Acquiring resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:02:05 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
heartbeat: 2007/12/13_15:02:06 info: Taking over resource group LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:02:06 info: Acquiring resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:02:06 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
heartbeat: 2007/12/13_15:02:06 info: ipvs_syncbackup down
heartbeat: 2007/12/13_15:02:06 info: ipvs_syncmaster up
heartbeat: 2007/12/13_15:02:06 info: ipvs_syncmaster obtained
heartbeat: 2007/12/13_15:02:06 info: Taking over resource group IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:02:06 info: Acquiring resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:02:06 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 start
heartbeat: 2007/12/13_15:02:06 info: /sbin/ip -f inet addr add 10.0.1.152/23 brd 10.0.1.255 dev eth0
heartbeat: 2007/12/13_15:02:06 info: /sbin/ip link set eth0 up
heartbeat: 2007/12/13_15:02:06 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-10.0.1.152 eth0 10.0.1.152 auto 10.0.1.152 ffffffffffff
heartbeat: 2007/12/13_15:02:06 info: Taking over resource group IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:02:06 info: Acquiring resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:02:06 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 start
heartbeat: 2007/12/13_15:02:06 info: /sbin/ip -f inet addr add 192.168.0.254/24 brd 192.168.0.255 dev eth1
heartbeat: 2007/12/13_15:02:06 info: /sbin/ip link set eth1 up
heartbeat: 2007/12/13_15:02:06 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.0.254 eth1 192.168.0.254 auto 192.168.0.254 ffffffffffff
heartbeat: 2007/12/13_15:02:06 info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
heartbeat: 2007/12/13_15:02:06 info: mach_down takeover complete.
heartbeat: 2007/12/13_15:02:06 info: mach_down takeover complete for node lb0.
heartbeat: 2007/12/13_15:02:36 WARN: node lb0: is dead
heartbeat: 2007/12/13_15:02:36 info: Dead node lb0 gave up resources.
heartbeat: 2007/12/13_15:02:36 info: Link lb0:eth2 dead.

======================= when service at lb0 up ,it will take resource from lb2======log at lb0 ======

heartbeat: 2007/12/13_15:06:58 info: **************************
heartbeat: 2007/12/13_15:06:58 info: Configuration validated. Starting heartbeat 1.2.5
heartbeat: 2007/12/13_15:06:58 info: heartbeat: version 1.2.5
heartbeat: 2007/12/13_15:06:58 info: Heartbeat generation: 127
heartbeat: 2007/12/13_15:06:58 info: UDP Broadcast heartbeat started on port 694 (694) interface eth2
heartbeat: 2007/12/13_15:06:58 info: ping heartbeat started.
heartbeat: 2007/12/13_15:06:58 info: ping heartbeat started.
heartbeat: 2007/12/13_15:06:58 info: pid 24799 locked in memory.
heartbeat: 2007/12/13_15:06:58 info: Local status now set to: 'up'
heartbeat: 2007/12/13_15:06:59 info: pid 24802 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: pid 24803 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: pid 24804 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: Link lb2:eth2 up.
heartbeat: 2007/12/13_15:06:59 info: Status update for node lb2: status active
heartbeat: 2007/12/13_15:06:59 info: Link lb0:eth2 up.
heartbeat: 2007/12/13_15:06:59 info: pid 24805 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: pid 24806 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: Link 10.0.0.254:10.0.0.254 up.
heartbeat: 2007/12/13_15:06:59 info: Status update for node 10.0.0.254: status ping
heartbeat: 2007/12/13_15:06:59 info: pid 24807 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: pid 24808 locked in memory.
heartbeat: 2007/12/13_15:06:59 info: Status update for node 192.168.0.6: status ping
heartbeat: 2007/12/13_15:06:59 info: Local status now set to: 'active'
heartbeat: 2007/12/13_15:06:59 info: Link 192.168.0.6:192.168.0.6 up.
heartbeat: 2007/12/13_15:06:59 info: remote resource transition completed.
heartbeat: 2007/12/13_15:06:59 info: remote resource transition completed.
heartbeat: 2007/12/13_15:06:59 info: Local Resource acquisition completed. (none)
heartbeat: 2007/12/13_15:06:59 info: lb2 wants to go standby [foreign]
heartbeat: 2007/12/13_15:06:59 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2007/12/13_15:07:01 info: standby: acquire [foreign] resources from lb2
heartbeat: 2007/12/13_15:07:01 info: acquire local HA resources (standby).
heartbeat: 2007/12/13_15:07:01 info: Acquiring resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:07:01 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
heartbeat: 2007/12/13_15:07:02 info: Acquiring resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:07:02 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
heartbeat: 2007/12/13_15:07:02 info: ipvs_syncbackup down
heartbeat: 2007/12/13_15:07:02 info: ipvs_syncmaster up
heartbeat: 2007/12/13_15:07:02 info: ipvs_syncmaster obtained
heartbeat: 2007/12/13_15:07:02 info: Acquiring resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:07:02 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 start
heartbeat: 2007/12/13_15:07:02 info: /sbin/ip -f inet addr add 10.0.1.152/23 brd 10.0.1.255 dev eth0
heartbeat: 2007/12/13_15:07:02 info: /sbin/ip link set eth0 up
heartbeat: 2007/12/13_15:07:02 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-10.0.1.152 eth0 10.0.1.152 auto 10.0.1.152 ffffffffffff
heartbeat: 2007/12/13_15:07:02 info: Acquiring resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:07:02 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 start
heartbeat: 2007/12/13_15:07:02 info: /sbin/ip -f inet addr add 192.168.0.254/24 brd 192.168.0.255 dev eth1
heartbeat: 2007/12/13_15:07:02 info: /sbin/ip link set eth1 up
heartbeat: 2007/12/13_15:07:02 /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.0.254 eth1 192.168.0.254 auto 192.168.0.254 ffffffffffff
heartbeat: 2007/12/13_15:07:02 info: local HA resource acquisition completed (standby).
heartbeat: 2007/12/13_15:07:02 info: Standby resource acquisition done [foreign].
heartbeat: 2007/12/13_15:07:02 info: Initial resource acquisition complete (auto_failback)
heartbeat: 2007/12/13_15:07:02 info: remote resource transition completed.

===================================lb2 set is slave again log =======================================
heartbeat: 2007/12/13_15:06:59 info: Heartbeat restart on node lb0
heartbeat: 2007/12/13_15:06:59 info: Link lb0:eth2 up.
heartbeat: 2007/12/13_15:06:59 info: Status update for node lb0: status up
heartbeat: 2007/12/13_15:06:59 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2007/12/13_15:06:59 info: Status update for node lb0: status active
heartbeat: 2007/12/13_15:06:59 info: remote resource transition completed.
heartbeat: 2007/12/13_15:06:59 info: lb2 wants to go standby [foreign]
heartbeat: 2007/12/13_15:06:59 info: standby: lb0 can take our foreign resources
heartbeat: 2007/12/13_15:06:59 info: give up foreign HA resources (standby).
heartbeat: 2007/12/13_15:06:59 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2007/12/13_15:06:59 info: Releasing resource group: lb0 ldirectord::ldirectord.cf
heartbeat: 2007/12/13_15:06:59 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stop
heartbeat: 2007/12/13_15:06:59 info: Releasing resource group: lb0 LVSSyncDaemonSwap::master
heartbeat: 2007/12/13_15:06:59 info: Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master stop
heartbeat: 2007/12/13_15:07:00 info: ipvs_syncmaster down
heartbeat: 2007/12/13_15:07:00 info: ipvs_syncbackup up
heartbeat: 2007/12/13_15:07:00 info: ipvs_syncmaster released
heartbeat: 2007/12/13_15:07:00 info: Releasing resource group: lb0 IPaddr2::10.0.1.152/23/eth0
heartbeat: 2007/12/13_15:07:00 info: Running /etc/ha.d/resource.d/IPaddr2 10.0.1.152/23/eth0 stop
heartbeat: 2007/12/13_15:07:00 info: /sbin/ip -f inet addr delete 10.0.1.152 dev eth0
heartbeat: 2007/12/13_15:07:00 info: /sbin/ip -o -f inet addr show eth0
heartbeat: 2007/12/13_15:07:00 info: IP Address 10.0.1.152 released
heartbeat: 2007/12/13_15:07:00 info: Releasing resource group: lb0 IPaddr2::192.168.0.254/24/eth1
heartbeat: 2007/12/13_15:07:01 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.0.254/24/eth1 stop
heartbeat: 2007/12/13_15:07:01 info: /sbin/ip -f inet addr delete 192.168.0.254 dev eth1
heartbeat: 2007/12/13_15:07:01 info: /sbin/ip -o -f inet addr show eth1
heartbeat: 2007/12/13_15:07:01 info: IP Address 192.168.0.254 released
heartbeat: 2007/12/13_15:07:01 info: foreign HA resource release completed (standby).
heartbeat: 2007/12/13_15:07:01 info: Local standby process completed [foreign].
heartbeat: 2007/12/13_15:07:02 WARN: 1 lost packet(s) for [lb0] [11:13]
heartbeat: 2007/12/13_15:07:02 info: remote resource transition completed.
heartbeat: 2007/12/13_15:07:02 info: No pkts missing from lb0!
heartbeat: 2007/12/13_15:07:02 info: Other node completed standby takeover of foreign resources.
