IBM HACMP - Status and information administration tasks
Determine the cluster name and networks
/usr/es/sbin/cluster/utilities/cltopinfo
Cluster Name: cluster1
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
There are 2 node(s) and 3 network(s) defined
NODE node1:
Network net_ether_01
cluster 192.168.53.2
node1 192.168.53.8
node1_s 192.168.49.2
Network net_tmscsi_0
tmscsi0_node1 /dev/tmscsi0
Network net_tmscsi_1
tmscsi1_node1 /dev/tmscsi1
NODE node2:
Network net_ether_01
cluster 192.168.53.2
node2 192.168.53.9
node2_s 192.168.59.3
Network net_tmscsi_0
tmscsi0_node2 /dev/tmscsi0
Network net_tmscsi_1
tmscsi1_node2 /dev/tmscsi1
Resource Group cache
Startup Policy Online Using Distribution Policy
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes node1 node2
Service IP Label cluster
Total Heartbeats Missed: 788
Cluster Topology Start Time: 05/25/2009 21:41:14
Determine the cluster ID
/usr/es/sbin/cluster/utilities/clrsctinfo -p cllsclstr
1472902783 cluster1 Standard
Verify the operational status of the topology services subsystem
lssrc -ls topsvcs
Verify the HACMP configuration:
/usr/es/sbin/cluster/diag/clconfig -v -O
Example output from clconfig -v -0
HACMPnode ODM on node tomato verified.
HACMPnetwork ODM on node tomato verified.
HACMPcluster ODM on node tomato verified.
HACMPnim ODM on node tomato verified.
HACMPadapter ODM on node tomato verified.
HACMPtopsvcs ODM on node tomato verified.
HACMPsite ODM on node tomato verified.
HACMPnode ODM on node tomato verified.
HACMPgroup ODM on node tomato verified.
HACMPresource ODM on node tomato verified.
HACMPserver ODM on node tomato verified.
HACMPcommadapter ODM on node tomato verified.
HACMPcommlink ODM on node tomato verified.
HACMPx25 ODM on node tomato verified.
HACMPsna ODM on node tomato verified.
HACMPevent ODM on node tomato verified.
HACMPcustom ODM on node tomato verified.
HACMPlogs ODM on node tomato verified.
HACMPtape ODM on node tomato verified.
HACMPmonitor ODM on node tomato verified.
HACMPpager ODM on node tomato verified.
HACMPport ODM on node tomato verified.
HACMPnpp ODM on node tomato verified.
HACMPude ODM on node tomato verified.
HACMPrresmethods ODM on node tomato verified.
HACMPdisksubsys ODM on node tomato verified.
HACMPpprc ODM on node tomato verified.
HACMPpairtasks ODM on node tomato verified.
HACMPpathtasks ODM on node tomato verified.
HACMPercmf ODM on node tomato verified.
HACMPercmfglobals ODM on node tomato verified.
HACMPtimer ODM on node tomato verified.
HACMPsiteinfo ODM on node tomato verified.
HACMPtimersvc ODM on node tomato verified.
HACMPfilecollection ODM on node tomato verified.
HACMPfcfile ODM on node tomato verified.
HACMPrgdependency ODM on node tomato verified.
HACMPrg_loc_dependency ODM on node tomato verified.
HACMPsvc ODM on node tomato verified.
HACMPsvcpprc ODM on node tomato verified.
HACMPsvcrelationship ODM on node tomato verified.
HACMPsa_metadata ODM on node tomato verified.
HACMPcsserver ODM on node tomato verified.
HACMPoemfsmethods ODM on node tomato verified.
HACMPoemvgmethods ODM on node tomato verified.
HACMPoemvolumegroup ODM on node tomato verified.
HACMPoemfilesystem ODM on node tomato verified.
HACMPdisktype ODM on node tomato verified.
Verification to be performed on the following:
Cluster Topology
Cluster Resources
Retrieving data from available cluster nodes. This could take a few minutes.........
Verifying Cluster Topology...
WARNING: Network option "nonlocsrcroute" is set to 0 on the following nodes:
cabbage
WARNING: Network option "ipsrcrouterecv" is set to 0 on the following nodes:
cabbage
Verifying Cluster Resources...
WARNING: Application monitors are required for detecting application failures
in order for HACMP to recover from them. Application monitors are started
by HACMP when the resource group in which they participate is activated.
The following application(s), shown with their associated resource group,
do not have an application monitor configured:
Application Server Resource Group
-------------------------------- ---------------------------------
appserv data
A corrective action is available for the condition reported below:
WARNING: The LVM time stamp for shared volume group: datavg is inconsistent
with the time stamp in the VGDA for the following nodes:
node1
To correct the above condition, run verification & synchronization with
"Automatically correct errors found during verification?" set to either 'Yes'
or 'Interactive'. The cluster must be down for the corrective action to run.
Corrective actions can be enabled for Verification and Synchronization in the
HACMP extended Verification and Synchronization SMIT fastpath "cl_sync".
Alternatively use the Initialization and Standard Configuration -> Verification
and Synchronization path where corrective actions are always executed in
interactive mode.
Remember to redo automatic error notification if configuration has changed.
Verification has completed normally.
Check the version of HACMP:
lslpp -L | grep cluster.es.server.rte
Example output from lslpp -L | grep cluster.es.server.rte
cluster.es.server.rte 5.4.0.1 C F ES Base Server Runtime
Show cluster services:
/usr/es/sbin/cluster/utilities/clshowsrv -v
Example output from clshowsrv -v
Status of the RSCT subsystems used by HACMP: Subsystem Group PID Status topsvcs topsvcs 278684 active grpsvcs grpsvcs 332026 active grpglsm grpsvcs inoperative emsvcs emsvcs 446712 active emaixos emsvcs 294942 active ctrmc rsct 131212 active Status of the HACMP subsystems: Subsystem Group PID Status clcomdES clcomdES 204984 active clstrmgrES cluster 86080 active Status of the optional HACMP subsystems: Subsystem Group PID Status clinfoES cluster 360702 active
Monitor the cluster status:
/usr/sbin/cluster/clstat
Example output from clstat
clstat - HACMP Cluster Status Monitor
-------------------------------------
Cluster: data_cluster (1274902884)
Wed 24 Sep 10:37:41 2008
State: UP Nodes: 2
SubState: STABLE
Node: tomato State: UP
Interface: tomato_s (0) Address: 192.168.10.2
State: UP
Interface: tomato (0) Address: 192.168.12.4
State: DOWN
Interface: data (0) Address: 192.168.12.5
State: UP
Resource Group: data State: On line
Node: cabbage State: DOWN
Interface: cabbage_s (0) Address: 192.168.10.3
State: DOWN
Interface: cabbage (0) Address: 192.168.12.9
State: DOWN
SNMP-based tool to show cluster state
/usr/es/sbin/cluster/utilities/cldump
Obtaining information via SNMP from Node: node1...
_____________________________________________________________________________
Cluster Name: CLSUTER1
Cluster State: UP
Cluster Substate: STABLE
_____________________________________________________________________________
Node Name: node1 State: UP
Network Name: network1 State: UP
Address: 10.11.190.124 Label: net1_bootB State: UP
Address: 10.11.190.60 Label: net1_bootA State: UP
Address: 10.11.190.8 Label: net1_srvc State: UP
Network Name: network2 State: UP
Address: 10.11.191.10 Label: net2_srvc State: UP
Address: 10.11.191.126 Label: net2_bootB State: UP
Address: 10.11.191.62 Label: net2_bootA State: UP
Network Name: ds4700a State: UP
Address: Label: node1_hdisk22 State: UP
Network Name: ds4700b State: UP
Address: Label: node1_hdisk34_01 State: UP
Node Name: node2 State: UP
Network Name: network1 State: UP
Address: 10.11.190.125 Label: node2_bootB State: UP
Address: 10.11.190.61 Label: node2_bootA State: UP
Address: 10.11.190.9 Label: node2_srvc State: UP
Network Name: network2 State: UP
Address: 10.11.191.11 Label: node2_srvc State: UP
Address: 10.11.191.127 Label: node2_bootB State: UP
Address: 10.11.191.63 Label: node2_bootA State: UP
Network Name: ds4700a State: UP
Address: Label: node2_hdisk14 State: UP
Network Name: ds4700b State: UP
Address: Label: node2_hdisk35_01 State: UP
Cluster Name: CLUSTER1
Resource Group Name: res_gp1
Startup Policy: Online On Home Node Only
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Never Fallback
Site Policy: ignore
Node Group State
---------------------------- ---------------
node1 ONLINE
node2 OFFLINE
Resource Group Name: res_gp2
Startup Policy: Online On Home Node Only
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Never Fallback
Site Policy: ignore
Node Group State
---------------------------- ---------------
node1 ONLINE
node2 OFFLINE
Resource Group Name: res_gp3
Startup Policy: Online On Home Node Only
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Never Fallback
Site Policy: ignore
Node Group State
---------------------------- ---------------
node1 ONLINE
node2 OFFLINE
Resource Group Name: res_gp4
Startup Policy: Online On Home Node Only
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Never Fallback
Site Policy: ignore
Node Group State
---------------------------- ---------------
node1 ONLINE
node2 OFFLINE