If a companion server issues error message 18750, check the @@cmpstate of your servers. If your primary companion is in normal companion mode, but the secondary companion is in secondary failover mode, your cluster is in an inconsistent state, and you need to manually recover from this. This inconsistent state may be caused by an sp_companion 'prepare_failback' command failing on the secondary companion. You can determine whether this happened by examining the HACMP log on the secondary node, (located in /tmp/hacmp.out). To recover from this, perform the following steps manually:
Shut down both the primary and the secondary companions.
Reboot the secondary companion.
Repair all databases marked “suspect.” To determine which databases are suspect, issue:
select name, status from sysdatabases
Databases marked suspect have a status value of 320.
Allow updates to system tables:
sp_configure “allow updates”, 1
For each suspect failed over database, perform the following:
1> update sysdatabase set status=status-256 where name='database_name' 2> go 1> dbcc traceon(3604) 2> go 1> dbcc dbrecover(database_name) 2> go
From the secondary companion, issue:
sp_companion primary_companion_name, prepare_failback
For example, from primary companion MONEY1:
sp_companion MONEY1, prepare_failback
Make sure that this command executes successfully.
Restart the HACMP services on the primary node.
