If a companion server issues error message 18750, check the @@cmpstate of your servers. If your primary companion is in normal companion mode, but the secondary companion is in secondary failover mode, your cluster is in an inconsistent state, and you need to manually recover from this. This inconsistent state may be caused by an sp_companion 'prepare_failback' command failing on the secondary companion. You can determine whether this happened by examining the log on the secondary node. To recover from this, perform the following steps manually:
Shut down both the primary and the secondary companions by halting both their packages.
Reboot the secondary companion by starting the package for the secondary companion.
Repair all databases marked “suspect.” To determine which databases are suspect, issue:
select name, status from sysdatabases
Databases marked suspect have a status value of 320.
Allow updates to system tables:
sp_configure “allow updates”, 1
For each suspect, failed-over database, perform the following:
1> update sysdatabase set status=status-256 where name='database_name' 2> go 1> dbcc traceon(3604) 2> go 1> dbcc dbrecover(database_name) 2>go
From the secondary companion, issue:
sp_companion primary_companion_name, prepare_failback
For example, from secondary companion PERSONEL1:
sp_companion MONEY1, prepare_failback
Make sure that this command executes successfully.
Resume normal companion mode. From the primary companion, issue:
sp_companion secondary_companion, resume
For example, from the primary companion MONEY1:
sp_companion PERSONEL1, resume
