One more funny bug. We have the following
– 4 databases in the configuration (1 primary and 3 standby)
– we have restarted one of the standbys. Everything works as expected
dgmgrl shows something different:
DGMGRL> show configuration Configuration - DG Protection Mode: MaxAvailability Databases: primary - Primary database standby1 - Physical standby database standby2 - Physical standby database (disabled) standby3 - Logical standby database (disabled) Fast-Start Failover: DISABLED Configuration Status: SUCCESS
(disabled) – WTF? I have to say that we have restarted
standby1 – we didn’t even touch the others.
OK, let’s try the easiest solution for every DG problem:
DGMGRL> enable database standby2;
Here comes the surprise:
ORA-16631: Operation requires shutdown of database/instance ""
Errr… which instance do you need restarted?
The only good thing about this crappy error is that it lets you find the following note in MOS:
Metalink note 1258074.1
Ora-16631: Operation Requires Shutdown Of Database Or Instance “” On Physical Standby
Oracle Server – Enterprise Edition – Version: 18.104.22.168 and later [Release: 11.1 and later ]
Information in this document applies to any platform.
ORA-16631: Operation requires shutdown of database/instance “”
Both the primary and standby were shutdown in an graceful fashion.
Since standby is shutdown before primary, standby is marked as shutdown disabled
standby is restarted first and then the primary.
standby sends a bootstrap request to the primary but since it’s not started yet,
it’s request goes unanswered. When primary is then started and as part of
its bootstrap processing the primary intentionally leaves the standby in a
shutdown disabled state. The FSFO state then transitions to an unsynchronized
state since the standby is not available.
— on primary —-dgmgrl / edit database 'standby db unique name here' set state='ONLINE'; show configuration; exit
( or )
—- on standby —sqlplus / as sysdba shu immediate startup mount
OK, we have problem with both physical, and logical standby. And we did not restart any of them, nor the primary database. I can only guess that during the restart
standby1 sent to
primary the “I’m leaving the party” signal. Then
primary made a mistake and dropped the connection to all the standbys. After taht, when
standby1 has beens started, he sent a bootstrap signal.
Primay have added him to the configuration, leaving the others away.
Fortunately the solution worked. Of course I choose the first one. I’m not sure what will happen is I restart
standby2, maybe we will loose
standby1 again… 🙂