ORA-16631: Operation requires shutdown of database/instance “”

 Oracle  Comments Off on ORA-16631: Operation requires shutdown of database/instance “”
Mar 052013

One more funny bug. We have the following
– 4 databases in the configuration (1 primary and 3 standby)
– we have restarted one of the standbys. Everything works as expected
dgmgrl shows something different:

DGMGRL> show configuration

Configuration - DG

  Protection Mode: MaxAvailability
    primary  - Primary database
    standby1 - Physical standby database
    standby2 - Physical standby database (disabled)
    standby3 - Logical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:

(disabled) – WTF? I have to say that we have restarted standby1 – we didn’t even touch the others.

OK, let’s try the easiest solution for every DG problem:

DGMGRL> enable database standby2;

Here comes the surprise:

ORA-16631: Operation requires shutdown of database/instance ""

Errr… which instance do you need restarted?

The only good thing about this crappy error is that it lets you find the following note in MOS:
Metalink note 1258074.1

Ora-16631: Operation Requires Shutdown Of Database Or Instance “” On Physical Standby

Applies to:
Oracle Server – Enterprise Edition – Version: and later [Release: 11.1 and later ]
Information in this document applies to any platform.


ORA-16631: Operation requires shutdown of database/instance “”

Both the primary and standby were shutdown in an graceful fashion.
Since standby is shutdown before primary, standby is marked as shutdown disabled
standby is restarted first and then the primary.
standby sends a bootstrap request to the primary but since it’s not started yet,
it’s request goes unanswered. When primary is then started and as part of
its bootstrap processing the primary intentionally leaves the standby in a
shutdown disabled state. The FSFO state then transitions to an unsynchronized
state since the standby is not available.

— on primary —-

dgmgrl /
edit database 'standby db unique name here' set state='ONLINE';
show configuration;

( or )

—- on standby —

sqlplus / as sysdba
shu immediate
startup mount

OK, we have problem with both physical, and logical standby. And we did not restart any of them, nor the primary database. I can only guess that during the restart standby1 sent to primary the “I’m leaving the party” signal. Then primary made a mistake and dropped the connection to all the standbys. After taht, when standby1 has beens started, he sent a bootstrap signal. Primay have added him to the configuration, leaving the others away.
Fortunately the solution worked. Of course I choose the first one. I’m not sure what will happen is I restart standby2, maybe we will loose standby1 again… 🙂

 Posted by at 9:43