ORA-00600: ORA-10567: ORA-10560: Physical standby redo applying error

It is not the first time we face the problem with applying redo on physical standby. Very old environment – Oracle 10gr2 with raw devices on old SUN storage believed to be suffering from physical lost writes. Particularly at the heavy load on primary where loads of redo log files are being generated.
So, the errors in alert.log are as follow:

ORA-00600: internal error code, arguments: [3020], [87], [154002], [365058450], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 87, block# 154002)
ORA-10564: tablespace CB_IND
ORA-01110: data file 87: '/dev/vx/rdsk/ora02dg/cb_ind05'
ORA-10560: block type '0'

Famous ORA-00600 error when metalink gets handful. It says:

“This error can be reported if any of these updates are lost for some reason.
Therefore, thoroughly check your operating system and disk hardware.
In the case of a lost update, restore an old copy of the datafile and attempt to recover and roll forward again. (Doc ID 30866.1)”

Our only option was a restore by oracle as os investigation would take significant time. Even more, you still need to restore damaged file from oracle at the end. So, as a recommendation we suggest not thinking much and start immediate restore.

Steps are as follow:

1. primary database –

 backup as copy datafile 87 format '/share/sbtarch/ind5'; 

(raw devices add more trick to backup and restore operations. In case of normal fs it would be possible to put corresponding TS in backup mode and just a copy the datafile )

2. primary database –

 copy current controlfile for standby TO '/share/sbtarch/control03.dbf'  

3. standby database – mount instance with newly copied control file

4. standby database –

 	run { 
             ALLOCATE CHANNEL ch02 TYPE DISK;
 	     restore datafile 87; 
            }

5. on standby (in case of rman catalog you might avoid this step)

 catalog start with '/oracle/arch';  after control file copied 

6. on standby –

 ALTER DATABASE RECOVER MANAGED STANDBY DATABASE NODELAY disconnect from session; 

At this point, recovery started and all archived log files in queue applied successfully.

Thanks,

One thought on “ORA-00600: ORA-10567: ORA-10560: Physical standby redo applying error

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s