Recovery of Database with Corrupt
Redologs and Datafiles at Leading Regional Telecom

Case Study: Recovery of Database with Corrupt Redologs and Datafiles at Leading Regional Telecom

Overview

A leading telecom was performing maintenance work on their storage and in the process an Oracle database was corrupted. The client needed to recover the database by any means because the information it contained was critical for business operations. The recovery process also had to be performed as fast as possible to minimize regulatory sanctions and revenue loss. Itzdata experts devised a procedure to recover the database.

Situation

Maintenance work was being performed on a SAN system that contained an Oracle Database (11.2.0.4) including redologs and undo blocks (rollback segments). Possibly as a consequence of the maintenance procedure, the database was corrupted, showing error message “ORA-00600: internal error code, argument: [2662]”. The client assumed that the database was unrecoverable and the last backup was a week old.

Issues

The main issue was a discrepancy in the scn on the datafile headers, in particular the datafile for the undo tablespace. There was no documented procedure for correcting the problem and the only available solutions found online could not guarantee results and had estimated recovery times that could have severe business impacts.

Solution

Itzdata’s experts were brought in to assess the situation of the database and recover the data as fast as possible. Initially, our experts evaluated whether a previously used solution could work. The previous solution worked by matching the scn on the datafile headers. From our experience, we found that as a database is opened repeatedly, the difference in scn decreased. This solution is discarded due to the long recovery time. A new solution was designed using a hexadecimal editing tool. The procedure involves finding the correct scn to implement and writing it on the datafiles through a massive procedure, then opening the database in read-only mode, exporting and importing the data to reestablish service.

Implementation

The solution was implemented with the following procedure:

  1. Write a script that lets us substitute information in the datafile header blocks in a massive manner
  2. After several iterations, the correct blocks are identified where the scn is kept at the datafile level
  3. Using oracledebug, the correct scn is obtained and applied to all the datafile headers
  4. The database is opened in read-only mode and a full data export is performed

Results

The procedure is successful, reestablishing database service in just 4 hours, recovering 100% of the data.

Lead Consultant

Gerber Bautista