A leading telecom was performing maintenance work on their storage and in the process an Oracle database was corrupted. The client needed to recover the database by any means because the information it contained was critical for business operations. The recovery process also had to be performed as fast as possible to minimize regulatory sanctions and revenue loss. Itzdata experts devised a procedure to recover the database.
Maintenance work was being performed on a SAN system that contained an Oracle Database (184.108.40.206) including redologs and undo blocks (rollback segments). Possibly as a consequence of the maintenance procedure, the database was corrupted, showing error message “ORA-00600: internal error code, argument: ”. The client assumed that the database was unrecoverable and the last backup was a week old.
The main issue was a discrepancy in the scn on the datafile headers, in particular the datafile for the undo tablespace. There was no documented procedure for correcting the problem and the only available solutions found online could not guarantee results and had estimated recovery times that could have severe business impacts.
Itzdata’s experts were brought in to assess the situation of the database and recover the data as fast as possible. Initially, our experts evaluated whether a previously used solution could work. The previous solution worked by matching the scn on the datafile headers. From our experience, we found that as a database is opened repeatedly, the difference in scn decreased. This solution is discarded due to the long recovery time. A new solution was designed using a hexadecimal editing tool. The procedure involves finding the correct scn to implement and writing it on the datafiles through a massive procedure, then opening the database in read-only mode, exporting and importing the data to reestablish service.
The solution was implemented with the following procedure:
The procedure is successful, reestablishing database service in just 4 hours, recovering 100% of the data.