Making all other blogs seem exciting!
RSS icon Home icon
  • EMC Avamar RMAN backup fails with: RMAN-06403, ORA-01034 and ORA-27101

    Posted on April 24th, 2013 ashinn 1 comment

    Hello folks,

    I had a really interesting (to me) Avamar RMAN backup failure after a system was rebooted. The system had been backing up perfectly for months, and then all of the sudden:

    Recovery Manager: Release 11.2.0.3.0 – Production on Wed Apr 24 13:59:45 2013
    Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
    RMAN> @@RMAN-blahblah-1543-cred1.rman
    2> connect target *;
    3> **end-of-file**
    4> run {
    5> configure controlfile autobackup on;
    6> set controlfile autobackup format for device type sbt to ‘CONTROLFILE.blahblah.%F’;
    7> allocate channel c0 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
    8> send channel ‘c0’ ‘”–libport=32110″ “–cacheprefix=blahblah_c0” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log0” “–ctlcallport=32108″‘;
    9> allocate channel c1 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
    10> send channel ‘c1’ ‘”–libport=32110″ “–cacheprefix=blahblah_c1” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log1” “–ctlcallport=32108″‘;
    11> backup database plus archivelog delete input;
    12> }
    13>
    connected to target database (not started)
    using target database control file instead of recovery catalog
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of configure command at 04/24/2013 13:59:45
    RMAN-06403: could not obtain a fully authorized session
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Recovery Manager complete.

    Sorry for the craptastic formatting there, gotta love Oracle’s gigantic errors 🙂

    Being the type of guy who tries to figure things out myself, …I did the usual and checked support.emc.com, legacy Powerlink and of course Google. In my search, I came up with this EMC KB article: esg130176. It stepped me through making sure the instance was running and that the tnsnames.ora & /etc/oratab were correctly formatted.

    All of this seemed to check out, and I was even able to perform a manual RMAN backup to local disk, which boggled me further.

    I tried to get our DBA’s to help out, but got the usual “everything looks good on our side” answer.

    I resigned myself to creating an SR with EMC. The gentleman who returned my call took a quick look and immediately knew what was wrong. He mentioned that when he originally discovered this issue for another customer, it had taken a while to figure out what was going on.

    Somehow since the last restart, someone had managed to add a / to the end of oracle’s ORACLE_HOME environment variable:

    $ whoami
    oracle
    $ env |grep ORACLE_HOME
    ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/

    Avamar apparently strips off that erroneous trailing / as it sets its own ORACLE_HOME when it starts the job and thus breaks sqlplus, rman… all of the those tools we so dearly need.

    I want my DBA’s to fix that environment variable, but that would require restarting this production database. To work around this, create a file called /usr/local/avamar/var/avoracle.cmd and put this inside it. This will FORCE Avamar to tack on the /.

    –orahome=/u01/app/oracle/product/11.2.0/dbhome_1/

    Obviously you need to make that match YOUR environment!

    Try the backup again, and with any luck you’ll be “fixed”! I suggested to the EMC support engineer that he should add that to the above mentioned KB article, and he said he would do that.

    Till next time…

     

    One response to “EMC Avamar RMAN backup fails with: RMAN-06403, ORA-01034 and ORA-27101”

    1. I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.

    Leave a reply