Making all other blogs seem exciting!
RSS icon Home icon
  • EMC Avamar RMAN backup fails with: RMAN-06403, ORA-01034 and ORA-27101

    Posted on April 24th, 2013 ashinn No comments

    Hello folks,

    I had a really interesting (to me) Avamar RMAN backup failure after a system was rebooted. The system had been backing up perfectly for months, and then all of the sudden:

    Recovery Manager: Release 11.2.0.3.0 – Production on Wed Apr 24 13:59:45 2013
    Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
    RMAN> @@RMAN-blahblah-1543-cred1.rman
    2> connect target *;
    3> **end-of-file**
    4> run {
    5> configure controlfile autobackup on;
    6> set controlfile autobackup format for device type sbt to ‘CONTROLFILE.blahblah.%F’;
    7> allocate channel c0 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
    8> send channel ‘c0’ ‘”–libport=32110″ “–cacheprefix=blahblah_c0” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log0” “–ctlcallport=32108″‘;
    9> allocate channel c1 type sbt PARMS=”SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so” format ‘%d_%U’;
    10> send channel ‘c1’ ‘”–libport=32110″ “–cacheprefix=blahblah_c1” “–sysdir=/usr/local/avamar/etc” “–bindir=/usr/local/avamar/bin” “–vardir=/usr/local/avamar/var” “–logfile=/usr/local/avamar/var/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log1” “–ctlcallport=32108″‘;
    11> backup database plus archivelog delete input;
    12> }
    13>
    connected to target database (not started)
    using target database control file instead of recovery catalog
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of configure command at 04/24/2013 13:59:45
    RMAN-06403: could not obtain a fully authorized session
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Recovery Manager complete.

    Sorry for the craptastic formatting there, gotta love Oracle’s gigantic errors 🙂

    Being the type of guy who tries to figure things out myself, …I did the usual and checked support.emc.com, legacy Powerlink and of course Google. In my search, I came up with this EMC KB article: esg130176. It stepped me through making sure the instance was running and that the tnsnames.ora & /etc/oratab were correctly formatted.

    All of this seemed to check out, and I was even able to perform a manual RMAN backup to local disk, which boggled me further.

    I tried to get our DBA’s to help out, but got the usual “everything looks good on our side” answer.

    I resigned myself to creating an SR with EMC. The gentleman who returned my call took a quick look and immediately knew what was wrong. He mentioned that when he originally discovered this issue for another customer, it had taken a while to figure out what was going on.

    Somehow since the last restart, someone had managed to add a / to the end of oracle’s ORACLE_HOME environment variable:

    $ whoami
    oracle
    $ env |grep ORACLE_HOME
    ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/

    Avamar apparently strips off that erroneous trailing / as it sets its own ORACLE_HOME when it starts the job and thus breaks sqlplus, rman… all of the those tools we so dearly need.

    I want my DBA’s to fix that environment variable, but that would require restarting this production database. To work around this, create a file called /usr/local/avamar/var/avoracle.cmd and put this inside it. This will FORCE Avamar to tack on the /.

    –orahome=/u01/app/oracle/product/11.2.0/dbhome_1/

    Obviously you need to make that match YOUR environment!

    Try the backup again, and with any luck you’ll be “fixed”! I suggested to the EMC support engineer that he should add that to the above mentioned KB article, and he said he would do that.

    Till next time…