Hello folks,<\/p>\n
I had a really interesting (to me) Avamar RMAN backup failure after a system was rebooted. The system had been backing up perfectly for months, and then all of the sudden:<\/p>\n
Recovery Manager: Release 11.2.0.3.0 – Production on Wed Apr 24 13:59:45 2013
\nCopyright (c) 1982, 2011, Oracle and\/or its affiliates. All rights reserved.
\nRMAN> @@RMAN-blahblah-1543-cred1.rman
\n2> connect target *;
\n3> **end-of-file**
\n4> run {
\n5> configure controlfile autobackup on;
\n6> set controlfile autobackup format for device type sbt to ‘CONTROLFILE.blahblah.%F’;
\n7> allocate channel c0 type sbt PARMS=”SBT_LIBRARY=\/usr\/local\/avamar\/lib\/libobk_avamar64.so” format ‘%d_%U’;
\n8> send channel ‘c0’ ‘”–libport=32110″ “–cacheprefix=blahblah_c0” “–sysdir=\/usr\/local\/avamar\/etc” “–bindir=\/usr\/local\/avamar\/bin” “–vardir=\/usr\/local\/avamar\/var” “–logfile=\/usr\/local\/avamar\/var\/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log0” “–ctlcallport=32108″‘;
\n9> allocate channel c1 type sbt PARMS=”SBT_LIBRARY=\/usr\/local\/avamar\/lib\/libobk_avamar64.so” format ‘%d_%U’;
\n10> send channel ‘c1’ ‘”–libport=32110″ “–cacheprefix=blahblah_c1” “–sysdir=\/usr\/local\/avamar\/etc” “–bindir=\/usr\/local\/avamar\/bin” “–vardir=\/usr\/local\/avamar\/var” “–logfile=\/usr\/local\/avamar\/var\/Oracle_RMAN-blahblah_Group-1366836935662-5002-Oracle-blahblah-avtar.log1” “–ctlcallport=32108″‘;
\n11> backup database plus archivelog delete input;
\n12> }
\n13>
\nconnected to target database (not started)
\nusing target database control file instead of recovery catalog
\nRMAN-00571: ===========================================================
\nRMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
\nRMAN-00571: ===========================================================
\nRMAN-03002: failure of configure command at 04\/24\/2013 13:59:45
\nRMAN-06403: could not obtain a fully authorized session
\nORA-01034: ORACLE not available
\nORA-27101: shared memory realm does not exist
\nRecovery Manager complete.<\/p><\/blockquote>\nSorry for the craptastic formatting there, gotta love Oracle’s gigantic errors \ud83d\ude42<\/p>\n
Being the type of guy who tries to figure things out myself, …I did the usual and checked support.emc.com, legacy Powerlink and of course Google. In my search, I came up with this EMC KB article: esg130176. It stepped me through making sure the instance was running and that the tnsnames.ora & \/etc\/oratab were correctly formatted. <\/p>\n
All of this seemed to check out, and I was even able to perform a manual RMAN backup to local disk, which boggled me further.<\/p>\n
I tried to get our DBA’s to help out, but got the usual “everything looks good on our side” answer.<\/p>\n
I resigned myself to creating an SR with EMC. The gentleman who returned my call took a quick look and immediately knew what was wrong. He mentioned that when he originally discovered this issue for another customer, it had taken a while to figure out what was going on.<\/p>\n
Somehow since the last restart, someone had managed to add a \/ to the end of oracle’s ORACLE_HOME environment variable:<\/p>\n
\n$ whoami
\noracle
\n$ env |grep ORACLE_HOME
\nORACLE_HOME=\/u01\/app\/oracle\/product\/11.2.0\/dbhome_1\/<\/p><\/blockquote>\nAvamar apparently strips off that erroneous trailing \/ as it sets its own ORACLE_HOME when it starts the job and thus breaks sqlplus, rman… all of the those tools we so dearly need.<\/p>\n
I want my DBA’s to fix that environment variable, but that would require restarting this production database. To work around this, create a file called \/usr\/local\/avamar\/var\/avoracle.cmd and put this inside it. This will FORCE Avamar to tack on the \/.<\/p>\n
\n–orahome=\/u01\/app\/oracle\/product\/11.2.0\/dbhome_1\/\n<\/p><\/blockquote>\n
Obviously you need to make that match YOUR environment!<\/p>\n
Try the backup again, and with any luck you’ll be “fixed”! I suggested to the EMC support engineer that he should add that to the above mentioned KB article, and he said he would do that.<\/p>\n
Till next time…<\/p>\n","protected":false},"excerpt":{"rendered":"
Hello folks, I had a really interesting (to me) Avamar RMAN backup failure after a system was rebooted. The system had been backing up perfectly for months, and then all of the sudden: Recovery Manager: Release 11.2.0.3.0 – Production on Wed Apr 24 13:59:45 2013 Copyright (c) 1982, 2011, Oracle and\/or its affiliates. All rights […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[78],"tags":[113,82,114,80,81],"class_list":["post-428","post","type-post","status-publish","format-standard","hentry","category-avamar","tag-avamar","tag-backup-failure","tag-emc","tag-oracle","tag-rman"],"_links":{"self":[{"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/428","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=428"}],"version-history":[{"count":17,"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/428\/revisions"}],"predecessor-version":[{"id":445,"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/428\/revisions\/445"}],"wp:attachment":[{"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=428"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ecimulti.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}