RAC Attack - Oracle Cluster Database at Home/Patching Grid and Database Software

Patching Overview

 * Upgrades to the "base version" are very complicated and always use the full Oracle installer (runInstaller). Major new features are only introduced in new base versions.
 * Patch Sets are also installed with the full Oracle installer. Historically, each patchset was installed on top of the base version (top row in the illustration) by using runInstaller.  However, starting with 11.2.0.2 the patch sets can be installed as a new installation without the base version.  It is now recommended to perform Patch Set upgrades "out-of-place" in this manner.  Sometimes new features are also included with Patch Sets (for example RAT data collection).
 * PSUs are installed with opatch. They include security updates and important bug fixes.  They are released quarterly and always include the latest CPU.
 * CPUs are installed with opatch. CPUs include only security updates, and are also released quarterly.  They cannot be applied after you have applied any PSU.  (Until you upgrade to a new patch set or base version.)

Support Status and Known Issues
Before performing any installation or upgrade of Oracle, you should always check the Support Status and Known Issues for the release. Metalink note 161818.1 is always the starting point – open this note and review it. Next, follow the link for 11.2.0.X to metalink note 880782.1 and review that note. Finally, follow the link to note 880707.1 and review the known issues with Oracle 11.2.0.1 which is the version we will be using for this lab.

Practice Rolling Application
For this lab, the instructor has provided recent PSUs. PSUs and CPUs are collections of one-off patches. One-off patches can only be applied to an Oracle database in a rolling manner if they have been certified for rolling upgrades.

 Review the installation instructions. We're going to install three patches and you can find the README files at these locations:
 * /mnt/cdrom5/patch/psu6-db-12419378/12419378/README.html
 * /mnt/cdrom5/patch/psu2-gi-9655006/README.html
 * /mnt/cdrom5/patch/opatch-6880880/README.txt

 First we need to update the OPatch utility. Find patch 6880880 on the instructor-provided CDROM and unzip it directly into both the grid home and the database home. Before unzipping the file, backup the existing OPatch programs.

$ oenv SIDs here are: grid +ASM1 RAC1 ORACLE_SID = [RAC1] ? grid $ cd $ORACLE_HOME $ tar czvf ~/gi-opatch-backup_$(date +%Y-%m-%d).tgz OPatch/ ... $ unzip -o /mnt/cdrom*/patch/opatch-6880880/p6880880_112000_LINUX.zip ... $ OPatch/opatch version Invoking OPatch 11.2.0.1.5 OPatch Version: 11.2.0.1.5 OPatch succeeded. $ oenv SIDs here are: grid +ASM1 RAC1 ORACLE_SID = [grid] ? RAC1 $ cd $ORACLE_HOME $ mv OPatch OPatch.backup $ unzip /mnt/cdrom*/patch/opatch-6880880/p6880880_112000_LINUX.zip ... $ OPatch/opatch version Invoking OPatch 11.2.0.1.5 OPatch Version: 11.2.0.1.5 OPatch succeeded.

 Repeat the previous step on node collabn2.

$ ssh collabn2 ... repeat previous step ... $ exit

 This new version of OPatch requires an "OCM response file" for certain operations. Use the OCM utility to generate this file. We don't want to configure OCM; leave your username blank and confirm that "YES" you don't want to enter any account information.

$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -output /home/oracle/ocm.rsp OCM Installation Response Generator 10.3.4.0.0 - Production Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name:   You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y The OCM configuration response file (/home/oracle/ocm.rsp) was successfully created.

 Find the Grid Infrastructure and Database PSUs on the instructor-provided DVD.

$ cd /mnt/cdrom5/patch $ ls opatch-6880880 psu2-gi-9655006  psu6-db-12419378

 We know that these PSUs can be applied in a rolling manner, but verify this.

$ for D in psu*/[0-9]*; do echo -n $D: cd $D $ORACLE_HOME/OPatch/opatch query -is_rolling_patch | grep rolling cd ../.. done psu2-gi-9655006/9654983: Patch is a rolling patch: true psu2-gi-9655006/9655006: Patch is a rolling patch: true psu6-db-12419378/12419378: Patch is a rolling patch: true

 We also know that there won't be any patch conflicts (since these are the first patches we're installing) – but verify this too.

$ oenv ORACLE_SID = [RAC1] ? grid [grid]$ cd psu2-gi-9655006 psu2-gi-9655006[grid]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./9655006 ... Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. [grid]$ oenv ORACLE_SID = [RAC1] ? RAC1 [RAC1]$ cd ../psu6-db-12419378/ psu6-db-12419378[RAC1]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./12419378 ... Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed.

</li><li style="margin-top:3em"> The automated patch application process will automatically shutdown and restart all database processes on the node. However, we don't want the automatic restart – because we are applying two PSUs (one for grid and one for database). Disable the instance auto-start for node collabn1 and manually shutdown the instance for patch application.

collabn1$ srvctl disable instance -d rac -i RAC1 collabn1$ srvctl stop instance -d rac -i RAC1

</li><li style="margin-top:3em"> We will use automated patch application to apply the grid infrastructure PSU. First use oenv to enter the grid environment, then switch to the root user and run opatch auto.
 * Note: any database processes on this node will be automatically shutdown during patching.
 * When prompted for the OCM response file path, enter /home/oracle/ocm.rsp and press enter.
 * When prompted about the grid home being shared, type yes to continue.

$ oenv SIDs here are: grid +ASM1 RAC1 ORACLE_SID = [RAC1] ? grid The Oracle base for ORACLE_HOME=/u01/grid/oracle/product/11.2.0/grid_1 is /u01/app/oracle [grid]$ cd /mnt/cdrom5/patch/psu2-gi-9655006 psu2-gi-9655006[grid]$ su Password: racattack psu2-gi-9655006# $'''ORACLE_HOME/OPatch/opatch auto. -oh $ORACLE_HOME''' Executing /usr/bin/perl /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/patch112.pl -patchdir. -patchn. -oh /u01/grid/oracle/product/11.2.0/grid_1 -paramfile /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params opatch auto log file location is /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-01_22-44-08.log Detected Oracle Clusterware install Using configuration parameter file: /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params OPatch is bundled with OCM, Enter the absolute OCM response file path:  /home/oracle/ocm.rsp  Can't change permissions of ./.: Read-only file system Unable to determine if /u01/grid/oracle/product/11.2.0/grid_1 is shared oracle home Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no): yes  Successfully unlock /u01/grid/oracle/product/11.2.0/grid_1 patch ././9655006 apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1 patch ././9654983 apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1 ACFS-9300: ADVM/ACFS distribution files found. ACFS-9312: Existing ADVM/ACFS installation detected. ACFS-9314: Removing previous ADVM/ACFS installation. ACFS-9315: Previous ADVM/ACFS components successfully removed. ACFS-9307: Installing requested ADVM/ACFS software. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9321: Creating udev for ADVM/ACFS. ACFS-9323: Creating module dependencies - this may take some time. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9309: ADVM/ACFS installation correctness verified. CRS-4123: Oracle High Availability Services has been started.

</li><li style="margin-top:3em"> Return to the oracle user and find the Database PSU on the instructor-provided DVD. Use oenv to enter the database environment, then run opatch apply.
 * Run OPatch in local mode so that it does not attempt an automatic rolling install across the entire cluster.
 * When prompted about OCM, leave your username blank and confirm with "y" that you don't want to enter any account information.
 * When prompted whether the local home is ready for patching, type "y" to continue.

$ cd ../psu6-db-12419378 psu6-db-12419378$ oenv SIDs here are: grid +ASM1 RAC1 ORACLE_SID = [grid] ? RAC1 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle psu6-db-12419378[RAC1]$ cd 12419378 12419378[RAC1]$ $ORACLE_HOME/OPatch/opatch apply -local Invoking OPatch 11.2.0.1.5 Oracle Interim Patch Installer version 11.2.0.1.5 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home      : /u01/app/oracle/product/11.2.0/db_1 Central Inventory : /u01/app/oraInventory from          : /etc/oraInst.loc OPatch version   : 11.2.0.1.5 OUI version      : 11.2.0.1.0 OUI location     : /u01/app/oracle/product/11.2.0/db_1/oui Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-04-02_16-10-50PM.log Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt ApplySession applying interim patch '12419378' to OH '/u01/app/oracle/product/11.2.0/db_1' Running prerequisite checks... Patch 12419378: Optional component(s) missing : [ oracle.client, 11.2.0.1.0 ] Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name:   You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y You selected -local option, hence OPatch will patch the local system only. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/11.2.0/db_1') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files and inventory (not for auto-rollback) for the Oracle Home Backing up files affected by the patch '12419378' for restore. This might take a while... Backing up files affected by the patch '12419378' for rollback. This might take a while... Execution of 'sh /mnt/cdrom5/patch/psu6-db-12419378/12419378/custom/scripts/pre -apply 12419378 ': Return Code = 0 Patching component oracle.rdbms.rsf, 11.2.0.1.0... ... ... ... ... Running make for target itnslsnr ApplySession adding interim patch '12419378' to inventory Verifying the update... Inventory check OK: Patch ID 12419378 is registered in Oracle Home inventory with proper meta-data. Files check OK: Files from Patch ID 12419378 are present in Oracle Home. The local system has been patched and can be restarted. OPatch succeeded.
 * 1) exit

</li><li style="margin-top:3em"> Enable and start the Oracle database instance on node collabn1. After the instance is running, stop and disable the instance on node collabn2. There should be no point at which the database is not running.

$ srvctl enable instance -d rac -i RAC1 $ srvctl start instance -d rac -i RAC1

$ srvctl disable instance -d rac -i RAC2 $ srvctl stop instance -d rac -i RAC2

</li><li style="margin-top:3em"> Use automated patch application to apply the grid infrastructure PSU to collabn2.
 * Note: any database processes on this node will be automatically shutdown during patching.
 * Copy the OCM response file (/home/oracle/ocm.rsp) to collabn2.
 * When prompted about the grid home being shared, type yes to continue.

$ scp /home/oracle/ocm.rsp collabn2:/home/oracle/ $ ssh collabn2 $ oenv SIDs here are: grid +ASM2 RAC2 ORACLE_SID = [RAC2] ? grid The Oracle base for ORACLE_HOME=/u01/grid/oracle/product/11.2.0/grid_1 is /u01/app/oracle [grid]$ cd /mnt/cdrom5/patch/psu2-gi-9655006 psu2-gi-9655006[grid]$ su Password: racattack psu2-gi-9655006# $'''ORACLE_HOME/OPatch/opatch auto. -oh $ORACLE_HOME''' Executing /usr/bin/perl /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/patch112.pl -patchdir. -patchn. -oh /u01/grid/oracle/product/11.2.0/grid_1 -paramfile /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params opatch auto log file location is /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-02_16-42-52.log Detected Oracle Clusterware install Using configuration parameter file: /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params OPatch is bundled with OCM, Enter the absolute OCM response file path:  /home/oracle/ocm.rsp  Can't change permissions of ./.: Read-only file system Unable to determine if /u01/grid/oracle/product/11.2.0/grid_1 is shared oracle home Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no): yes  Successfully unlock /u01/grid/oracle/product/11.2.0/grid_1 patch ././9655006 apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1 patch ././9654983 apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1 ACFS-9300: ADVM/ACFS distribution files found. ACFS-9312: Existing ADVM/ACFS installation detected. ACFS-9314: Removing previous ADVM/ACFS installation. ACFS-9315: Previous ADVM/ACFS components successfully removed. ACFS-9307: Installing requested ADVM/ACFS software. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9321: Creating udev for ADVM/ACFS. ACFS-9323: Creating module dependencies - this may take some time. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9309: ADVM/ACFS installation correctness verified. CRS-4123: Oracle High Availability Services has been started.

</li><li style="margin-top:3em"> Return to the oracle user and find the Database PSU on the instructor-provided DVD. Use oenv to enter the database environment, then run opatch apply.
 * Run OPatch in local mode so that it does not attempt an automatic rolling install across the entire cluster.
 * When prompted about OCM, leave your username blank and confirm with "y" that you don't want to enter any account information.
 * When prompted whether the local home is ready for patching, type "y" to continue.

$ cd ../psu6-db-12419378 psu6-db-12419378$ oenv SIDs here are: grid +ASM2 RAC2 ORACLE_SID = [grid] ? RAC2 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle psu6-db-12419378[RAC2]$ cd 12419378 12419378[RAC2]$ $ORACLE_HOME/OPatch/opatch apply -local Invoking OPatch 11.2.0.1.5 Oracle Interim Patch Installer version 11.2.0.1.5 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home      : /u01/app/oracle/product/11.2.0/db_1 Central Inventory : /u01/app/oraInventory from          : /etc/oraInst.loc OPatch version   : 11.2.0.1.5 OUI version      : 11.2.0.1.0 OUI location     : /u01/app/oracle/product/11.2.0/db_1/oui Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-04-02_17-11-00PM.log Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt ApplySession applying interim patch '12419378' to OH '/u01/app/oracle/product/11.2.0/db_1' Running prerequisite checks... Patch 12419378: Optional component(s) missing : [ oracle.client, 11.2.0.1.0 ] Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name:   You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y You selected -local option, hence OPatch will patch the local system only. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/11.2.0/db_1') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files and inventory (not for auto-rollback) for the Oracle Home Backing up files affected by the patch '12419378' for restore. This might take a while... Backing up files affected by the patch '12419378' for rollback. This might take a while... Execution of 'sh /mnt/cdrom5/patch/psu6-db-12419378/12419378/custom/scripts/pre -apply 12419378 ': Return Code = 0 Patching component oracle.rdbms.rsf, 11.2.0.1.0... ... ... ... ... Running make for target itnslsnr ApplySession adding interim patch '12419378' to inventory Verifying the update... Inventory check OK: Patch ID 12419378 is registered in Oracle Home inventory with proper meta-data. Files check OK: Files from Patch ID 12419378 are present in Oracle Home. The local system has been patched and can be restarted. OPatch succeeded.
 * 1) exit

</li><li style="margin-top:3em"> Enable and restart the Oracle database instance on node collabn2.

$ srvctl enable instance -d rac -i RAC2 $ srvctl start instance -d rac -i RAC2 $ srvctl status database -d rac Instance RAC1 is running on node collabn1 Instance RAC2 is running on node collabn2

</li><li style="margin-top:3em"> Run the catbundle.sql script.

$ ss SQL> @?/rdbms/admin/catbundle.sql psu apply $ less /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/catbundle/ catbundle_PSU_RAC_APPLY_2011Apr02_17_28_19.log

</li><li style="margin-top:3em"> Optional: if you want more practice working with patches, then try rolling back the database PSU and then try applying it in automated rolling mode (without local flag) or in the “minimum downtime” mode.

</li></ol>