I was recently tasked with checking the patch level on a large scale implementation which I was not familiar with. The first action was to do the opatch -lsinventory on each server to get a list of applied patches. After meticulously categorizing each patch level indicated by lsinventory I discovered some inconsistencies. This led me to question the accuracy of the results of the lsinventory command. Consequently, I started to analyze the summary log file and discovered that some of the patch numbers were actually applied to the server which were not being returned by opatch. The summary file is:
G:\Oracle\Middleware\EPMSystem11R1\cfgtoollogs\opatch\opatch_history
In the summary file it lists the particular log file for each opatch operation.
For example,
G:\Oracle\Middleware\EPMSystem11R1\cfgtoollogs\opatch\opatch2012-01-17_17-37-26PM
So I dug a little deeper and analyzed the opatch logs for the patches which were not showing up in lsinventory. It appears the patch WAS successfully applied. In addition, to confirm my suspicions, I compared file by file using WinDiff directory comparison. Two servers, which should be identical, showed inconsistencies in the opatch lsinventory. The results showed, even with lsinventory implying the patch levels were different, the binary WinDiff comparison as well as the opatch logs confirmed the correct patches had been applied to both servers.
To further illustrate this point, the following knowledgebase article identifies that some commands can completely destroy the patch level information that opatch contains:
ID 1357748.1
IMPORTANT! Running createinventory.bat discards all patching information, reverting to a simple list of installed products, even if there are installed patches on the system. This does not affect the correct operation of the products and patches, but it could cause issues if a new patch must be applied over an existing patch. Because there is no record of the existing patch in the inventory, OPatch will refuse to apply the newer patch. The only way to restore the patch information in the inventory is to re-apply the patches.
What good is a patching tool which has the possibility of permanently truncating your patch history?
My conclusion is that the lsinventory does not necessarly give the correct results when it comes to patching, which is disappointing when using opatch as a patching tool.
So, assuming that opatch -lsinventory is inaccurate, what is an alternative method for determining the patch levels within the environment? As indicated previously, each opatch operation is carefully logged. It is possible to review the logs and determine which patches have been successfully applied. However, over time many logs can accumulate.
Using windows grepping utility, AstroGrep, it is possible to quickly sift through the log directory and point out the patch instances which were successful. For example, search for the string, "check OK: Patch ID (.+) is registered"
The OPatch inventory is saved in D:\Oracle\Middleware\EPMSystem11R1\inventory\ContentsXML\comps.xml (adjust to match your Middleware home location). You can get your inventory back by restoring this file from filesystem backups.
ReplyDeleteThere is another workaround too. OPatch stores a local backup of the inventory before applying a patch and even a batch file that can be used to restore the inventory to this saved state. The directories are stored in D:\Oracle\Middleware\EPMSystem11R1\.patch_storage and look like this:
13936087_Aug_2_2012_09_03_38
14020629_Aug_06_2012_20_32_09
14036837_13983542_backup
14036837_Aug_3_2012_00_56_12
15850273_Jan_29_2013_19_55_10
16403500_Mar_5_2013_16_27_03
16516547_Mar_29_2013_15_21_54 <<---
In the latest directory you can find restore.bat that you can run. I needed to set ORACLE_HOME first to get it working:
cd /d D:\Oracle\Middleware\EPMSystem11R1\.patch_storage\16516547_Mar_29_2013_15_21_54
set ORACLE_HOME=D:\Oracle\Middleware\EPMSystem11R1
restore.bat
Now your OPatch should be able to list installed patches again. The only problem is that the saved state is stored *before* applying the latest patch so you need to add it to the inventory manually (in my case it was patch 16516547 and the patch folder was still stored under the OPatch directory). The no_sysmod option will make OPatch skip any actual changes to the installation files so only the inventory is updated.
cd /d D:\Oracle\Middleware\EPMSystem11R1\OPatch
opatch apply -no_sysmod hfm_pse16516547 -oh D:\Oracle\Middleware\EPMSystem11R1 -jre d:\Oracle\Middleware\jdk160_29
Now your inventory should be back in business!
Hi Nicholas. I too, ran into the same EXACT issue. I was finally able to determine in the Oracle inventory logs that a reinstall (installTool.cmd) rolled my opatches back. Upon this discovery, I was quite frustrated, and irritated that no trace or logs were left ANYWHERE in the OPatch directory or it's corresponding logs...
ReplyDelete