Why... Why... Why?
This blog is dedicated to documenting error resolution and other tidbits that I discover while working as a Consultant in the Oracle EPM (Hyperion) field. As much of my job revolves around issue resolution, I see an opportunity to supplement the typical troubleshooting avenues such as the Oracle Knowledgebase and Oracle Forums with more pinpointed information about specific errors as they are encountered. Beware, the information found in this blog is for informational purposes only and comes without any warranty or guarantee of accuracy.

EPMVirt: Create your own Oracle Hyperion Virtual Environment:

Tuesday, January 31, 2012

Opatch Not Displaying Correct Patches (Opatch Accuracy)

I was recently tasked with checking the patch level on a large scale implementation which I was not familiar with. The first action was to do the opatch -lsinventory on each server to get a list of applied patches. After meticulously categorizing each patch level indicated by lsinventory I discovered some inconsistencies. This led me to question the accuracy of the results of the lsinventory command. Consequently, I started to analyze the summary log file and discovered that some of the patch numbers were actually applied to the server which were not being returned by opatch. The summary file is:

In the summary file it lists the particular log file for each opatch operation.
For example,

So I dug a little deeper and analyzed the opatch logs for the patches which were not showing up in lsinventory. It appears the patch WAS successfully applied. In addition, to confirm my suspicions, I compared file by file using WinDiff directory comparison. Two servers, which should be identical, showed inconsistencies in the opatch lsinventory. The results showed, even with lsinventory implying the patch levels were different, the binary WinDiff comparison as well as the opatch logs confirmed the correct patches had been applied to both servers.

To further illustrate this point, the following knowledgebase article identifies that some commands can completely destroy the patch level information that opatch contains:
ID 1357748.1

IMPORTANT! Running createinventory.bat discards all patching information, reverting to a simple list of installed products, even if there are installed patches on the system. This does not affect the correct operation of the products and patches, but it could cause issues if a new patch must be applied over an existing patch. Because there is no record of the existing patch in the inventory, OPatch will refuse to apply the newer patch. The only way to restore the patch information in the inventory is to re-apply the patches.

What good is a patching tool which has the possibility of permanently truncating your patch history?

My conclusion is that the lsinventory does not necessarly give the correct results when it comes to patching, which is disappointing when using opatch as a patching tool.

So, assuming that opatch -lsinventory is inaccurate, what is an alternative method for determining the patch levels within the environment?  As indicated previously, each opatch operation is carefully logged. It is possible to review the logs and determine which patches have been successfully applied. However, over time many logs can accumulate.

Using windows grepping utility, AstroGrep, it is possible to quickly sift through the log directory and point out the patch instances which were successful. For example, search for the string, "check OK: Patch ID (.+) is registered"


  1. The OPatch inventory is saved in D:\Oracle\Middleware\EPMSystem11R1\inventory\ContentsXML\comps.xml (adjust to match your Middleware home location). You can get your inventory back by restoring this file from filesystem backups.

    There is another workaround too. OPatch stores a local backup of the inventory before applying a patch and even a batch file that can be used to restore the inventory to this saved state. The directories are stored in D:\Oracle\Middleware\EPMSystem11R1\.patch_storage and look like this:

    16516547_Mar_29_2013_15_21_54 <<---

    In the latest directory you can find restore.bat that you can run. I needed to set ORACLE_HOME first to get it working:

    cd /d D:\Oracle\Middleware\EPMSystem11R1\.patch_storage\16516547_Mar_29_2013_15_21_54
    set ORACLE_HOME=D:\Oracle\Middleware\EPMSystem11R1

    Now your OPatch should be able to list installed patches again. The only problem is that the saved state is stored *before* applying the latest patch so you need to add it to the inventory manually (in my case it was patch 16516547 and the patch folder was still stored under the OPatch directory). The no_sysmod option will make OPatch skip any actual changes to the installation files so only the inventory is updated.

    cd /d D:\Oracle\Middleware\EPMSystem11R1\OPatch
    opatch apply -no_sysmod hfm_pse16516547 -oh D:\Oracle\Middleware\EPMSystem11R1 -jre d:\Oracle\Middleware\jdk160_29

    Now your inventory should be back in business!

  2. Hi Nicholas. I too, ran into the same EXACT issue. I was finally able to determine in the Oracle inventory logs that a reinstall (installTool.cmd) rolled my opatches back. Upon this discovery, I was quite frustrated, and irritated that no trace or logs were left ANYWHERE in the OPatch directory or it's corresponding logs...