Why... Why... Why?
This blog is dedicated to documenting error resolution and other tidbits that I discover while working as a Consultant in the Oracle EPM (Hyperion) field. As much of my job revolves around issue resolution, I see an opportunity to supplement the typical troubleshooting avenues such as the Oracle Knowledgebase and Oracle Forums with more pinpointed information about specific errors as they are encountered. Beware, the information found in this blog is for informational purposes only and comes without any warranty or guarantee of accuracy.

Create your own Oracle Hyperion Virtual Environment:

Sunday, December 14, 2014

EPMVirt - New PLANDEMO sample app

The EPMVirt virtual image now includes a pre-built sample planning application. The application setup, data loads and calcs are done automatically during the installation process. Also a sample Financial Report is included. 

Sample Financial Report:


Sample Planning app:

Sunday, November 9, 2014

EPMVirt Debugging


For anyone needing assistance debugging the EPMvirt installation please run the following script to gather some debug info (as oracle)


#!/bin/sh
rm -f /tmp/epmvirt_logs.zip
rm -rf /tmp/epmvirt_debug/
mkdir /tmp/epmvirt_debug
ps -wwef > /tmp/epmvirt_debug/ps
env > /tmp/epmvirt_debug/env
tnsping HYPDB > /tmp/epmvirt_debug/tnsping
rpm -qa > /tmp/epmvirt_debug/rpm
df -h > /tmp/epmvirt_debug/df
free -m > /tmp/epmvirt_debug/free
zip /tmp/epmvirt_logs.zip /tmp/epmvirt_debug/* \
    /root/ks-post.log \
    /u0/Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/config \
    /u0/Oracle/Middleware/EPMSystem11R1/diagnostics/logs/install \
    /u0/automation/epm/installAll.log \
    /u0/app/oracle/admin/diag/rdbms/hypdb/HYPDB/trace/alert_HYPDB.log \
    /u0/automation/*.log \
    /u0/automation/database/*.log \
    /u0/automation/epm/*.log \
    /u0/automation/root/*.log 


The script will generate a file /tmp/epmvirt_logs.zip

Please email this log file to info@interopconsulting.com and I will take a closer look.

Thanks!

Sunday, August 3, 2014

Windows WebLogic Servies Fail to Start - Trouble Encrypting boot.properties

I have run across this error a few times and have finally gotten around to documenting it. The issue occurs typically after a multi node Hyperion installation and all the WebLogic managed servers not residing on the WebLogic domain fail to start up. Essentially, you start the WebLogic service, and after a minute it shuts down. Looking into the WebLogic managed server log files: 


java.lang.AssertionError: java.lang.reflect.InvocationTargetException
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:175)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1092)
at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)
at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)
at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1109)
at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)
at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:296)
at weblogic.server.channels.ChannelService.start(ChannelService.java:258)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Caused By: weblogic.security.internal.encryption.EncryptionServiceException
at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:139)
at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:187)
at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1092)
at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)
at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:63)
at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)
at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1109)
at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)
at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:296)
at weblogic.server.channels.ChannelService.start(ChannelService.java:258)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


WebLogic is trying to read the boot.properties file in order to get the credentials to startup the managed server. However, it appears it needs to contact the WebLogic Domain in order to properly encrypt the boot.properties. You would also see the failed connection to the WebLogic Domain in the logs:

<Aug 2, 2014 6:52:04 AM UTC> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at http://wldomain:7001.>
<Aug 2, 2014 6:52:04 AM UTC> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>

Having a look at the boot.properties file it looks like the password has not been encrypted, hence the EncryptionServiceException. It is trying to read what it thinks is the encrypted password but the password is plain text, causing the exception. 

C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\RaFramework0\security\boot.properties
#
#Fri Aug 01 05:06:31 UTC 2014
password=epm_admin
username=mypassword


So the question is how to get the Managed Server to properly contact the WebLogic domain to encrypt the boot.properties file. 

I usually do this by starting it from the command line, 

cd C:\Oracle\Middleware\user_projects\domains\EPMSystem\bin
startManagedWeblogic.cmd RaFramework1 http://wldomain:7001

Note, the second argument is the WebLogic Domain URL. 

While starting up, assuming the Managed Server is able to connect to the domain you will see:

<Aug 2, 2014 6:57:37 AM UTC> <Notice> <Security> <BEA-090083> <Storing boot identity in the file: C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\RaFramework1\security\boot.properties>

This is a good indication that the problem has been resolved. Looking in the boot.properties you should see the encrypted password.

This approach is what I find best for debugging. Additionally, the startup parameters for starting the WebLogic managed server as a Windows service are in the Windows Registry. Specifically for Hyperion there is a JVMOption for setting the WL Domain server. Make sure that this is set correctly when starting as a Windows service. "-Dweblogic.management.server=http://wldomain:7001"

Monday, July 28, 2014

Some Bad Patches?


I recently came across a few bad patches.

Specifically, HSS 17537889, 11.1.2.2.303

In Shared Services, Application Groups, when trying to expand LCM for HFM applications (also with EPMA applications) SSL URLs will be appended with a :80, invalidating the URL.

EPMLCM-30034:Failed in communicating to Hyperion Financial Management remote service at URL https://servername:80/hfmlcmservice/LCMWS.asmx

Note the https protocol with the :80 appended to the end. The patch cannot form SSL URL's properly. This was confirmed by rolling back the patch and seeing the URLs form properly.


Additionally, HFM 11.1.2.2.306 & 307 cause certain Financial Reports to Fail against HFM. The error in HFM is:
File: CDQISource.cpp Version: 11.1.2.2.306.4365 Line: 6408 Error: (-3)(0xFFFFFFFD)(An unknown error occurred. (-3))

Specifically, looking at the .307 Readme, under known issues:
· 18528346 – Financial reports using ‘Base of Current POV for entity’ is not working with conditional suppression.
I think there are broader issues with FR and HFM in this patch that only occur with certain report options in FR.
I suspect these two issues also reside in the 11.1.2.2.500 patch set because it was released before Oracle fixed these issues.

Sunday, June 15, 2014

Fun with the Hyperion Log Analysis Utility


 I was recently playing around with the Log Analysis tool on 11.1.2.2.

John Goodwin has a good write up of the basic operation of the log analysis utility here:
http://john-goodwin.blogspot.com/2013/09/epm-standalone-log-analysis-utility.html

After reviewing the tool, I was curious how to make it a bit more useful.

First, John seems to be pinpointing specific product log directories with the "-d" option. I am curious if it is possible to get more generic and just have the utility tell us what is wrong without specifying the product. This would be beneficial if an error is occurring in the system and you just want a quick report of what error messages might be coming out to help troubleshoot.

I tried very generically:
loganalysis.bat -system -tmin 60  -d d:\Oracle\Middleware\user_projects

However, this seemed to only get the logs in user_projects\epmsystem1\diagnostics. It was missing the Weblogic logs under user_projects\domains. These logs are important because they provide insight into what is happening at the web application server layer. To get more of a complete picture of the system, it is possible to generate a report for both the domains folder and the diagnostics folder. 
loganalysis.bat -system -tmin 60  -d d:\Oracle\Middleware\user_projects\domains -o domains_report
loganalysis.bat -system -tmin 60  -d d:\Oracle\Middleware\\user_projects\epmsystem1\diagnostics -o diagnostics_report


Now this is on the right track. The next issue is that the utility is only scanning a single node. In a distributed environment, this doesn't help the problem of having to log into multiple nodes to gather the logs. Digging a little deeper, it looks like UNC paths work well in the Log Analysis utility.
For instance,
loganalysis.bat -system -tmin 60 -d \\node1\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics -o node1_diagnostics

Using this information, it is possible to orchestrate reports for all nodes in the environment at the same time from a single node. When each report finishes it opens a html report as a tab in Internet Explorer for review. When completed the results are easily viewed by scanning through the multiple tabs in IE.

The final script to run reports for all nodes at once might look something like this:

set SEARCH_IN_MINS=60

start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS% -d \\node1\d$\Oracle\Middleware\user_projects\domains -o web1_domains"
start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS% -d \\node1\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics -o web1_diagnostics"

start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS%  -d \\node2\d$\Oracle\Middleware\user_projects\domains -o web2_domains"
start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS%  -d \\node2\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics -o web2_diagnostics"

start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS%  -d \\node3\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics -o planning1_diagnostics"
start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS%  -d \\node3\d$\Oracle\Middleware\user_projects\domains -o planning1_domains"

start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS%  -d \\node4\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics -o planning2_diagnostics"
start cmd /C "loganalysis.bat -system -tmin %SEARCH_IN_MINS%  -d \\node4\d$\Oracle\Middleware\user_projects\domains -o planning2_domains"

....

I think this is infinity more useful. Much more useful than the single node, single product example. Next time Hyperion is down and everyone is scrambling looking for answers, this script might just be a quick way to discover the issue.

Friday, May 30, 2014

Testing the FDM 32-Bit Database Connection

I recently had some issues with FDM and suspected an issue with the database client setup. However, both the 32 bit and 64 bit Oracle clients were installed on the machine to support both FDM (32 bit) and HFM (64 bit). 

Specifically, I wanted to test the Windows OLE DB Oracle Driver connection, rather than relying on sqlplus to ensure the OLE DB connection was working with the Oracle client. The 11.2.0.3 Oracle client installer has a known bug where it fails to correctly register OLE DB. However, if I was to just create a simple test .UDL file using the Oracle OLE DB provider it would default to using the 64 bit Oracle client. How could I test the 32 bit client using a UDL file when both the 32 and 64 bit Oracle client is installed?

This blog article was very helpful:
http://blogs.msdn.com/b/chaitanya_medikonduri/archive/2008/04/09/how-to-run-32-bit-udl-file-on-a-64-bit-operating-system.aspx

Ultimately, this command was able to successfully test the 32 bit Oracle Client using Oracle's OLE DB provider: 
C:\Windows\system32>C:\WINDOWS\SysWOW64\Rundll32.exe "C:\Program Files (x86)\Common Files\System\Ole DB\oledb32.dll",OpenDSLFile c:\Windows\Temp\test.udl32

This post is a tad bit "in the weeds". However, FDM provides very little indication of what is actually going on when it fails to open. This might just be useful to verify all aspects of connectivity when troubleshooting.