Why... Why... Why?
This blog is dedicated to documenting error resolution and other tidbits that I discover while working as a Consultant in the Oracle EPM (Hyperion) field. As much of my job revolves around issue resolution, I see an opportunity to supplement the typical troubleshooting avenues such as the Oracle Knowledgebase and Oracle Forums with more pinpointed information about specific errors as they are encountered. Beware, the information found in this blog is for informational purposes only and comes without any warranty or guarantee of accuracy.

Thursday, April 16, 2015

Linux Tips - Beyond the Basics (Part 2)

Find log files in / last modified within 1 day
This is great when you are on an unfamiliar system or don’t know which log file you need to look at.
find / -name "*.log" -mtime -1

Interactively look at recent logs
Each log file modified within 1 day is constantly streamed. Simply retry the operation and watch the new log messages appear on the screen.
find / -name "*.log" -mtime -1 | xargs tail -f

Parsing Command Output
It is good to be able to quickly parse output from the command line. Fortunately with awk and grep, many operations are possible with basic knowledge.

Hypothetically, let’s say you want to find the number of network packets on the eth1 interface.
There are some easy commands to get this other commands to get this information, but, for the sake of learning, this example just uses basic parsing.

First the ifconfig command tells us this info on each interface, including the number of received packets, RX packets:

eth1      Link encap:Ethernet  HWaddr 08:00:27:B0:6A:94
         inet addr:  Bcast:  Mask:
         inet6 addr: fe80::a00:27ff:feb0:6a94/64 Scope:Link
         RX packets:1522 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1461 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:133938 (130.7 KiB)  TX bytes:339721 (331.7 KiB)

lo        Link encap:Local Loopback
         inet addr:  Mask:
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:65536  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

But we want to only focus on the RX packets. Let's first eliminate the loopback interface. You could easily run ifconfig eth1, but it is also possible to grep for the data.

The following will focus on the "eth1" line and also display 5 lines after the eth1 match to get the RX packets line:

ifconfig | grep eth1 -A5

eth1      Link encap:Ethernet  HWaddr 08:00:27:B0:6A:94
         inet addr:  Bcast:  Mask:
         inet6 addr: fe80::a00:27ff:feb0:6a94/64 Scope:Link
         RX packets:1556 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1502 errors:0 dropped:0 overruns:0 carrier:0

Next, let’s keep trying to focus on the RX packets:
ifconfig | grep eth1 -A5 | grep RX  
[root@localhost tmp]# ifconfig | grep eth1 -A5 | grep RX
         RX packets:1572 errors:0 dropped:0 overruns:0 frame:0
Now it is down to parsing this line for the number of packets, say 1572...

The awk command can be used to split a line into columns where spaces separate the fields.
Therefore, if each space in the line above represents a column we can focus on column two:

ifconfig | grep eth1 -A5 | grep RX | awk ' {  print $2 } '

Great, now how can we strip out the “packets:” from the raw number we are looking for?
We already know that awk is good for selecting columns from text data. As the string is now, it can also be interpreted in column format by switching the delimiter from a space to colon. Therefore, selecting column 2 using a ":" delimiter with awk will finally give us the number
[root@localhost tmp]# ifconfig | grep eth1 -A5 | grep RX | awk ' {  print $2 } ' | awk -F ":" ' { print $2 } '

That's it!

What can be done with the number of packets? If you take a sample before after, say for 60 seconds one can derive a rough estimate of packets per second on the interface.

START=`ifconfig | grep eth1 -A5 | grep RX | awk ' {  print $2 } ' | awk -F ":" ' { print $2 } '`;
sleep $LENGTH
END=`ifconfig | grep eth1 -A5 | grep RX | awk ' {  print $2 } ' | awk -F ":" ' { print $2 } '`;
echo $START $END $LENGTH | awk ' { print ($2-$1)/$3 } '

Again the awk command in the last line is used to parse columns but additionally it can be used for doing simple arithmetic with the columns. For instance, packets/sec = ($END-$START)/$LENGTH

The thing to remember about this example is that this is not difficult. The solution only uses simple features of awk and grep.

This exercise can also be done without using ifconfig. As an exercise on your own try using
only the file /proc/net/dev and grep/awk commands. The same metric, RX packets, can be found here:
Inter-|   Receive                                                |  Transmit
face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
 eth1:  359967    4110    0    0    0     0          0         1   786631    3985    0    0    0     0       0          0
 eth2:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
   lo:       0    

Thursday, April 9, 2015

Linux Tips - Beyond the Basics (Part 1)

As a lot of recent posts have focused on Windows tips, a coworker of mine suggested writing up some Unix tips. Many day to day operations in Unix can be performed using the same old techniques and a lot of Unix users become stagnant and do not progress to help improve Unix skills beyond the basics. Here are everyday tips I still find useful from my days as a Unix admin.

There is a lot that can be covered, at this point it looks like this will be a multi-part series of articles. Feel free to comment with any suggested topics.

Simple Bash Navigation
There is no excuse for fumbling around on the command line. This is your primary interface to Unix and Hyperion. However, this all breaks down when your boss is looking over your shoulder and you start fumbling around like a novice.

(emacs key bindings)
CTRL +a beginning of line,
CTRL+ e end of line
CTRL + k - cut from current position to end of the line
CTRL + y - yank back the cut contents at the current cursor position
Learn other cool tips like swapping the position of command line arguments:

Ctrl+r at the Bash Shell
This is the recall command; the most useful feature in the bash shell. Maneuvering in the Unix shell often requires complex commands and long paths. Most users are familiar with the history command and the up and down arrows to access recent history. However, this is inefficient. At the command prompt press CTRL+r and type any unique part of the string containing the command that you want to recall. This powerful feature lets you pull up anything from the history by typing in a few unique characters. You may have used this before. Web browsers have copied this feature in the URL prompt. For instance, typing /em in Chrome can pull back "https://server:7002/em". After some time using this feature your brain starts to rewire and it becomes a snap to use "recall" mode to rapidly pull up commands to run or edit.

If there is more than one match, pressing CTRL+r again will go to a prior match the command history. CTRL + p and CTRL + r also cycle back and forth between matches.

Advanced users will find that the more commands you type in with full paths, the more the recall feature helps. For instance, if you run a command by changing into 6 directories with individual cd commands and then run the command with a relative path, the recall feature won't help piece this together. It is encouraged to enter the full paths to the command so it is easily run from any location at a different time.

Become Very Familiar with your Editor of Choice
The text editor is the most powerful tool on the Linux box. Vi or EMACS or Other? Either way - learn more than the basics, these are powerful editors can can save tons of time.



The Unix command prompt is a fragile thing when running over a network connection via SSH. Ever run a command, have it take longer than expected and now it's time to go home? Logging out will surely kill the long running process. This is very useful when running database backups that will take a while, yet the default timeout for your bash shell is only 20 minutes. This prevents the backup terminating when your shell is killed.

Screen basics:
Before running long running command type "screen" to enter screen session.

screen -S ScreenTest

sleep 1000000

Now detach your session
CTRL+a d

[detached from 11504.ScreenTest]

Log out and go home for the day.

When you get home, check what's running:
:~$ screen -ls
There are screens on:
        11504.ScreenTest        (04/08/2015 08:44:26 PM)        (Detached)
1 Socket in /var/run/screen

reattach to the running screen session:
screen -r ScreenTest
sleep 1000000 (sleep command is still sitting there executing)

kill all processes matching the word "java"

killall java
ps -wwef | grep java | awk ' { print $2 } ' | xargs kill -9

Swiftly kill everything. If you have a service account such as "hyperion" and you need to quickly make sure all processes are down, assuming you have tried all sane ways to do this, you can kill everything running under the current user id by:
kill -9 -1
Tip: Don't do this as root.

Network Testing
The old "telnet" command to test if a service is running on a port is being deprecated. The nc command, "netcat" command can be used for basic connectivity.

:~$ nc -v localhost 80
nc: connect to localhost port 80 (tcp) failed: Connection refused

Also nc can be used to startup a mock network server on a port for testing connectivity between two machines. This allows you to test firewall rules long before you have setup the environment and realize the connectivity is not working.

On server1:
nc -l 1234    (start a server on port 1234)

from another machine, try to connect:
nc -v server1 1234
typing text on the client side should appear on the server side.

Copying Files Between Servers

rsync -a ~/tmp/dir1/ oracle@epmvirt:/tmp/destination

Bash For Loops
for i in a b c; do  echo $i ; done

for server in server1 server2 server3 server4 ; do scp file $server:/tmp/; done

Disk Space Consumption
du -m /u0 | sort -n

36643   /u0/app/oracle/product
68393   /u0/app/oracle/backups
320977  /u0/app/oracle/arch
576131  /u0/app/oracle
576230  /u0/app
605054  /u0/

This shows where to find the largest disk space consumers to quickly mitigate disk space issues. In this case we could check the archive logs directory and reduce disk space.

Learning a Programming Language
Often users get sucked into writing complex shell scripts. They start off innocent enough, but quickly shell scripting will show its weaknesses. Shell scripts become unwieldy with syntax and flow control - loops all over the place, difficult command line argument handling, poor error handling... etc. I encourage having a more elegant language to write scripts in. Any will do, but python can be a quick way to get started and is proven in the real world. Even if you completely new, spending a few hours on a tutorial site and then tackling a simple program in python will pay off in the long run over using a bash script.
A tutorial can be found here:

Monday, April 6, 2015

Auto Restarting Windows Services

I wanted to discuss a very simple task to help with keeping services running. In most cases if you have some service running and it unexpectedly crashes, you want it to automatically come back up. This holds true whether you are an EPM pro or just starting out.

In an earlier post, I talked about a more complex event like running an external script upon failure.
The external script solution can be good for very specific cases, but it takes a bit of work to setup. To expand upon this specific case, it is a good idea to have every Hyperion service to auto restart upon failure. This is much simpler than setting up custom scripts. It is a simple property in the Windows service settings.

The settings are located in Windows Services, under the service properties there is a Recovery tab. Here you can set the action to "Restart the Service"

An easy way to script this is to get the Service names for all Hyperion components and write up something like this:
sc \\localhost failure HyS9FRReports actions= restart/60000/restart/60000/""/60000 reset= 0
Localhost can be replaced by the server name.

Some additional tips:
  • Don't infinity restart the service by setting the "subsequent failures" box. At some point you'll want to have it crash and stay down rather than flap up and down constantly.
  • This restart action is not intended to solve the root cause of problems. You should be detecting when the services crash to take additional action on the root cause.
  • This customization is overwritten whenever you run the Hyperion Config Tool. Config Tool resets the service properties and the custom settings would need reapplied. 

Sunday, March 29, 2015

Handling WebLogic Failures

The default behavior when a critical error occurs in a running WebLogic application is to do nothing about the failure condition. This can cause the web server, OHS, to keep routing traffic to the service, even in a highly available situation where there is more than one server available. This situation leaves many with HA setups vulnerable to a single point of failure.

The basic problem is that when you have a WebLogic service running, such as HFM, and it encounters a critical situation such as out of memory (OOM) error, the service comes to a halt and cannot process users requests. End users are left with their requests spinning in the web browser until they eventually timeout. It is important to configure the WebLogic Managed Server to acknowledge the failure and shut itself down quickly so that the other member of the cluster can take over.

Two settings in WebLogic can help with this, under the theme Overload protection. The settings are Failure Action and Panic Action.


This can be scripted to set for all managed servers using WLST
servers = cmo.getServers()
for server in servers:
               cd('OverloadProtection/' + name)

More Details about overload protection can be found here:

Taking this scenario one step further, rather than just failing, Node Manager can be used automatically restart the failed service. However, Node Manager is most often used as an optional component, so you would need to decide whether to implement Node Manager to use these features.

Wednesday, March 25, 2015

Useful Administrative Tools in Windows

Do you work with Hyperion in Windows and feel like your hands are tied? Like a fish out of water? OK, maybe this is an exaggeration for most admins, but as a former Linux sysadmin it is how I feel most days. I wanted to share some tools that can be used to help bridge the gap and become a Windows power user as it relates to Hyperion. Please feel free to share your own tools and ideas in the comments.

Note: Each of these tools has its own licencing agreement. Please carefully read and adhere to each licence. 


Unix equivalent: grep
Find strings in files. AstroGrep is much more powerful than windows find utility. The Windows find utility only searches for whole words. For example, "this is an exception" vs. "java.lang.exception". The Windows find utility only finds the first example, yet AstroGrep finds both strings.

If Hyperion throws a weird message or error, you can use AstroGrep to search the file system for the file to help pinpoint where it is coming from and get more context around the error. It can be useful to find specific server names, passwords, usernames, etc in files such as MAXL scripts.

In the example below I am searching the *.log files in the EPM Diagnostics folder for the string "Exception".


Unix equivalent: diff

Diff files, or recursively entire directories for changes. For instance, compare the EPMA accounts from an LCM export taken at two different times to identify the differences in EPMA metadata.


Unix equivalent: tail -f
Windows error log tailer. It is used to monitor logs in real time. Simply drag and drop a file from windows explorer into mTAIL, click the start button and it will start displaying updates to the log file in real time. This is handy when you need to check a file over and over for updates, such as looking for new errors. It is also useful if you are constantly restarting WebLogic to check log for errors during startup.

Process Explorer and Handle

Unix equivalent: lsof
Ever try to rename or delete that certain folder and it keeps saying it is in use, but you do not see anything open? Ever have trouble with Hyperion patches failing to apply because some files in use? Obviously, the first step is to search task manager for Hyperion related processes. However, there are some pesky situations where this alone cannot identify the culprit. Process Explorer has an option to search file handles, and is easier to use. The Handle tool is a command line tool and kind of archaic, but can also find what is using the files you want to get access to.


Unix equivalent: du -k | sort -n
Out of disk space? Need a quick tool to discover the where the most space is being consumed? WinDirStat gives you an ordered list of disk space consumption and lets you drill through, finding what is using all the disk space on your system.


  Sadly, the opensource form of this tool is deprecated. It it a very useful remote desktop/SSH session manager for Windows. Integrates nicely with PuTTY.

Process Monitor

Unix equivalent: strace
A very low level tool to see the system calls a process is making while executing. Typically this tool comes out when all of the normal avenues have been tried to solve the problem. Sometimes getting down to the level of system calls can pinpoint things like missing files, bad security, and bugs. This helps figure out why a process is crashing by seeing what unexpected conditions the process encountered while running.

Internet Explorer Developer tools

Similar tools also available in Chrome and Firefox

Network profiling can help find latency in your requests. For instance, you can measure the time it takes to get resources from different geographic regions and pinpoint certain latency issues. From the example below, there are a ton of images and background activity after each click in Hyperion. Sometimes slow performance can be found on the front-end using this method. It is also useful to inspect in detail certain processes for debugging purposes.

Simply go to Tools -> F12 Developer Tools. Select the Network tab, and "start capturing".

Monday, February 23, 2015

HFM Architecture

There seems to be great deal of changes in the HFM release for Linux. I have reviewed the typical Oracle docs for info and did not find anything detailing the architecture. I figure it is worth fleshing out what I have learned. Keep in mind that I do not have any special internal knowledge so this is all just speculation based on observation.

The main tier which the user interacts with is the familiar WebLogic ADF application. This is pretty familiar to prior versions so I will gloss over this.

Moving into the application layer, the primary interface is the Java based HsxServer application. It is involved when you use LCM, create/add/delete or interact with HFM through Workspace. In Oracle terms, this is also referred to as JHsxServer (Java based). The main Java application class is oracle.epm.fm.hsxserver.HsxServer. The server does not perform any of the actual calculations. It spawns another process to act as the main calculation engine.

The HsxServer process kicks off a separate process for each HFM application, called xfmdatasource. The xfmdatasource is written in C and seems to have been ported to Unix via some abstraction layers. There is still a very Windows feel to the process, as you will see.

To start let's begin by investigating xfmdatasource.
Start with a ps listing:
 6483 oracle    20   0 1501m  54m  32m S 14.3  1.1   0:00.58 xfmdatasource.exe

Using the pid we can get the command line and environment variables used to start the process.

[oracle@EPMVirt ~]$ cat /proc/6483/cmdline

One thing to note here is the various ports xfmdatasource uses. Also useful is the logging location. To investigate errors during startup you could look in this folder. However this only works if the process starts up enough to initialize the logger.

Taking a look at the environment variables one can see some hints at ODBC and something called MW... 
xargs -n 1 -0 < /proc/6483/environ

What is the MW? MainWin...

Poking around in the MWHOME folder you can see some utilities to get a better idea. 
 reg - an mwadm command, administers the MainWin System Core (MSC)
         registry services.

Usage: mwadm command [parameters]
Following is the list of commands supported by the mwadm utility:

version displays the version number of the MSC.
help            displays help on using mwadm and its commands.
stop            stops all MSC services in an orderly manner.
start           manually starts all MSC services.
status          provides vital signs of the MSC services.
log             administers the MSC logs.
sec             administers MSC DCOM security settings.
reg             administers the MSC registry service.

export MWREGISTRY=/home/oracle/.mw/hklm_amd64_linux.bin

[oracle@EPMVirt bin]$ ./mwadm status
Run-time mode       : professional
Core Services status: running
Core Data Directory : /home/oracle/.mw/core_data/EPMVirt
RPC Port            : dynamic
[oracle@EPMVirt bin]$

Here are the unix processes involved:
[oracle@EPMVirt bin]$ ./mwadm reg -ps
uid:    500     pid:   6553     hkcu: /home/oracle/.mw/core_data/EPMVirt/.mw/hkcu.bin
uid:    500     pid:   6483     hkcu: /home/oracle/.mw/hkcu.bin
[oracle@EPMVirt bin]$

The next question to answer is how is MainWin used by xfmdatasource?

Further investigation uncovers a file, hfm_post_install_steps.sh

Towards the end of the script we see some references to MWHOME

echo "Sourcing the mainwin runtime environment required for registration ..."
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EPM_ORACLE_HOME/products/FinancialManagement/Server
export MWHOME=$EPM_ORACLE_HOME/products/FinancialManagement/Server/mw
source $MWHOME/setmwruntime

echo "Registering COM objects ...."
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhfmcassecurity.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsxdssecurity.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsxsecurity.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsvsystemactivity.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsvsharedappdata.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsxsystemconfig.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsvscriptengine.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsvdsmemberlistscript.so
regsvr32 -c -nodisplay $EPM_ORACLE_HOME/products/FinancialManagement/Server/libhsvdsrulesom.so

What is going on here? Essentially after HFM is installed it registers COM objects using regsvr32. What is regsvr32 and what's it doing on Linux? This is part of the MainWin installation 
located in /u0/Oracle/Middleware/EPMSystem11R1/products/FinancialManagement/Server/mw/bin-amd64_linux_optimized/regsvr32

Finally, this reveals the purpose of MainWin - to house the simulated Windows registry components for COM objects required by HFM xfmdatasource process.

Next, let's shift away from MainWin for a bit and focus on the dependencies for xfmdatasource.

the xfmdatasource.exe process has the boost libraries as a prerequisite:
[oracle@EPMVirt ~]$ ldd /u0/Oracle/Middleware/EPMSystem11R1/products/FinancialManagement/Server/xfmdatasource.exe | grep boost
[oracle@EPMVirt ~]$

What is libboost?

Boost is essentially an open source set of C libraries focusing on being portable between different platforms. HFM uses boost to be more platform independent. Specifically, it looks like Boost is used in a lot of low level calls like threading, file system access. Presumably, code written using Boost would work on Linux and Windows, providing portability without having to rewrite code.

One last thing we discover looking at the xfmdatasource environment is the ODBC datasource dependency. Now let's investigate the ODBC piece. We are familiar with ODBC from Essbase drill through. However, this is the first it is being used with HFM. 

The main configuration takes place in a file called odbc.ini

[oracle@EPMVirt Oracle]$ cat ./Middleware/EPMSystem11R1/common/ODBC-64/Merant/7.1/odbc.ini
[ODBC Data Sources]
Oracle Wire Protocol=DataDirect 7.1 Oracle Wire Protocol


[Oracle Wire Protocol]
Description=DataDirect 7.1 Oracle Wire Protocol
[oracle@EPMVirt Oracle]$

The Hyperion configuration actually sets up a new TNS connection named HFMTNS located at /u0/Oracle/Middleware/user_projects/config/dbclient/tnsnames.ora. 

export TNS_ADMIN=/u0/Oracle/Middleware/user_projects/config/dbclient

[oracle@EPMVirt Oracle]$ tnsping HFMTNS

TNS Ping Utility for Linux: Version - Production on 23-FEB-2015 01:52:34

Copyright (c) 1997, 2013, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
OK (0 msec)
[oracle@EPMVirt Oracle]$

Sometimes it may be necessary to test the ODBC connection to ensure connectivity and help with troubleshooting. You can use the unix utility isql for basic connectivity.

First, setup the utility:
yum install -y unixODBC
ln -fs /u0/Oracle/Middleware/EPMSystem11R1/common/ODBC-64/Merant/7.1/odbc.ini ~/.odbc.ini
export LD_LIBRARY_PATH=/u0/Oracle/Middleware/EPMSystem11R1/common/ODBC-64/Merant/7.1/lib/

Next, try to connect:
[oracle@EPMVirt ~]$ isql -v "Oracle Wire Protocol" EPM_HSS PWD
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |

Trying apparently one step to far! Not able to perform any queries due to licencing restrictions. 
SQL> select max(starttime) from SMA_AUDIT_FACT;
[DataDirect][ODBC Oracle Wire Protocol driver]The DataDirect product you are attempting to access has been provided to you by Oracle for exclusive use with Oracle products.  You are not licensed to use this product with any application other than Oracle products.   If you would like to purchase this product for some other use, please call 800-876-3101 or visit DataDirect at www.datadirect.com
| MAX(STARTTIME)            |
|                           |
SQLRowCount returns -1
1 rows fetched


In conclusion, the post describes how HsxServer depends on xfmdatasource.exe. Some of the dependencies of xfmdatasource are Boost, MWHOME, and ODBC. Knowing more about these dependencies can greatly help in understanding the architecture.

Saturday, February 21, 2015

EPMVirt now with HFM

It has taken a bit longer than expected, but I am now releasing EPMVirt which includes HFM The two main issues were getting the correct Linux boost libraries and some proper tuning for such a small sized environment. The default HFM tuning parameters are setup for a 16G system, which is far too much for this tiny environment.