Basic WebLogic Tuning in EPM

Part of the EPM Infrastucture Tuning Guide is a section on WebLogic tuning. The guide can be found here, https://blogs.oracle.com/pa/resource/Oracle_EPM_11_1_2_3_Tuning_Guide_v2.pdf 

These steps are recommended, and will be required because any time you create an SR with Oracle they will ask, "Did you apply the recommended tuning guidelines?" Responding no to this question usually results in an impasse and failure to complete the SR.

The basic WebLogic Tuning steps are:
1. Increase number of threads in connection pool
2. Tune connection backlog buffering
3. Stuck thread detection behavior tuning
4. Enable Native IO Performance pack.

Analyzing this a bit, steps 2 and 4 are already configured by default in 10.3.x. Consequently, they can be ignored.

The tuning guide shows how to somewhat tediously go and set the database connection pool settings and stuck threads using some screenshots. However, a scripted approach might be better to apply the settings in multiple environments, and ensure uniformity over time. A quick WLST script is below to do this. Keep in mind the tuning settings are just suggested values. For instance, stuck thread time is more an indicator of how long your longest acceptable web transaction might be, which is customer specific. In the script below EAS is configured with a different thread value than the rest of the managed servers, as an example.

Important Note: Once the connection pool is increased, you may also need to increase the number of available connections the database allows. For example, you may want to increase the number of processes in the Oracle Database.

File: TuneWL.py
connect(.....) 
edit() 
startEdit()

servers = cmo.getServers()
for server in servers:
  name=server.getName()
  print 'Tuning Cluster: '+ name
  cd('/Servers/' + name)   
  t_time = 1200
  if "Essbase" in name:
    t_time = 2400
    print "Setting EAS thread time to " + str(t_time)
  cmo.setStuckThreadMaxTime(t_time)
  cmo.setStuckThreadTimerInterval(t_time)


# JDBC connection pool

dsName = 'calc_datasource'cd('/JDBCSystemResources/'+dsName+'/JDBCResource/'+dsName+'/JDBCConnectionPoolParams/'+dsName)cmo.setMaxCapacity(30)

dsName = 'eas_datasource'cd('/JDBCSystemResources/'+dsName+'/JDBCResource/'+dsName+'/JDBCConnectionPoolParams/'+dsName)cmo.setMaxCapacity(30)

dsName = 'EPMSystemRegistry'cd('/JDBCSystemResources/'+dsName+'/JDBCResource/'+dsName+'/JDBCConnectionPoolParams/'+dsName)cmo.setMaxCapacity(150)

dsName = 'planning_datasource'cd('/JDBCSystemResources/'+dsName+'/JDBCResource/'+dsName+'/JDBCConnectionPoolParams/'+dsName)cmo.setMaxCapacity(150)

dsName = 'raframework_datasource'cd('/JDBCSystemResources/'+dsName+'/JDBCResource/'+dsName+'/JDBCConnectionPoolParams/'+dsName)cmo.setMaxCapacity(150)

save() 
activate() 
Output:

C:\Oracle\Middleware\user_projects\domains\EPMSystem\bin>setDomainEnv.cmd
C:\Oracle\Middleware\user_projects\domains\EPMSystem>java weblogic.WLST TuneWL.py


Starting an edit session ...
Started edit session, please be sure to save and actictivate changes once you are done.

Tuning Cluster: FoundationServices0
Tuning Cluster: EssbaseAdminServices0
Setting EAS thread time to 2400
Tuning Cluster: EpmaDataSync0

...

Windows service restart in action...

A while back I wrote a post on automatic service restarts using Windows:
http://epm-errors.blogspot.com/2015/04/auto-restarting-windows-services.html

I recently had the opportunity to watch one of the Windows automatic service restarts happen in a real production scenario. Complete with pagers firing, alerts going off, and, at the end of it, a completely automatic service restart. This type of event is somewhat hard to trigger for testing, so it was nice to see it work as expected.

The event looked like this in Windows Event Viewer:

By the time I got logged in and was looking into the log file I noticed WebLogic starting back up. Ultimately, no action was required. I never did find any error in the logs. However, there were some odd access log entries relating to a security scan and I think this ultimately caused EAS to crash.

Handling WebLogic Failures

The default behavior when a critical error occurs in a running WebLogic application is to do nothing about the failure condition. This can cause the web server, OHS, to keep routing traffic to the service, even in a highly available situation where there is more than one server available. This situation leaves many with HA setups vulnerable to a single point of failure.

The basic problem is that when you have a WebLogic service running, such as HFM, and it encounters a critical situation such as out of memory (OOM) error, the service comes to a halt and cannot process users requests. End users are left with their requests spinning in the web browser until they eventually timeout. It is important to configure the WebLogic Managed Server to acknowledge the failure and shut itself down quickly so that the other member of the cluster can take over.

Two settings in WebLogic can help with this, under the theme Overload protection. The settings are Failure Action and Panic Action.

  

This can be scripted to set for all managed servers using WLST

connect(....)  

edit() 

startEdit()

servers = cmo.getServers()

for server in servers:

  name=server.getName()

               cd('OverloadProtection/' + name)
      cmo.setPanicAction('system-exit')               
      cmo.setFailureAction('force-shutdown')
    save()
    activate()


More Details about overload protection can be found here:
http://www.dba-oracle.com/t_weblogic_overload_protection.htm

Taking this scenario one step further, rather than just failing, Node Manager can be used automatically restart the failed service. However, Node Manager is most often used as an optional component, so you would need to decide whether to implement Node Manager to use these features.