Windows WebLogic Servies Fail to Start - Trouble Encrypting boot.properties

I have run across this error a few times and have finally gotten around to documenting it. The issue occurs typically after a multi node Hyperion installation and all the WebLogic managed servers not residing on the WebLogic domain fail to start up. Essentially, you start the WebLogic service, and after a minute it shuts down. Looking into the WebLogic managed server log files: 

java.lang.AssertionError: java.lang.reflect.InvocationTargetException

at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:175)

at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)

at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)

at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1092)

at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)

at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)

at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)

at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)

at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)

at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1109)

at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)

at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:296)

at weblogic.server.channels.ChannelService.start(ChannelService.java:258)

at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Caused By: weblogic.security.internal.encryption.EncryptionServiceException

at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:139)

at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:187)

at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:96)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)

at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)

at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)

at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1092)

at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)

at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)

at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)

at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:63)

at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)

at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1109)

at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:600)

at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:296)

at weblogic.server.channels.ChannelService.start(ChannelService.java:258)

at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

WebLogic is trying to read the boot.properties file in order to get the credentials to startup the managed server. However, it appears it needs to contact the WebLogic Domain in order to properly encrypt the boot.properties. You would also see the failed connection to the WebLogic Domain in the logs:

<Aug 2, 2014 6:52:04 AM UTC> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at http://wldomain:7001.>

<Aug 2, 2014 6:52:04 AM UTC> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>

Having a look at the boot.properties file it looks like the password has not been encrypted, hence the EncryptionServiceException. It is trying to read what it thinks is the encrypted password but the password is plain text, causing the exception. 

C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\RaFramework0\security\boot.properties

#

#Fri Aug 01 05:06:31 UTC 2014

password=epm_admin

username=mypassword

So the question is how to get the Managed Server to properly contact the WebLogic domain to encrypt the boot.properties file. 

I usually do this by starting it from the command line, 

cd C:\Oracle\Middleware\user_projects\domains\EPMSystem\bin
startManagedWeblogic.cmd RaFramework1 http://wldomain:7001

Note, the second argument is the WebLogic Domain URL. 

While starting up, assuming the Managed Server is able to connect to the domain you will see:

<Aug 2, 2014 6:57:37 AM UTC> <Notice> <Security> <BEA-090083> <Storing boot identity in the file: C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\RaFramework1\security\boot.properties>

This is a good indication that the problem has been resolved. Looking in the boot.properties you should see the encrypted password.

This approach is what I find best for debugging. Additionally, the startup parameters for starting the WebLogic managed server as a Windows service are in the Windows Registry. Specifically for Hyperion there is a JVMOption for setting the WL Domain server. Make sure that this is set correctly when starting as a Windows service. "-Dweblogic.management.server=http://wldomain:7001"