Monday, April 28, 2008

JAVA Mailer not Starting

Today was the bad day for us, we had an outage last weekend and after we finished our job lot of things start failing. Due to DBA errors and Java mailer is one off them. Here is what log says
------------------------------------------------------------------------------------
SVC-GSM-WFMLRSVC-67585-10006 : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.oracle.apps.fnd.cp.gsc.SvcComponentContainerException: Could not start component; performing rollback -> oracle.apps.fnd.cp.gsc.SvcComponentException: Validation failed for the following parameters -> {FRAMEWORK_USER=Unable to make a connection with the user and responsibility}. Parameters were -> {INBOUND_MAX_IGNORE_SIZE=1000, INBOUND_MAX_LOOKUP_CACHE_SIZE=100, ATTACH_IMAGES=N, ALLOW_FORWARDED_RESPONSE=Y, INBOUND_UNSOLICITED_THRESHOLD=2, NODENAME=PRODMAIL, SEND_UNSOLICITED_WARNING=N, AUTOCLOSE_FYI=N, PROCESSOR_READ_TIMEOUT_CLOSE=Y, INBOUND_PROTOCOL=IMAP, ATTACHED_URLS=WFMAIL:ATTACHED_URLS, TEST_ADDRESS=, EXPUNGE_ON_CLOSE=Y, PROCESSOR_OUT_THREAD_COUNT=1, ATTACH_STYLESHEET=N, OUTBOUND_CONNECTION_TIMEOUT=120, INBOUND_MAX_RET_EMAIL_SIZE=100, MAX_INVALID_ADDR_LIST_SIZE=100, PROCESSOR_MAX_LOOP_SLEEP=60, SEND_ACCESS_KEY=N, PROCESSOR_IN_THREAD_COUNT=1, FRAMEWORK_APP=1, CLOSED=WFMAIL:CLOSED, INBOUND_FETCH_SIZE=100, SUMMARY=WFMAIL:SUMMARY, ENABLE_STYLESHEET=N, PROCESSOR_ERROR_LOOP_SLEEP=60, OPEN_MAIL_FYI=WFMAIL:OPEN_MAIL_FYI, FRAMEWORK_RESP=20420, ALTERNATE_EMAIL_PARSER=oracle.apps.fnd.wf.mailer.DirectEmailParser, INBOUND_PASSWORD=_@8@!864^4$9#64#^@#8@@686##!#44@#!0!99@@`9+^+*#B$$@99*$, HTTP_USER_AGENT=Mozilla/4.76, RESET_NLS=N, PROCESS=Process, OPEN_MAIL_DIRECT=WFMAIL:OPEN_MAIL_DIRECT, PROCESSOR_LOOP_SLEEP=5, MESSAGE_FORMATTER=oracle.apps.fnd.wf.mailer.NotificationFormatter, REPLYTO=emailuser@edomain.com, FRAMEWORK_USER=0, CANCELED=WFMAIL:CANCELED, OUTBOUND_PROTOCOL=SMTP, ACCOUNT=emailuser, DISCARD=Discard, FROM=Expedite, PROCESSOR_DEFER_EVTDATA_READ=Y, SEND_CANCELED_EMAIL=N, WARNING=WFMAIL:WARNING, PROCESSOR_MAX_ERROR_COUNT=50, INBOUND_CONNECTION_TIMEOUT=120, HTMLAGENT=http://hostname.domainname.com:8000/pls/SID, INBOX=INBOX, OPEN_INVALID_MORE_INFO=WFMAIL:OPEN_INVALID_MORE_INFO, OUTBOUND_SSL_ENABLED=N, INBOUND_SERVER=mail.domain.com, OPEN_MORE_INFO=WFMAIL:OPEN_MORE_INFO, MAILER_SSL_TRUSTSTORE=NONE, INLINE_ATTACHMENT=N, OPEN_INVALID=WFMAIL:OPEN_INVALID, EMAIL_PARSER=oracle.apps.fnd.wf.mailer.TemplatedEmailParser, DIRECT_RESPONSE=N, OPEN_MAIL=WFMAIL:OPEN_MAIL, FRAMEWORK_URL_TIMEOUT=30, COMPONENT_LOG_LEVEL=1, SUMHTML=WFMAIL:SUMHTML, PROCESSOR_READ_TIMEOUT=10, DEBUG_MAIL_SESSION=N, INBOUND_SSL_ENABLED=N, OUTBOUND_SERVER=mailhost.domain.com}
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.handleComponentEvent(SvcComponentContainer.java:2211)
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(SvcComponentContainer.java:300)
at oracle.apps.fnd.wf.bes.DispatchThread.run(DispatchThread.java:57)
----------------------------------------------------
And I think I know the issue, it was because of autoconfig only, you might be asking why? Our DBA ran autconfig without clearing FND_Nodes table. And as we were running on virtual host (clustering at db tier). It created a mess there. We have duplicate server_id for virtual and active physical host. Server ID for both of them were same, that's the reason Java mailer was not able to start and failing because of above error. Now what, another downtime---- I don't think customer will agree, but this issue is critical how to resolve this without big downtimes for running autoconfig. We raised Sev 1 tar to see if Oracle Support can help on this, but hard luck, they said the same old story of running autoconfig and all. But after spending few minutes, I got the solutions with 10 min of downtime. Here is what I did :

1. Shutdown Concurrent Manager
2. Take a backup of FND_Nodes.
3. Delete Physical host from OAM (OAM > Administrator > Hosts)
4. Register same host again from OAM.
5. Start Concurrent Manager
6. Mailer was working fine after this.

Happy Troubleshooting !!!

3 comments:

Anonymous said...

Hi i have the exact same issue, but we are not using virtual hostname.But server_id of db tier and conc tier is same and mailer is not comming up.

How can i delete hosts if i am not using virtual hostname. there is no option in OAM. Please mail me at sumit.davre@gmail.com.

Thanks in Advance

Sundeep Dubey said...

In your case you need to re run autoconfig, but prior to that run this command as apps user.

EXEC FND_CONC_CLONE.SETUP_CLEAN;
Commit;

Thanks
Sundeep

Anonymous said...

i seriously enjoy all your writing taste, very exciting,
don't quit and keep writing for the reason that it simply nicely to look through it,
excited to view even more of your current writing, have a good day!