In every production environment there is a very complicated kind of issues to solve: the intermittent ones.
The kind of errors that happens “sometime” only: tricky to catch and reproduce.
But from last August, there is a tool that can help us to investigate errors like these
Executing from command line a very simple EAR file (trapit.ear) using Websphere Application client, you can define trigger events based on systemout messages.
Using a specific syntax you can generate javacores or activate traces or executing OS scripts depending on the messages are appearing on Websphere Application Server systemout file.
For example you can
sh> ./launchClient.sh /full/path/to/TrapIt.ear localhost:2809 EVENT=1,WSVR0605W JAVACORE=3,60
generate three javacores at 60 second intervals when the warning WSVR0605W shows up in the log.
Remember there will be always a small delay between the event and the action but this is a very interesting debug option for this kind of issues.