Which version of SC or SM are you using? We faced the same problem many times in our productive env when we were using SM 7.01. Then we raised a case to HP after which they suggested us to upgrade to 7.02, since then, we have hardly encountered this problem in 7.02, with some modifications as suggested by HP.
As told by HP: :-) "The following parm is based on the assumption that the customer has no other JVMOptions in the sm.ini. Assuming that they had only Option0, Option1, Option2, the next Option they can include in the sm.ini is: JVMOption3:-XX:+HeapDumpOnOutOfMemoryError This will generate a JavaHeap dump in the form of a java_pid.hprof file the next time they encounter a java.lang.OutofMemory error. Please have the customer add the above option to the sm.ini files and recycle the application. Once they encounter the out of memory error, an hprof file will be generated and we will be able to analyze the hprof file to determine root cause."
So we added this parameter in sm.ini file and sent the hprof file to HP for the analysis. They found out that one of the background scheduler was consuming huge % of JVM heap. So we tried to start this scheduler in sm.cfg file with this parameter: sm -bg scheduler Also we have have increased the memory of our servers. For reference, you may read SCR 42801 in 7.11 realease notes.