Thread: please help to analys Javacore analys

Hello,

This is a small part from javacore (we are using ibm websphere):

2XMFULLTHDDUMP Full thread dump J9 VM (J2RE 6.0 IBM J9 2.4 Linux x86-32 build jvmxi3260-20090215_2988320090215_029883_lHdSMr, native threads):
3XMTHREADINFO "P=656666:O=0:CT" TID:0x092B1700, j9thread_t:0x0929599C, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E80, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Thread.sleep(Native Method)
4XESTACKTRACE at java/lang/Thread.sleep(Thread.java:850)
4XESTACKTRACE at com/ibm/ws/runtime/WsServerImpl.main(WsServerImpl.java:679)
4XESTACKTRACE at com/ibm/ws/runtime/WsServer.main(WsServer.java:59)
4XESTACKTRACE at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccess orImpl.java:39(Compiled Code))
4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMeth odAccessorImpl.java:37(Compiled Code))
4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:599(Compiled Code))
4XESTACKTRACE at com/ibm/wsspi/bootstrap/WSLauncher.launchMain(WSLauncher.java:213)
4XESTACKTRACE at com/ibm/wsspi/bootstrap/WSLauncher.main(WSLauncher.java:93)
4XESTACKTRACE at com/ibm/wsspi/bootstrap/WSLauncher.run(WSLauncher.java:74)
4XESTACKTRACE at org/eclipse/core/internal/runtime/PlatformActivator$1.run(PlatformActivator.java:78)
4XESTACKTRACE at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.runApplication(EclipseAppLaunch er.java:92)
4XESTACKTRACE at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.start(EclipseAppLauncher.java:6 8)
4XESTACKTRACE at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:400)
4XESTACKTRACE at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:177)
4XESTACKTRACE at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccess orImpl.java:39)
4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMeth odAccessorImpl.java:37)
4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:599)
4XESTACKTRACE at org/eclipse/core/launcher/Main.invokeFramework(Main.java:340)
4XESTACKTRACE at org/eclipse/core/launcher/Main.basicRun(Main.java:282)
4XESTACKTRACE at org/eclipse/core/launcher/Main.run(Main.java:981)
4XESTACKTRACE at com/ibm/wsspi/bootstrap/WSPreLauncher.launchEclipse(WSPreLauncher.java:330 )
4XESTACKTRACE at com/ibm/wsspi/bootstrap/WSPreLauncher.main(WSPreLauncher.java:108)
3XMTHREADINFO "JIT Compilation Thread" TID:0x092B1D00, j9thread_t:0x09295EC4, state:CW, prio=10
3XMTHREADINFO1 (native thread ID:0x7E82, native priority:0xB, native policy:UNKNOWN)
3XMTHREADINFO "Signal Dispatcher" TID:0x735C4600, j9thread_t:0x092963EC, state:R, prio=5
3XMTHREADINFO1 (native thread ID:0x7E85, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
4XESTACKTRACE at com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:54)
3XMTHREADINFO "Gc Slave Thread" TID:0x735C4C00, j9thread_t:0x735C8188, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E88, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x09383D00, j9thread_t:0x735C841C, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E89, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x09384300, j9thread_t:0x735C86B0, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E8A, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x093DF300, j9thread_t:0x735C8944, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E8B, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x093DF900, j9thread_t:0x735C8BD8, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E8C, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x093EA500, j9thread_t:0x735C8EFC, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E8D, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x093EAB00, j9thread_t:0x735C9190, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E8E, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Thread-5" TID:0x733F5100, j9thread_t:0x735C9424, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E91, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:196(Compiled Code))
4XESTACKTRACE at java/util/Timer$TimerImpl.run(Timer.java:199(Compiled Code))
3XMTHREADINFO "Framework Event Dispatcher" TID:0x733F5700, j9thread_t:0x735C96B8, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E92, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:167)
4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/EventManager$EventThread.getNextEvent(EventManager .java:349)
4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/EventManager$EventThread.run(EventManager.java:287 )
3XMTHREADINFO "Start Level Event Dispatcher" TID:0x72E10100, j9thread_t:0x735C994C, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E93, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:167)
4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/EventManager$EventThread.getNextEvent(EventManager .java:349)
4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/EventManager$EventThread.run(EventManager.java:287 )
3XMTHREADINFO "Worker-0" TID:0x72E10700, j9thread_t:0x735C9BE0, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E94, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:196(Compiled Code))
4XESTACKTRACE at org/eclipse/core/internal/jobs/WorkerPool.sleep(WorkerPool.java:181(Compiled Code))
4XESTACKTRACE at org/eclipse/core/internal/jobs/WorkerPool.startJob(WorkerPool.java:218(Compiled Code))
4XESTACKTRACE at org/eclipse/core/internal/jobs/Worker.run(Worker.java:51)
3XMTHREADINFO "Approximate Time Keeping Thread" TID:0x72B95B00, j9thread_t:0x72B92E44, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E97, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Thread.sleep(Native Method)
4XESTACKTRACE at java/lang/Thread.sleep(Thread.java:850(Compiled Code))
4XESTACKTRACE at com/ibm/ws/timeutils/QuickApproxTimeT$TimeThread.run(QuickApproxTimeT.j ava:129)
4XESTACKTRACE at java/lang/Thread.run(Thread.java:735)
3XMTHREADINFO "Deferred Alarm Manager" TID:0x728C8B00, j9thread_t:0x72B9336C, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x7E99, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at java/lang/Object.wait(Native Method)

Here is screenshot from analysing tool from IBM:


server is hung with this errors in logs:

0000001b ThreadMonitor W WSVR0605W: Thread "WebContainer : 34" (00000044) has been active for 668273 milliseconds and may be hung. There is/are 14 thread(s) in total in the server that may be hung.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:196)
at java.lang.Thread.join(Thread.java:615)
at spinEngine.ServiceRunner.join(ServiceRunner.java:8 6)
at spinEngine.ServiceRunner.run(ServiceRunner.java:44 )
at spinEngine.Engine.process(Engine.java:371)

Thanks, let me know if you have any tips.

Have you tried a dump analyser, e.g. IBM's ?

Yes, screenshot above is from dump analyser.