Using gdb to analyse core dumps in Linux
Once core file is generated.
Steps to analyze the core dump file-
Run “gdb /usr/java/jdk1.6.0_24/bin/java (java path) core.11151 (core file)”
You will get the gdb prompt
Type “bt” for stack trace.
To switch the thread, type “thread [thread no]”
To check the stack trace for selected thread again type “bt”
To get the stack of all the thread run “thread apply all bt full”
Find “LowMemoryDetector::low_memory_detector_thread_entry” in stack trace.
To find exactly in which thread it occured, type "where"
Example of "where" output:
where
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x00b0ddf0 in raise () from /lib/libc.so.6
#2 0x00b0f701 in abort () from /lib/libc.so.6
#3 0xf78e123f in os::abort(bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#4 0xf7a28431 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#5 0xf7a28fe1 in crash_handler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#6 <signal handler called>
#7 0xf756310b in oopDesc::size_given_klass(Klass*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#8 0xf78f8fc7 in ParallelScavengeHeap::block_start(void const*) const () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#9 0xf78de576 in os::print_location(outputStream*, int, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#10 0xf78e8d0c in os::print_context(outputStream*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#11 0xf7a27bd3 in VMError::report(outputStream*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#12 0xf7a28370 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#13 0xf78e81dc in JVM_handle_linux_signal () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#14 0xf78e4124 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#15 <signal handler called>
#16 0x00b4ef5f in _int_malloc () from /lib/libc.so.6
#17 0x00b50fb7 in malloc () from /lib/libc.so.6
#18 0x4af28af7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#19 0x4cb67075 in KCDdFieldJDisp::CboInvoke(long) ()
from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#20 0x4cb56132 in KCboJDisp::Invoke(JNIEnv_*, int, _jobjectArray*) ()
from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#21 0x4cb3db43 in Java_com_clarify_cbo_KJniDisp_invoke ()
from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#22 0xf447154c in ?? ()
#23 0x4c640118 in ?? ()
#24 0x4a9fe868 in ?? ()
#25 0x3fe6b0b8 in ?? ()
#26 0x00000000 in ?? ()
(gdb)
Example of "full" stack of all thread
thread apply all bt full
Thread 81 (Thread 26737):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c46a77 in pthread_join () from /lib/libpthread.so.0
No symbol table info available.
#2 0x0804dd78 in ContinueInNewThread ()
No symbol table info available.
#3 0x080497f6 in main ()
No symbol table info available.
Thread 80 (Thread 26738):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3 0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xf79b31bb in ObjectMonitor::wait(long long, bool, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xf79b0383 in ObjectSynchronizer::wait(Handle, long long, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xf774f9bc in JVM_MonitorWait () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7 0xf428e62a in ?? ()
No symbol table info available.
#8 0x09db2918 in ?? ()
No symbol table info available.
#9 0xf731eff0 in ?? ()
No symbol table info available.
#10 0x00000000 in ?? ()
No symbol table info available.
Thread 79 (Thread 26739):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3 0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#7 0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8 0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9 0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
Thread 78 (Thread 26740):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3 0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7 0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8 0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9 0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
...
...
..
Steps to analyze the core dump file-
Run “gdb /usr/java/jdk1.6.0_24/bin/java (java path) core.11151 (core file)”
You will get the gdb prompt
Type “bt” for stack trace.
To switch the thread, type “thread [thread no]”
To check the stack trace for selected thread again type “bt”
To get the stack of all the thread run “thread apply all bt full”
Find “LowMemoryDetector::low_memory_detector_thread_entry” in stack trace.
To find exactly in which thread it occured, type "where"
Example of "where" output:
where
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x00b0ddf0 in raise () from /lib/libc.so.6
#2 0x00b0f701 in abort () from /lib/libc.so.6
#3 0xf78e123f in os::abort(bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#4 0xf7a28431 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#5 0xf7a28fe1 in crash_handler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#6 <signal handler called>
#7 0xf756310b in oopDesc::size_given_klass(Klass*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#8 0xf78f8fc7 in ParallelScavengeHeap::block_start(void const*) const () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#9 0xf78de576 in os::print_location(outputStream*, int, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#10 0xf78e8d0c in os::print_context(outputStream*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#11 0xf7a27bd3 in VMError::report(outputStream*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#12 0xf7a28370 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#13 0xf78e81dc in JVM_handle_linux_signal () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#14 0xf78e4124 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#15 <signal handler called>
#16 0x00b4ef5f in _int_malloc () from /lib/libc.so.6
#17 0x00b50fb7 in malloc () from /lib/libc.so.6
#18 0x4af28af7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#19 0x4cb67075 in KCDdFieldJDisp::CboInvoke(long) ()
from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#20 0x4cb56132 in KCboJDisp::Invoke(JNIEnv_*, int, _jobjectArray*) ()
from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#21 0x4cb3db43 in Java_com_clarify_cbo_KJniDisp_invoke ()
from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#22 0xf447154c in ?? ()
#23 0x4c640118 in ?? ()
#24 0x4a9fe868 in ?? ()
#25 0x3fe6b0b8 in ?? ()
#26 0x00000000 in ?? ()
(gdb)
Example of "full" stack of all thread
thread apply all bt full
Thread 81 (Thread 26737):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c46a77 in pthread_join () from /lib/libpthread.so.0
No symbol table info available.
#2 0x0804dd78 in ContinueInNewThread ()
No symbol table info available.
#3 0x080497f6 in main ()
No symbol table info available.
Thread 80 (Thread 26738):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3 0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xf79b31bb in ObjectMonitor::wait(long long, bool, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xf79b0383 in ObjectSynchronizer::wait(Handle, long long, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xf774f9bc in JVM_MonitorWait () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7 0xf428e62a in ?? ()
No symbol table info available.
#8 0x09db2918 in ?? ()
No symbol table info available.
#9 0xf731eff0 in ?? ()
No symbol table info available.
#10 0x00000000 in ?? ()
No symbol table info available.
Thread 79 (Thread 26739):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3 0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#7 0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8 0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9 0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
Thread 78 (Thread 26740):
#0 0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3 0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7 0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8 0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9 0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
...
...
..
That was great to read. Thanks for sharing.
ReplyDeleteLinux course in Pune