Using gdb to analyse core dumps in Linux

-
Once core file is generated.
Steps to analyze the core dump file-

Run “gdb /usr/java/jdk1.6.0_24/bin/java (java path) core.11151 (core file)”
You will get the gdb prompt
Type “bt” for stack trace.
To switch the thread, type “thread [thread no]”
To check the stack trace for selected thread again type “bt”
To get the stack of all the thread run “thread apply all bt full”
Find “LowMemoryDetector::low_memory_detector_thread_entry” in stack trace.
To find exactly in which thread it occured, type "where"

Example of "where" output:
where
#0  0xffffe410 in __kernel_vsyscall ()
#1  0x00b0ddf0 in raise () from /lib/libc.so.6
#2  0x00b0f701 in abort () from /lib/libc.so.6
#3  0xf78e123f in os::abort(bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#4  0xf7a28431 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#5  0xf7a28fe1 in crash_handler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#6  <signal handler called>
#7  0xf756310b in oopDesc::size_given_klass(Klass*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#8  0xf78f8fc7 in ParallelScavengeHeap::block_start(void const*) const () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#9  0xf78de576 in os::print_location(outputStream*, int, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#10 0xf78e8d0c in os::print_context(outputStream*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#11 0xf7a27bd3 in VMError::report(outputStream*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#12 0xf7a28370 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#13 0xf78e81dc in JVM_handle_linux_signal () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#14 0xf78e4124 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#15 <signal handler called>
#16 0x00b4ef5f in _int_malloc () from /lib/libc.so.6
#17 0x00b50fb7 in malloc () from /lib/libc.so.6
#18 0x4af28af7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#19 0x4cb67075 in KCDdFieldJDisp::CboInvoke(long) ()
   from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#20 0x4cb56132 in KCboJDisp::Invoke(JNIEnv_*, int, _jobjectArray*) ()
   from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#21 0x4cb3db43 in Java_com_clarify_cbo_KJniDisp_invoke ()
   from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#22 0xf447154c in ?? ()
#23 0x4c640118 in ?? ()
#24 0x4a9fe868 in ?? ()
#25 0x3fe6b0b8 in ?? ()
#26 0x00000000 in ?? ()
(gdb)

Example of "full" stack of all thread
thread apply all bt full
Thread 81 (Thread 26737):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c46a77 in pthread_join () from /lib/libpthread.so.0
No symbol table info available.
#2  0x0804dd78 in ContinueInNewThread ()
No symbol table info available.
#3  0x080497f6 in main ()
No symbol table info available.
Thread 80 (Thread 26738):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3  0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4  0xf79b31bb in ObjectMonitor::wait(long long, bool, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5  0xf79b0383 in ObjectSynchronizer::wait(Handle, long long, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6  0xf774f9bc in JVM_MonitorWait () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7  0xf428e62a in ?? ()
No symbol table info available.
#8  0x09db2918 in ?? ()
No symbol table info available.
#9  0xf731eff0 in ?? ()
No symbol table info available.
#10 0x00000000 in ?? ()
No symbol table info available.
Thread 79 (Thread 26739):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3  0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4  0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5  0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6  0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#7  0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8  0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9  0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
Thread 78 (Thread 26740):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3  0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4  0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5  0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6  0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7  0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8  0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9  0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
...
...
..

Comments

Post a Comment

Popular posts from this blog

Steps to Analyze AWR Report in Oracle

-
Image
Steps to Analyze AWR Report in Oracle AWR -   Automatic   workload repository is a collection of persistent   system performance   statistics   owned by SYS. It resides in SYSAUX   tablespace . By default snapshot are generated once every 60 min and maintained for 7 days . Each snapshot has a unique ID know as "snap_id". Snapshot detail can be found in " dba_hist_snapshot " view.   If we have Database performance issue and not the Database machine, then AWR Report is the place to look at. AWR is not used for real-time performance monitoring like the v$ tables. It is used for historical analysis of performance. AWR complements, but doesnot replace real-time monitoring. Once AWR Report is generated in Oracle, the next task is to analyze it. By going through the AWR Report we can easily solve issues like slow database, high wait events, slow query and many more issues. Even though the report is lengthy, Analyzing or Reading re...
Read more

Vmstat Output explained

-
vmstat : vmstat is a tool in Unix/Linux which is used to Report virtual memory statistics. It shows how much virtual memory there is, how much is free and paging activity. Most important, you can observe page-ins and page-outs as they happen.   vmstat   reports   information about processes, memory, paging, block IO, traps, and cpu activity.   > vmstat procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----   r   b    swpd    free    buff   cache         si    so     bi     bo    in    cs us sy id wa st   0   0   21688 12797396 850716 15372668     0     0      3     35     0     0   1   0 99   0   0   0   0   21688 12789292 850716 15372668 ...
Read more

Verifications and Error Handling in LoadRunner *Web_reg_find and Web_reg_save_param*

-
Verification's and Error Handling Verification points must be inserted into the scripts to verify the application being tested with the load. Verification function that are used in our application are:  Web_reg_find, web_reg_save_param These verification should be outside towards the end of the transaction. Mandatory verification points: ■          Login – whenever there is a login, verification points must be added to verify that the user logged in correctly. ■          Update Transactions – transactions that update the database must be verified that they are successful. ■          Each page should be verified using Web_reg_find function Using Web _ reg_find for verifications. Example: web_reg_find("Text= Main Package","SaveCount=Verify_Count1",LAST); Verification: if (strcmp(lr_eval_string("{Verify_Count1}"),"0")==0) { ...
Read more