Using gdb to analyse core dumps in Linux

-
Once core file is generated.
Steps to analyze the core dump file-

Run “gdb /usr/java/jdk1.6.0_24/bin/java (java path) core.11151 (core file)”
You will get the gdb prompt
Type “bt” for stack trace.
To switch the thread, type “thread [thread no]”
To check the stack trace for selected thread again type “bt”
To get the stack of all the thread run “thread apply all bt full”
Find “LowMemoryDetector::low_memory_detector_thread_entry” in stack trace.
To find exactly in which thread it occured, type "where"

Example of "where" output:
where
#0  0xffffe410 in __kernel_vsyscall ()
#1  0x00b0ddf0 in raise () from /lib/libc.so.6
#2  0x00b0f701 in abort () from /lib/libc.so.6
#3  0xf78e123f in os::abort(bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#4  0xf7a28431 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#5  0xf7a28fe1 in crash_handler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#6  <signal handler called>
#7  0xf756310b in oopDesc::size_given_klass(Klass*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#8  0xf78f8fc7 in ParallelScavengeHeap::block_start(void const*) const () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#9  0xf78de576 in os::print_location(outputStream*, int, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#10 0xf78e8d0c in os::print_context(outputStream*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#11 0xf7a27bd3 in VMError::report(outputStream*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#12 0xf7a28370 in VMError::report_and_die() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#13 0xf78e81dc in JVM_handle_linux_signal () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#14 0xf78e4124 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
#15 <signal handler called>
#16 0x00b4ef5f in _int_malloc () from /lib/libc.so.6
#17 0x00b50fb7 in malloc () from /lib/libc.so.6
#18 0x4af28af7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#19 0x4cb67075 in KCDdFieldJDisp::CboInvoke(long) ()
   from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#20 0x4cb56132 in KCboJDisp::Invoke(JNIEnv_*, int, _jobjectArray*) ()
   from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#21 0x4cb3db43 in Java_com_clarify_cbo_KJniDisp_invoke ()
   from /test/CoreNFT/users/crmnft1/yogesh/everest/AmdocsCRM8.1.1/AmdocsCRMApplication/bin/libJDispBase.so
#22 0xf447154c in ?? ()
#23 0x4c640118 in ?? ()
#24 0x4a9fe868 in ?? ()
#25 0x3fe6b0b8 in ?? ()
#26 0x00000000 in ?? ()
(gdb)

Example of "full" stack of all thread
thread apply all bt full
Thread 81 (Thread 26737):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c46a77 in pthread_join () from /lib/libpthread.so.0
No symbol table info available.
#2  0x0804dd78 in ContinueInNewThread ()
No symbol table info available.
#3  0x080497f6 in main ()
No symbol table info available.
Thread 80 (Thread 26738):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3  0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4  0xf79b31bb in ObjectMonitor::wait(long long, bool, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5  0xf79b0383 in ObjectSynchronizer::wait(Handle, long long, Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6  0xf774f9bc in JVM_MonitorWait () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7  0xf428e62a in ?? ()
No symbol table info available.
#8  0x09db2918 in ?? ()
No symbol table info available.
#9  0xf731eff0 in ?? ()
No symbol table info available.
#10 0x00000000 in ?? ()
No symbol table info available.
Thread 79 (Thread 26739):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3  0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4  0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5  0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6  0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#7  0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8  0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9  0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
Thread 78 (Thread 26740):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00c49bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00c4a3fe in pthread_cond_wait@GLIBC_2.0 () from /lib/libpthread.so.0
No symbol table info available.
#3  0xf78e5fbe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4  0xf78c302e in Monitor::IWait(Thread*, long long) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5  0xf78c3865 in Monitor::wait(bool, long, bool) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6  0xf7689ab6 in GCTaskManager::get_task(unsigned int) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7  0xf768ac1b in GCTaskThread::run() () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#8  0xf78e6a6e in java_start(Thread*) () from /usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
No symbol table info available.
#9  0x00c45832 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00bb70ae in clone () from /lib/libc.so.6
No symbol table info available.
...
...
..

Comments

Post a Comment

Popular posts from this blog

How to use a value between two different threads in Jmeter

-
Image
How to use a value between two different threads: Let us see how we can use the extracted value in one thread can be used in a different thread: As shown below we have Two threads (Thread Group – 1 and Thread Group -2). Under Thread Group -1 we have a HTTP Request where we extracted Request_Id and Session_Id. In our example we will use the same Ids in Thread Group -2 We have BeanShell PostProcessor to log the output of the variables in the output log. The Lines in the both BeanShell is as below: log.info("Request Id in Thread 1 = " + "${Request_Id}"); log.info("Session Id in Thread 1 = " + "${Session_Id}"); Now if we run the Script we get the below output: As we see the values of both the IDs are not passed to Thread Group-2. Now we will add few lines in the BeanShell PostProcessor as below and run again: ${__setProperty(Request_Id,"${Request_Id}")}; ${__setProperty(Session_Id,"${Session_Id}")};
Read more

Steps to Analyze AWR Report in Oracle

-
Image
Steps to Analyze AWR Report in Oracle AWR -   Automatic   workload repository is a collection of persistent   system performance   statistics   owned by SYS. It resides in SYSAUX   tablespace . By default snapshot are generated once every 60 min and maintained for 7 days . Each snapshot has a unique ID know as "snap_id". Snapshot detail can be found in " dba_hist_snapshot " view.   If we have Database performance issue and not the Database machine, then AWR Report is the place to look at. AWR is not used for real-time performance monitoring like the v$ tables. It is used for historical analysis of performance. AWR complements, but doesnot replace real-time monitoring. Once AWR Report is generated in Oracle, the next task is to analyze it. By going through the AWR Report we can easily solve issues like slow database, high wait events, slow query and many more issues. Even though the report is lengthy, Analyzing or Reading relevant part of AWR Report can
Read more

Correlation and checking in Results

-
Image
Correlation and checking In Previous post we have randomly selected value for depart and arrive. Each Depart and arrive have a outboundFlight and returnFlight field, which changes according to the selected values. For E.g.: When we recorded the script with depart as “London” and arrive as “Paris” the below flight details was recorded accordingly to the selection.      /*web_submit_form("reservations.pl_2",          "Snapshot=t5.inf",          ITEMDATA,          "Name=outboundFlight", "Value=241;97;03/28/2014", ENDITEM,          "Name=returnFlight", "Value=422;102;03/29/2014", ENDITEM,          "Name=reserveFlights.x", "Value=26", ENDITEM,          "Name=reserveFlights.y", "Value=8", ENDITEM,          LAST); */ As we see above the outboundFlight and returnFlight is hard coded and needs to be correlated according to the selected value. So inserting the c
Read more