Common list of Bottlenecks

-
Common list of Bottlenecks:

  • CPU:
    • CPU overload
    • Context switches -> 
      • too many threads on a core, 
      • bad luck with the linux scheduler 
      • too many system calls, etc...
    • IO waits -> all CPUs wait at the same speed
    • CPU Caches: Caching data is a fine grained process, in order to find the right balance between having multiple instances with different values for data and heavy synchronization to keep the cached data consistent.
    • Backplane throughput

  • Network:
    • NIC maxed out, IRQ saturation, soft interrupts taking up 100% CPU
    • DNS lookups
    • Dropped packets
    • Unexpected routes with in the network
    • Network disk access
    • Shared SANs
    • Server failure -> no answer anymore from the server
  • Memory:
    • Out of memory -> kills process, go into swap & grind to a halt
    • Out of memory causing Disk Thrashing (related to swap)
    • Memory library overhead
    • Memory fragmentation
      • In Java requires GC pauses
      • In C, malloc's start taking forever
  • Database:
    • Working size exceeds available RAM
    • Long & short running queries
    • Write-write conflicts
    • Large joins taking up memory
  • Virtualization:
    • Sharing a HDD, disk seek death
    • Network I/O fluctuations in the cloud
  • Programming:
    • Threads: deadlocks, heavyweight as compared to events, debugging, non-linear scalability, etc...
    • Event driven programming: callback complexity, how-to-store-state-in-function-calls, etc...
    • Lack of profiling, lack of tracing, lack of logging
    • One piece can't scale, SPOF, non-horizontally scalable, etc...
    • Stateful apps
    • Bad design : The developers create an app which runs fine on their computer. The app goes into production, and runs fine, with a couple of users. Months/Years later, the application can't run with thousands of users and needs to be totally re-architectured and rewritten.
    • Algorithm complexity
    • Dependent services like DNS lookups and whatever else you may block on.
    • Stack space
  • Disk:
    • Local disk access
    • Random disk I/O -> disk seeks
    • Disk fragmentation
    • SSDs performance drop once  data written is greater than SSD size
  • OS:
    • Fsync flushing, Linux buffer cache filling up
    • TCP buffers too small
    • File descriptor limits
    • Power budget
  • Caching:
    • Not using memcached (database pummeling)
    • In HTTP: headers, etags, not gzipping, etc..
    • Not utilizing the browser's cache enough
    • Byte code caches (e.g. PHP)
    • L1/L2 caches. This is a huge bottleneck. Keep important hot/data in L1/L2. This spans so much: snappy for network I/O, column DBs run algorithms directly on compressed data, etc. Then there are techniques to not destroy your TLB. The most important idea is to have a firm grasp on computer architecture in terms of CPUs multi-core, L1/L2, shared L3, NUMA RAM, data transfer bandwidth/latency from DRAM to chip, DRAM caches DiskPages, DirtyPages, TCP packets travel thru CPU<->DRAM<->NIC.

Comments

  1. AviDecember 14, 2017 at 2:31 PM

    sir update with some more bottleneck information if you have any new.

    ReplyDelete

Post a Comment

Popular posts from this blog

How to use a value between two different threads in Jmeter

-
Image
How to use a value between two different threads: Let us see how we can use the extracted value in one thread can be used in a different thread: As shown below we have Two threads (Thread Group – 1 and Thread Group -2). Under Thread Group -1 we have a HTTP Request where we extracted Request_Id and Session_Id. In our example we will use the same Ids in Thread Group -2 We have BeanShell PostProcessor to log the output of the variables in the output log. The Lines in the both BeanShell is as below: log.info("Request Id in Thread 1 = " + "${Request_Id}"); log.info("Session Id in Thread 1 = " + "${Session_Id}"); Now if we run the Script we get the below output: As we see the values of both the IDs are not passed to Thread Group-2. Now we will add few lines in the BeanShell PostProcessor as below and run again: ${__setProperty(Request_Id,"${Request_Id}")}; ${__setProperty(Session_Id,"${Session_Id}")};
Read more

Steps to Analyze AWR Report in Oracle

-
Image
Steps to Analyze AWR Report in Oracle AWR -   Automatic   workload repository is a collection of persistent   system performance   statistics   owned by SYS. It resides in SYSAUX   tablespace . By default snapshot are generated once every 60 min and maintained for 7 days . Each snapshot has a unique ID know as "snap_id". Snapshot detail can be found in " dba_hist_snapshot " view.   If we have Database performance issue and not the Database machine, then AWR Report is the place to look at. AWR is not used for real-time performance monitoring like the v$ tables. It is used for historical analysis of performance. AWR complements, but doesnot replace real-time monitoring. Once AWR Report is generated in Oracle, the next task is to analyze it. By going through the AWR Report we can easily solve issues like slow database, high wait events, slow query and many more issues. Even though the report is lengthy, Analyzing or Reading relevant part of AWR Report can
Read more

Correlation and checking in Results

-
Image
Correlation and checking In Previous post we have randomly selected value for depart and arrive. Each Depart and arrive have a outboundFlight and returnFlight field, which changes according to the selected values. For E.g.: When we recorded the script with depart as “London” and arrive as “Paris” the below flight details was recorded accordingly to the selection.      /*web_submit_form("reservations.pl_2",          "Snapshot=t5.inf",          ITEMDATA,          "Name=outboundFlight", "Value=241;97;03/28/2014", ENDITEM,          "Name=returnFlight", "Value=422;102;03/29/2014", ENDITEM,          "Name=reserveFlights.x", "Value=26", ENDITEM,          "Name=reserveFlights.y", "Value=8", ENDITEM,          LAST); */ As we see above the outboundFlight and returnFlight is hard coded and needs to be correlated according to the selected value. So inserting the c
Read more