Skip to main content

Linux tuning while running huge load using Jmeter

While running our jmeter load tests on Unix box with a target of 2500 User Load we got the exception “Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address” in jmeter log

We have set open files limit to 50000 and ran the test, but still we were getting the errors.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31182
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 50000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimit

Error in Jmeter Log file :
1487747366269,9,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-30,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0
1487747366276,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-297,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0
1487747366279,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-630,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0
1487747366279,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-272,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0



On Unix machine we performed the below steps for debug and resolution:

We see that all the local ports are occupied with TIME_WAIT or ESTABLISHED.


$ netstat -a -n|grep -E “^(tcp)”| cut -c 68-|sort|uniq -c|sort -n

Output:
$ netstat -a -n|grep -E "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n
     817 TIME_WAIT
     28046 ESTABLISHED

Checked for the Local Port range:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768   60999

Resolution:
1.    Increased the local port range using below command:
echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
This allows more local ports to be available.
2.    Enable fast recycling TIME_WAIT sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_recycle =1
By default,
cat /proc/sys/net/ipv4/tcp_tw_recycle
Output : 0 (disabled by default)
Be cautious if enabled in production environments, since this is our internal Environment and machine used only for Jmeter load tests, we enabled recycle and resolved the issue.
3.    Enable reuse of sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1
By default,
cat /proc/sys/net/ipv4/tcp_tw_reuse
Output : 0 (disabled by default)
Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.
After enabling fast recycling and reuse the server could support 5K user Load with single Unix box.

Note: The Load test was targeted for Login/Logout tests and not a heavy workload tests. 

Comments

  1. Nice Information

    We are the best piping design course in Hyderabad, India. Sanjary academy Offers Piping Design Course and Best Piping Design Training Institute in Hyderabad. Piping Design Institute in India Piping Design Engineering.
    Piping Design Course
    Piping Design Course in india
    Piping Design Course in hyderabad

    ReplyDelete
  2. Thanks for sharing information
    "Sanjary Academy provides excellent training for Piping design course. Best Piping Design Training Institute in Hyderabad,
    Telangana. We have offer professional Engineering Course like Piping Design Course,QA / QC Course,document Controller
    course,pressure Vessel Design Course, Welding Inspector Course, Quality Management Course, #Safety officer course."
    Piping Design Course
    Piping Design Course in India­
    Piping Design Course in Hyderabad
    QA / QC Course
    QA / QC Course in india
    QA / QC Course in Hyderabad
    Document Controller course
    Pressure Vessel Design Course
    Welding Inspector Course
    Quality Management Course
    Quality Management Course in india
    Safety officer course

    ReplyDelete

Post a Comment

Popular posts from this blog

Steps to Analyze AWR Report in Oracle

Steps to Analyze AWR Report in Oracle
AWR -Automaticworkload repository is a collection of persistentsystem performancestatisticsowned by SYS. It resides in SYSAUXtablespace. Bydefault snapshot are generated once every 60 min and maintained for 7 days. Each snapshot has a unique ID know as "snap_id". Snapshot detail can be found in "dba_hist_snapshot" view.
If we have Database performance issue and not the Database machine, then AWR Report is the place to look at. AWR is not used for real-time performance monitoring like the v$ tables. It is used for historical analysis of performance. AWR complements, but doesnot replace real-time monitoring.
Once AWR Report is generated in Oracle, the next task is to analyze it. By going through the AWR Report we can easily solve issues like slow database, high wait events, slow query and many more issues. Even though the report is lengthy, Analyzing or Reading relevant part of AWR Report can help to troubleshoot issues in easy and …

Using SQL Developer to create and view Tablespaces

Below are the steps Create and View Table Spaces settings using SQL Developer.
Required SQL developer version is version 3.0
To Create TableSpace :
Click on Menu View/DBA - DBA navigator window will appear.
In the DBA window add a new connection to the DB, and click connect. Then under storage option right click on Tablespaces and choose New Tablespace to create new one. Fill the Details as shown below:


To View the created table spaces: Under Storage, Select Data Files:
It will display the below: