Linux tuning while running huge load using Jmeter

While running our jmeter load tests on Unix box with a target of 2500 User Load we got the exception “Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address” in jmeter log

We have set open files limit to 50000 and ran the test, but still we were getting the errors.

$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 31182

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 50000

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 4096

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimit

Error in Jmeter Log file :

1487747366269,9,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-30,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0

1487747366276,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-297,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0

1487747366279,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-630,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0

1487747366279,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-272,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0

On Unix machine we performed the below steps for debug and resolution:

We see that all the local ports are occupied with TIME_WAIT or ESTABLISHED.

$ netstat -a -n|grep -E “^(tcp)”| cut -c 68-|sort|uniq -c|sort -n

Output:

$ netstat -a -n|grep -E "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n

     817 TIME_WAIT

     28046 ESTABLISHED

Checked for the Local Port range:

$ cat /proc/sys/net/ipv4/ip_local_port_range

32768   60999

Resolution:

1.    Increased the local port range using below command:

echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range

This allows more local ports to be available.

2.    Enable fast recycling TIME_WAIT sockets as below:

$ sudo sysctl -w net.ipv4.tcp_tw_recycle =1

By default,

cat /proc/sys/net/ipv4/tcp_tw_recycle

Output : 0 (disabled by default)

Be cautious if enabled in production environments, since this is our internal Environment and machine used only for Jmeter load tests, we enabled recycle and resolved the issue.

3.    Enable reuse of sockets as below:

$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1

By default,

cat /proc/sys/net/ipv4/tcp_tw_reuse

Output : 0 (disabled by default)

Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.

After enabling fast recycling and reuse the server could support 5K user Load with single Unix box.

Note: The Load test was targeted for Login/Logout tests and not a heavy workload tests.