Linux tuning while running huge load using Jmeter

While running our jmeter load tests on Unix box with a target of 2500 User Load we got the exception “Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address” in jmeter log

We have set open files limit to 50000 and ran the test, but still we were getting the errors.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31182
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 50000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimit

Error in Jmeter Log file :
1487747366269,9,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-30,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0
1487747366276,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-297,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0
1487747366279,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-630,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0
1487747366279,2,GET /sample/,Non HTTP response code: java.net.NoRouteToHostException,Non HTTP response message: Cannot assign requested address,Thread Group 1-272,text,false,Test failed: text expected to contain /Logout/,2128,738,738,0,0



On Unix machine we performed the below steps for debug and resolution:

We see that all the local ports are occupied with TIME_WAIT or ESTABLISHED.


$ netstat -a -n|grep -E “^(tcp)”| cut -c 68-|sort|uniq -c|sort -n

Output:
$ netstat -a -n|grep -E "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n
     817 TIME_WAIT
     28046 ESTABLISHED

Checked for the Local Port range:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768   60999

Resolution:
1.    Increased the local port range using below command:
echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
This allows more local ports to be available.
2.    Enable fast recycling TIME_WAIT sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_recycle =1
By default,
cat /proc/sys/net/ipv4/tcp_tw_recycle
Output : 0 (disabled by default)
Be cautious if enabled in production environments, since this is our internal Environment and machine used only for Jmeter load tests, we enabled recycle and resolved the issue.
3.    Enable reuse of sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1
By default,
cat /proc/sys/net/ipv4/tcp_tw_reuse
Output : 0 (disabled by default)
Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.
After enabling fast recycling and reuse the server could support 5K user Load with single Unix box.

Note: The Load test was targeted for Login/Logout tests and not a heavy workload tests. 

Comments

  1. Nice Information

    We are the best piping design course in Hyderabad, India. Sanjary academy Offers Piping Design Course and Best Piping Design Training Institute in Hyderabad. Piping Design Institute in India Piping Design Engineering.
    Piping Design Course
    Piping Design Course in india
    Piping Design Course in hyderabad

    ReplyDelete
  2. Thanks for sharing information
    "Sanjary Academy provides excellent training for Piping design course. Best Piping Design Training Institute in Hyderabad,
    Telangana. We have offer professional Engineering Course like Piping Design Course,QA / QC Course,document Controller
    course,pressure Vessel Design Course, Welding Inspector Course, Quality Management Course, #Safety officer course."
    Piping Design Course
    Piping Design Course in India­
    Piping Design Course in Hyderabad
    QA / QC Course
    QA / QC Course in india
    QA / QC Course in Hyderabad
    Document Controller course
    Pressure Vessel Design Course
    Welding Inspector Course
    Quality Management Course
    Quality Management Course in india
    Safety officer course

    ReplyDelete
  3. Good post.....I appreciate yor way of writing that make the blog attractive and make reader to hold longer to your blog. Thank you for sharing.Performance testing services is a highly specialized field of testing.

    ReplyDelete
  4. In this fashion my friend Wesley Virgin's autobiography launches with this shocking and controversial VIDEO.

    As a matter of fact, Wesley was in the military-and soon after leaving-he found hidden, "mind control" tactics that the CIA and others used to get everything they want.

    These are the exact same tactics lots of celebrities (especially those who "became famous out of nothing") and elite business people used to become wealthy and successful.

    You probably know how you use less than 10% of your brain.

    Mostly, that's because most of your brainpower is UNCONSCIOUS.

    Perhaps that thought has even taken place INSIDE your own brain... as it did in my good friend Wesley Virgin's brain seven years back, while riding an unlicensed, beat-up trash bucket of a car without a license and $3.20 in his bank account.

    "I'm very fed up with going through life check to check! Why can't I turn myself successful?"

    You've taken part in those types of conversations, ain't it so?

    Your own success story is waiting to start. You just need to take a leap of faith in YOURSELF.

    Take Action Now!

    ReplyDelete
  5. This is really a worthy and wonderful blog about Software Testing Companies in Chennai to read and further more tips on the Software Testing Companies in Bangalore have been learnt. thanks for sharing your views among us and its great time spending on this. I am waiting for new post here about Software Testing Services in Chennai and Please keep it up in future..

    ReplyDelete

Post a Comment

Popular posts from this blog

Steps to Analyze AWR Report in Oracle

Verifications and Error Handling in LoadRunner *Web_reg_find and Web_reg_save_param*

Examples on Date and Time in LoadRunner