Unable to create new native threads resolution

In one of our new machines, we are trying to start our servers. For 4 Servers, the system was working properly, but with 8 Servers it was throwing below exception.

[gsc][3/20922] Caused by: java.lang.OutOfMemoryError: unable to create new native thread
2012-09-19 20:06:24,217 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][3/20922]: Reported Error [Restart Regex [.*java.lang.OutOfMemoryError.*] matching [Caused by: java.lang.OutOfMemoryError: unable to create new native thread]], restarting
[gsc][1/] /<path>//bin/gs.sh: line 92: 21421 Aborted (core dumped) "$JAVACMD" ${JAVA_OPTIONS} -DagentId=${AGENT_ID} -DgsaServiceID=${GSA_SERVICE_ID} $bootclasspath $script_classpath ${RMI_OPTIONS} $libpath ${LOOKUP_GROUPS_PROP} ${LOOKUP_LOCATORS_PROP} -Dcom.gs.logging.debug=false ${GS_LOGGING_CONFIG_FILE_PROP} $NETWORK $DEBUG $launchTarget $command_line


#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (gcTaskThread.cpp:46), pid=26777, tid=140045563242240
#
# JRE version: 7.0_04-b20
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.0-b21 mixed mode linux-amd64 compressed oops)
I cannot even call java -version.
$ /usr/java/jre1.7.0_04/bin/java -version
Picked up JAVA_TOOL_OPTIONS: -Djava.security.egd=file:/dev/../dev/urandom
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# <path>/hs_err_pid29487.log
$ /usr/java/jre1.7.0_04/bin/java -version
Picked up JAVA_TOOL_OPTIONS: -Djava.security.egd=file:/dev/../dev/urandom
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# <Path>/hs_err_pid29491.log

Solution:
Change the ulimit parameters as below:

In /etc/security/limits.conf add the following to the end of the file:

* soft nproc 15349
* hard nproc 16384
* soft nofile 32768
* hard nofile 65536
* soft core 2097152
* hard core unlimited

We had done that previously, however one more thing needed to be changed for the number of user processes.

In /etc/security/limits.d/90-nproc.conf change the soft nproc from 1024 to 15349:
*          soft    nproc     15349

This change is not applied until the user logs out and then logs back in.

$ ulimit -u
15349

Now try to run the program, there will be no out of memory exceptions.