Tuning JDBC Data Sources Connection Pool in WebLogic

Tuning JDBC Data Sources Connection Pool in WebLogic:

While we are Load testing Web services with our application deployed on WebLogic Server Version: 10.3.6.0 most of transactions started failing and when checked in the logs we found the below exception:

####<28.08.2015 17:32:54 MSK> <Info> <Common> <server.com> <clust1> <[ACTIVE] ExecuteThread: '50' for queue: 'weblogic.kernel.Default (self-tuning)'> <internal> <BEA1-17FBAA30098FE2C54C98> <> <1439904774589> <BEA-000627> <Reached maximum capacity of pool "AppDataSource", making "0" new resource instances instead of "1".>

####<28.08.2015 17:32:54 MSK> <Error> <server.com> <clust1> <[ACTIVE] ExecuteThread: '82' for queue: 'weblogic.kernel.Default (self-tuning)'> <guest> <> <> <1439904774589> <BEA-000000> <ERROR wf - WfSubscriberSysJMSBean.onMessage:

WF Engine detected error:

com.App.Exception: java.security.PrivilegedActionException: java.rmi.RemoteException: EJB Exception: ; nested exception is:

        <……Error Stack>

Caused by: com.app.framework.jdbc.DataAccessException: Error due to access to database:

<…. Error Stack>

Caused by: weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool AppDataSource to allocate to applications, please increase the size of the pool and retry..

        at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:252)

        at weblogic.jdbc.jts.Driver.newConnection(Driver.java:938)

        at weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:336)

        at weblogic.jdbc.jts.Driver.connect(Driver.java:173)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)

       at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)

        at com.App.framework.jdbc.JDBCTemplates.execute(JDBCTemplates.java:130)

        ... 25 more

> 

We have the below values set for JDBC data pool settings:

<initial-capacity>10</initial-capacity>

<max-capacity>35</max-capacity>

   

As we see in the above screenshot all the 35 were occupied when test was running. So the exception is obvious.

We need to increase the JDBC Connection Pool.

So before increasing the number to some random number we re-calculated the required capacity based on required work manager threads.

In work managers we have types defined are as following :

Work Manager, Fair Share Request Class and Maximum Threads Constraint.

More information on work managers can be found at http://docs.oracle.com/cd/E17904_01/web.1111/e13701/self_tuned.htm#CNFGD112

For MaxThreadsConstraint the count is set to 25 as below. Also same is set for two more work managers

For XA DataSource:

The simple approach to sizing the pool is to set the initial/minimum capacity to the maximum capacity.  Doing this creates all connections at startup, avoiding creating connections on demand and the pool is stable.

So the initial-capacity is set equal to max-capacity. Both are set to 80.

We have in total 21 work Managers configured. Out of 21, 3 are MaxThreadConstaints. So calculation is done as below:

(MaxThreadsConstraint (25) + widgetJobsMaxThreadsConstraint(16) + widgetCancelJobsMaxThreadsConstraint (4)) * 1.2 + 18 (Work Manager, Fair Share Request Class) = 72 connection to take a small margin 80.

Equality initial-capacity = max-capacity is also driven by the desire to reduce the cost base in create / delete / shrink connections in the pools.

 In WLS 10.3.6, minimum capacity is used to specify the lower limit to which a pool will shrink.  If minimum capacity is not set, it defaults to the initial capacity for upward compatibility.

So we have initial-capacity = max-capacity = min-capacity = 80

More details on data source connection pool sizing can be found at https://blogs.oracle.com/WebLogicServer/entry/data_source_connection_pool_sizing

So we have below settings:

For Non-XA data source the settings are as below:

Increased from 5 to 15

Also in all descriptors, data sources are necessary to register two additional parameters designed to protect the stability of the Productions in the future from a different kind of problem.

 The two parameters added are as below:

  <internal-properties>

    <property>

      <name>CountOfRefreshFailuresTillDisable</name>

      <value>0</value>

    </property>

    <property>

      <name>CountOfTestFailuresTillFlush</name>

      <value>0</value>

    </property>

  </internal-properties>

1. CountOfRefreshFailuresTillDisable

Specifies the number of reconnect failures allowed before WebLogic Server disables a connection pool to minimize the delay in handling the connection request caused by a database failure. Zero means it is disabled. Default value is 2

2. CountOfTestFailuresTillFlush

Specifies the number of test failures allowed before WebLogic Server closes all unused connections in a connection pool to minimize the delay caused by further database testing. Zero means it is disabled. Default value is 2

After running the test with these settings we no more have the “weblogic.jdbc.extensions.PoolLimitSQLException” in the logs