AnsweredAssumed Answered

Activiti can't get DB connection

Question asked by tomsko on Mar 20, 2014
Latest reply on Mar 21, 2014 by trademak
Hi,
I've put Activiti under some high load and faced problems with DB connections. For first ~minute processing was fine, but later everything freezed just to throw "IJ000453: Unable to get managed connection for java:jboss/activitids" exception 90 secs later.
<!–break–>
Activiti Engine instance has it's own db connection pool from JBoss 7.2 and is configured to consume 20 DB connections at max.

When analyzing thread dump in the 90sec period between the freeze and the stack, I've found that every Activiti operation which keeps DB connection is blocked at DbIdGenerator.getNextId e.g.:

"pool-15-thread-125" prio=10 tid=0x00007ff8381a8800 nid=0x2179 waiting for monitor entry [0x00007ff7e3324000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   at org.activiti.engine.impl.db.DbIdGenerator.getNextId(DbIdGenerator.java:35)
   - waiting to lock <0x000000074c929640> (a org.activiti.engine.impl.db.DbIdGenerator)
   at org.activiti.engine.impl.db.DbSqlSession.insert(DbSqlSession.java:111)
   at org.activiti.engine.impl.persistence.entity.JobEntity.insert(JobEntity.java:83)
   at org.activiti.engine.impl.persistence.entity.JobEntityManager.send(JobEntityManager.java:43)
   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.scheduleAtomicOperationAsync(ExecutionEntity.java:545)
        ….


Such threads consume all available pool connections.

On the other hand, threads (there are 2) which actually try to do DbIdGenerator.getNextId are pending on Db connection. E.g.:

"pool-8-thread-101" prio=10 tid=0x0000000041098800 nid=0x2189 waiting on condition [0x00007ff7e4d2e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007461fd8e8> (a java.util.concurrent.Semaphore$FairSync)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
   at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
   at org.jboss.jca.core.connectionmanager.pool.mcp.Semaphore.tryAcquire(Semaphore.java:59)
   at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:253)
   at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:495)
   at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:374)
   at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:364)
   at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
   at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
   at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
   at org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:84)
   at org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:58)
   at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:271)
   at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:69)
   at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:56)
   at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)
   at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)
   at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)
   at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:104)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:98)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:62)
   at org.activiti.engine.impl.db.DbSqlSession.selectById(DbSqlSession.java:308)
   at org.activiti.engine.impl.persistence.entity.PropertyEntityManager.findPropertyById(PropertyEntityManager.java:25)
   at org.activiti.engine.impl.cmd.GetNextIdBlockCmd.execute(GetNextIdBlockCmd.java:34)
   at org.activiti.engine.impl.cmd.GetNextIdBlockCmd.execute(GetNextIdBlockCmd.java:24)
   at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
   at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
   at org.activiti.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:64)
   at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:37)
   at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
   at org.activiti.engine.impl.db.DbIdGenerator.getNewBlock(DbIdGenerator.java:43)
   - locked <0x0000000749ac3cc8> (a org.activiti.engine.impl.db.DbIdGenerator)
   at org.activiti.engine.impl.db.DbIdGenerator.getNextId(DbIdGenerator.java:36)
   - locked <0x0000000749ac3cc8> (a org.activiti.engine.impl.db.DbIdGenerator)
   at org.activiti.engine.impl.db.DbSqlSession.insert(DbSqlSession.java:111)
   at org.activiti.engine.impl.persistence.AbstractManager.insert(AbstractManager.java:50)
        …


How to handle such case? Is it a bug or a configuration issue?

Outcomes