dbcp 连接池不合理的锁导致连接耗尽解决方案
dbcp连接池不合理的锁导致连接耗尽解决方案
应用报错,表象来看是连接池爆满了。
org.springframework.transaction.CannotCreateTransactionException:CouldnotopenJDBCConnectionfortransaction;nestedexceptionisorg.apache.commons.dbcp.SQLNestedException:Cannotgetaconnection,poolexhausted atorg.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241)~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE] atcom.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)[dubbo-2.5.3.jar:2.5.3] atjava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[na:1.6.0_33] atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[na:1.6.0_33] atjava.lang.Thread.run(Thread.java:662)[na:1.6.0_33] Causedby:org.apache.commons.dbcp.SQLNestedException:Cannotgetaconnection,poolexhausted atorg.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103)~[commons-dbcp.jar:1.2.1] atorg.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)~[commons-dbcp.jar:1.2.1] atcom.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)~[crmpub-jdbc-1.0.jar:1.0] atorg.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE] ...32commonframesomitted Causedby:java.util.NoSuchElementException:Timeoutwaitingforidleobject atorg.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)~[commons-pool-1.6.jar:1.6] atorg.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)~[commons-dbcp.jar:1.2.1] atorg.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)~[commons-dbcp.jar:1.2.1] ...35commonframesomitted
应用堆栈，几十条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject方法了，此方法在等待0x0000000709a638a0对象锁，但0x0000000709a638a0对象一直被其中一条线程执行PoolableConnectionFactory.makeObject时锁了，而且这条线程获取到锁后还阻塞住了，所以导致后面几十条线程都BLOCKED了。
"DubboServerHandler-132.121.95.136:28101-thread-300"daemonprio=10tid=0x00007f02b4136800nid=0x4954waitingformonitorentry[0x00007f0286018000] java.lang.Thread.State:BLOCKED(onobjectmonitor) atorg.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) -waitingtolock<0x0000000709a638a0>(aorg.apache.commons.dbcp.PoolableConnectionFactory) atorg.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188) atorg.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) atorg.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) atorg.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) atcom.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) atorg.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) atorg.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372) atorg.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417) atorg.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255) atorg.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) atorg.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) atcom.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74) atsun.reflect.GeneratedMethodAccessor253.invoke(UnknownSource) atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) atjava.lang.reflect.Method.invoke(Method.java:597) atorg.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) atorg.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) atorg.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) atorg.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631) atcom.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>) atcom.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java) atcom.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) atcom.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) atcom.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) atcom.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90) atcom.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) atcom.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) atcom.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) atcom.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) atcom.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) atcom.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) atjava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) atjava.lang.Thread.run(Thread.java:662) Lockedownablesynchronizers: -<0x000000070a120f00>(ajava.util.concurrent.locks.ReentrantLock$NonfairSync)
"DubboServerHandler-132.121.95.136:28101-thread-290"daemonprio=10tid=0x00007f03203da800nid=0x4948runnable[0x00007f0286a22000] java.lang.Thread.State:RUNNABLE atjava.net.SocketInputStream.socketRead0(NativeMethod) atjava.net.SocketInputStream.read(SocketInputStream.java:129) atcom.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112) atcom.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159) atcom.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187) -locked<0x0000000715a9ce70>(acom.mysql.jdbc.util.ReadAheadInputStream) atcom.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140) atcom.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597) atcom.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085) atcom.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494) atcom.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527) atcom.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309) -locked<0x00000007159d1898>(acom.mysql.jdbc.JDBC4Connection) atcom.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) atcom.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46) atsun.reflect.GeneratedConstructorAccessor78.newInstance(UnknownSource) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) atjava.lang.reflect.Constructor.newInstance(Constructor.java:513) atcom.mysql.jdbc.Util.handleNewInstance(Util.java:408) atcom.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419) atcom.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344) atorg.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) atorg.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) -locked<0x0000000709a638a0>(aorg.apache.commons.dbcp.PoolableConnectionFactory) atorg.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188) atorg.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) atorg.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) atorg.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) atcom.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) atorg.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) atorg.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372) atorg.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417) atorg.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255) atorg.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) atorg.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) atcom.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74) atsun.reflect.GeneratedMethodAccessor253.invoke(UnknownSource) atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) atjava.lang.reflect.Method.invoke(Method.java:597) atorg.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) atorg.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) atorg.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) atorg.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631) atcom.eshore.crm.service.cachemgr.sca.common.DictCacheServiceImpl$$EnhancerByCGLIB$$e27e8e25.getDict(<generated>) atcom.alibaba.dubbo.common.bytecode.Wrapper93.invokeMethod(Wrapper93.java) atcom.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) atcom.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) atcom.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) atcom.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90) atcom.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) atcom.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) atcom.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) atcom.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) atcom.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) atcom.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) atjava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) atjava.lang.Thread.run(Thread.java:662) Lockedownablesynchronizers: -<0x000000070a0f9218>(ajava.util.concurrent.locks.ReentrantLock$NonfairSync)
可以看到其实获取锁的线程是阻塞在这里了，做mysql握手包的时候一直在等待读，阻塞在网络IO了。
java.lang.Thread.State:RUNNABLE atjava.net.SocketInputStream.socketRead0(NativeMethod) atjava.net.SocketInputStream.read(SocketInputStream.java:129) atcom.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112) atcom.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159) atcom.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187) -locked<0x0000000715a9ce70>(acom.mysql.jdbc.util.ReadAheadInputStream) atcom.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140) atcom.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597) atcom.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085) atcom.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494) atcom.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527) atcom.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)
这个是dbcp1.x版本在创建新的连接对象时会把整个连接工厂类对象锁了，一旦创建过程中发生阻塞就会导致整个池都死掉。这个问题很大。
解决方案，升级dbcp2.x版本或tomcatjdbc或druid连接池，推荐druid连接池，功能强大自带监控。
感谢阅读，希望能帮助到大家，谢谢大家对本站的支持！