乐趣区

Lock-wait-timeout-exceeded代码该优化了

jiezi

背景

最近在排查问题时发现，偶尔会发生关于数据库锁超时的现象，会发生像如下的报错信息：

Exception in thread "pool-3-thread-1" org.springframework.dao.CannotAcquireLockException: 
### Error updating database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may involve com.zr.center.mybatis.auto.mapper.UserMapper.updateByExampleSelective-Inline
### The error occurred while setting parameters
### SQL: update user      SET user_name = ?                          WHERE (user_id = ?)
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; ]; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:262)
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
    at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:75)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:447)
    at com.sun.proxy.$Proxy101.update(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:295)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:59)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53)
    at com.sun.proxy.$Proxy103.updateByExampleSelective(Unknown Source)
    at com.zr.center.framework.web.service.BaseService.updateByExampleSelective(BaseService.java:97)
    at com.zr.center.api.test.service.TestService.updateUserName(TestService.java:34)
    at com.zr.center.api.test.service.TestService$$FastClassBySpringCGLIB$$bd3aa32.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
    at com.zr.center.api.test.service.TestService$$EnhancerBySpringCGLIB$$59b19302.updateUserName(<generated>)
    at com.zr.center.ApplicationTests.lambda$testTxLockWaiting$0(ApplicationTests.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
    at com.mysql.jdbc.Util.getInstance(Util.java:408)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:952)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)

排查

经过排查，DBA 给出的日志中并未有死锁，死锁的原因排除，查询业务日志发现在高并发的时期有时会有重复请求过来，也有一个服务在处理某个逻辑时会发一条 mq 消息，而同时会消费这条消息，此时也会导致锁超时。超时原因就是因为一个事务中处理的逻辑过多，有调外部服务 (超时)，有更新其它多张表的操作，这样就会导致后面事务请求超时，报以上错误。

重现步骤

  • 数据库配置

关于配置信息查看，可以看到事务隔离是 RR, 事务锁等待时长为默认的 50s

  • 事务代码 

/**
 * @description: 用户服务超时测试
 * @author: chong guo
 * @create: 2018-12-10 14:44
 */
@Service
@Slf4j
public class TestService extends BaseService<User, UserExample, Long> {

    /**
     * 更新用户名称
     *
     * @param userId
     * @param name
     */
    @Transactional(rollbackFor = Exception.class)
    public void updateUserName(Long userId, String name) throws InterruptedException {log.info("开始更新用户名【{}】，用户 ID 为【{}】", name, userId);
        UserExample userExample = new UserExample();
        userExample.createCriteria().andUserIdEqualTo(userId);

        User user = new User();
        user.setUserName(name);

        super.updateByExampleSelective(user, userExample);

        // 模拟业务超时，有可能调用外部远程服务超时，也有可能处理其它逻辑
        Thread.sleep(55000);
        log.info("结束更新用户名【{}】，用户 ID 为【{}】", name, userId);

    }

}

  • 模拟并发 

/**
 * @author  Chong Guo
 */
@RunWith(SpringRunner.class)
@SpringBootTest
@Slf4j
public class ApplicationTests {
    @Resource
    TestService testService;

    private final int threadCount = 5;

    @Test
    public void testTxLockWaiting() throws InterruptedException {CountDownLatch countDownLatch = new CountDownLatch(threadCount);

        ExecutorService threadPool = Executors.newFixedThreadPool(300);
        for (int i = 0; i < threadCount; i++) {threadPool.execute(() -> {
                try {testService.updateUserName(611526166943105024L, "chongguo");
                } catch (InterruptedException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();} finally {countDownLatch.countDown();
                }

            });
        }
        countDownLatch.await();
        threadPool.shutdown();
        log.info("Test tx lock is over");
        Thread.sleep(100000);
    }
}

运行代码后会现三次失败，二次成功，失败原因都是锁超时

总结

  1. 代码优化，将大事务中无关的逻辑拆出来，异步处理
  2. 业务提供方涉及到外部调用的接口需要把超时时长设置短一些
  3. 如果调用方有重试机制，可以针对业务去掉重试，将 connetion time 设置大一些
退出移动版