起源：blog.csdn.net/kisscatforever/article/details/79817039
一、前言
前两天做了一个导入的性能，导入开始的时候十分慢，导入2w条数据要1分多钟，起初一点一点的优化，从间接把list怼进Mysql中，到调配把list导入Mysql中，到多线程把list导入Mysql中。
工夫是一点一点的变少了。十分的爽，最初变成了10s以内。
上面就展现一下过程。
二、间接把list怼进Mysql
应用mybatis的批量导入操作：
@Transactional(rollbackFor = Exception.class)public int addFreshStudentsNew2(List<FreshStudentAndStudentModel> list, String schoolNo) { if (list == null || list.isEmpty()) { return 0; } List<StudentEntity> studentEntityList = new LinkedList<>(); List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>(); List<AllusersEntity> allusersEntityList = new LinkedList<>(); for (FreshStudentAndStudentModel freshStudentAndStudentModel : list) { EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity(); StudentEntity studentEntity = new StudentEntity(); BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity); BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity); String operator = TenancyContext.UserID.get(); String studentId = BaseUuidUtils.base58Uuid(); enrollStudentEntity.setId(BaseUuidUtils.base58Uuid()); enrollStudentEntity.setStudentId(studentId); enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard()); enrollStudentEntity.setOperator(operator); studentEntity.setId(studentId); studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard()); studentEntity.setOperator(operator); studentEntityList.add(studentEntity); enrollStudentEntityList.add(enrollStudentEntity); AllusersEntity allusersEntity = new AllusersEntity(); allusersEntity.setId(enrollStudentEntity.getId()); allusersEntity.setUserCode(enrollStudentEntity.getNemtCode()); allusersEntity.setUserName(enrollStudentEntity.getName()); allusersEntity.setSchoolNo(schoolNo); allusersEntity.setTelNum(enrollStudentEntity.getTelNum()); allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //明码设置为考生号 allusersEntityList.add(allusersEntity); } enResult = enrollStudentDao.insertAll(enrollStudentEntityList); stuResult = studentDao.insertAll(studentEntityList); allResult = allusersFacade.insertUserList(allusersEntityList); if (enResult > 0 && stuResult > 0 && allResult) { return 10; } return -10;}Mapper.xml
<insert id="insertAll" parameterType="com.dmsdbj.itoo.basicInfo.entity.EnrollStudentEntity"> insert into tb_enroll_student <trim prefix="(" suffix=")" suffixOverrides=","> id, remark, nEMT_aspiration, nEMT_code, nEMT_score, student_id, identity_card_id, level, major, name, nation, secondary_college, operator, sex, is_delete, account_address, native_place, original_place, used_name, pictrue, join_party_date, political_status, tel_num, is_registry, graduate_school, create_time, update_time </trim> values <foreach collection="list" item="item" index="index" separator=","> ( #{item.id,jdbcType=VARCHAR}, #{item.remark,jdbcType=VARCHAR}, #{item.nemtAspiration,jdbcType=VARCHAR}, #{item.nemtCode,jdbcType=VARCHAR}, #{item.nemtScore,jdbcType=VARCHAR}, #{item.studentId,jdbcType=VARCHAR}, #{item.identityCardId,jdbcType=VARCHAR}, #{item.level,jdbcType=VARCHAR}, #{item.major,jdbcType=VARCHAR}, #{item.name,jdbcType=VARCHAR}, #{item.nation,jdbcType=VARCHAR}, #{item.secondaryCollege,jdbcType=VARCHAR}, #{item.operator,jdbcType=VARCHAR}, #{item.sex,jdbcType=VARCHAR}, 0, #{item.accountAddress,jdbcType=VARCHAR}, #{item.nativePlace,jdbcType=VARCHAR}, #{item.originalPlace,jdbcType=VARCHAR}, #{item.usedName,jdbcType=VARCHAR}, #{item.pictrue,jdbcType=VARCHAR}, #{item.joinPartyDate,jdbcType=VARCHAR}, #{item.politicalStatus,jdbcType=VARCHAR}, #{item.telNum,jdbcType=VARCHAR}, #{item.isRegistry,jdbcType=TINYINT}, #{item.graduateSchool,jdbcType=VARCHAR}, now(), now() ) </foreach> </insert> 代码阐明：
底层的mapper是通过逆向工程来生成的，批量插入如下，是拼接成相似：insert into tb_enroll_student（）values （），（）…….（）;
这样的毛病是，数据库个别有一个默认的设置，就是每次sql操作的数据不能超过4M。这样插入，数据多的时候，数据库会报错Packet for query is too large (6071393 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.，尽管咱们能够通过
相似 批改 my.ini 加上 max_allowed_packet =67108864，67108864=64M，默认大小4194304 也就是4M
批改实现之后要重启mysql服务，如果通过命令行批改就不必重启mysql服务。
实现本次操作，然而咱们不能保障我的项目单次最大的大小是多少，这样是有弊病的。所以能够思考进行分组导入。
三、分组把list导入Mysql中
同样实用mybatis批量插入，区别是对每次的导入进行分组计算，而后分屡次进行导入：
@Transactional(rollbackFor = Exception.class)public int addFreshStudentsNew2(List<FreshStudentAndStudentModel> list, String schoolNo) { if (list == null || list.isEmpty()) { return 0; } List<StudentEntity> studentEntityList = new LinkedList<>(); List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>(); List<AllusersEntity> allusersEntityList = new LinkedList<>(); for (FreshStudentAndStudentModel freshStudentAndStudentModel : list) { EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity(); StudentEntity studentEntity = new StudentEntity(); BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity); BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity); String operator = TenancyContext.UserID.get(); String studentId = BaseUuidUtils.base58Uuid(); enrollStudentEntity.setId(BaseUuidUtils.base58Uuid()); enrollStudentEntity.setStudentId(studentId); enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard()); enrollStudentEntity.setOperator(operator); studentEntity.setId(studentId); studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard()); studentEntity.setOperator(operator); studentEntityList.add(studentEntity); enrollStudentEntityList.add(enrollStudentEntity); AllusersEntity allusersEntity = new AllusersEntity(); allusersEntity.setId(enrollStudentEntity.getId()); allusersEntity.setUserCode(enrollStudentEntity.getNemtCode()); allusersEntity.setUserName(enrollStudentEntity.getName()); allusersEntity.setSchoolNo(schoolNo); allusersEntity.setTelNum(enrollStudentEntity.getTelNum()); allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //明码设置为考生号 allusersEntityList.add(allusersEntity); } int c = 100; int b = enrollStudentEntityList.size() / c; int d = enrollStudentEntityList.size() % c; int enResult = 0; int stuResult = 0; boolean allResult = false; for (int e = c; e <= c * b; e = e + c) { enResult = enrollStudentDao.insertAll(enrollStudentEntityList.subList(e - c, e)); stuResult = studentDao.insertAll(studentEntityList.subList(e - c, e)); allResult = allusersFacade.insertUserList(allusersEntityList.subList(e - c, e)); } if (d != 0) { enResult = enrollStudentDao.insertAll(enrollStudentEntityList.subList(c * b, enrollStudentEntityList.size())); stuResult = studentDao.insertAll(studentEntityList.subList(c * b, studentEntityList.size())); allResult = allusersFacade.insertUserList(allusersEntityList.subList(c * b, allusersEntityList.size())); } if (enResult > 0 && stuResult > 0 && allResult) { return 10; } return -10;}代码阐明：
这样操作，能够防止下面的谬误，然而分屡次插入，无形中就减少了操作实际，很容易超时。所以这种办法还是不值得提倡的。
再次改良，应用多线程分批导入。
四、多线程分批导入Mysql
仍然应用mybatis的批量导入，不同的是，依据线程数目进行分组，而后再建设多线程池，进行导入。
@Transactional(rollbackFor = Exception.class)public int addFreshStudentsNew(List<FreshStudentAndStudentModel> list, String schoolNo) { if (list == null || list.isEmpty()) { return 0; } List<StudentEntity> studentEntityList = new LinkedList<>(); List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>(); List<AllusersEntity> allusersEntityList = new LinkedList<>(); list.forEach(freshStudentAndStudentModel -> { EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity(); StudentEntity studentEntity = new StudentEntity(); BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity); BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity); String operator = TenancyContext.UserID.get(); String studentId = BaseUuidUtils.base58Uuid(); enrollStudentEntity.setId(BaseUuidUtils.base58Uuid()); enrollStudentEntity.setStudentId(studentId); enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard()); enrollStudentEntity.setOperator(operator); studentEntity.setId(studentId); studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard()); studentEntity.setOperator(operator); studentEntityList.add(studentEntity); enrollStudentEntityList.add(enrollStudentEntity); AllusersEntity allusersEntity = new AllusersEntity(); allusersEntity.setId(enrollStudentEntity.getId()); allusersEntity.setUserCode(enrollStudentEntity.getNemtCode()); allusersEntity.setUserName(enrollStudentEntity.getName()); allusersEntity.setSchoolNo(schoolNo); allusersEntity.setTelNum(enrollStudentEntity.getTelNum()); allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //明码设置为考生号 allusersEntityList.add(allusersEntity); }); int nThreads = 50; int size = enrollStudentEntityList.size(); ExecutorService executorService = Executors.newFixedThreadPool(nThreads); List<Future<Integer>> futures = new ArrayList<Future<Integer>>(nThreads); for (int i = 0; i < nThreads; i++) { final List<EnrollStudentEntity> EnrollStudentEntityImputList = enrollStudentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1)); final List<StudentEntity> studentEntityImportList = studentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1)); final List<AllusersEntity> allusersEntityImportList = allusersEntityList.subList(size / nThreads * i, size / nThreads * (i + 1)); Callable<Integer> task1 = () -> { studentSave.saveStudent(EnrollStudentEntityImputList,studentEntityImportList,allusersEntityImportList); return 1; }; futures.add(executorService.submit(task1)); } executorService.shutdown(); if (!futures.isEmpty() && futures != null) { return 10; } return -10;}代码阐明：
下面是通过利用ExecutorService 建设了固定的线程数，而后依据线程数目进行分组，批量顺次导入。一方面能够缓解数据库的压力，另一个面线程数目多了，肯定水平会进步程序运行的工夫。
毛病就是要看服务器的配置，如果配置好的话就能够开多点线程，配置差的话就开小点。
五、小结
通过应用这个操作真是一直的进步了，我的项目应用技巧也是不错。
加油 多线程哦
近期热文举荐：
1.1,000+ 道 Java面试题及答案整顿(2022最新版)
2.劲爆！Java 协程要来了。。。
3.Spring Boot 2.x 教程，太全了！
4.别再写满屏的爆爆爆炸类了，试试装璜器模式，这才是优雅的形式！！
5.《Java开发手册（嵩山版）》最新公布，速速下载！
感觉不错，别忘了顺手点赞+转发哦！