共计 40622 个字符,预计需要花费 102 分钟才能阅读完成。
作者:卢文双 资深数据库内核研发
序言:
以前对 MySQL 测试框架 MTR 的应用,次要集中于 SQL 正确性验证。近期因为工作须要,深刻理解了 MTR 的方方面面,发现 MTR 的能力不仅限于此,还反对单元测试、压力测试、代码覆盖率测试、内存谬误检测、线程竞争与死锁等性能,因而,本着分享的精力,将其总结成一个系列。
次要内容如下:
- 入门篇:工作机制、编译装置、参数、指令示例、举荐用法、增加 case、常见问题、异样调试
- 进阶篇:高阶用法,包含单元测试、压力测试、代码覆盖率测试、内存谬误检测、线程竞争与死锁
- 源码篇:剖析 MTR 的源码
- 语法篇:单元测试、压力测试、mysqltest 语法、异样调试
因为集体程度无限,所述不免有谬误之处,望雅正。
本文是第二篇进阶篇。
本文首发于 2023-05-01 21:03:44
MTR 系列基于 MySQL 8.0.29 版本(编译状况也在 8.0.32 版本验证过),且次要在 Ubuntu 22.04 X86_64 验证(局部指令也在 Ubuntu 20.04 X86_64、Ubuntu 22.04 ARM64、MacOS M1 做了验证),如有例外,会特地阐明。
前言
上一篇文章《MySQL 测试框架 MTR 系列教程(一):入门篇》介绍了 mtr 的原理、目录构造、参数及常见用法,侧重于 最常见的 SQL 正确性验证 ,但 mtr 能做更多的事件,比方 内存谬误、线程竞争、代码覆盖率、压力测试等,本文次要介绍这些内容,波及的相干工具如下:
- valgrind:用于内存调试、内存透露检测以及性能剖析。
-
Sanitizier:谷歌发动的开源工具集。
- ASAN/AddressSanitizier:查看内存地址相干问题,包含内存透露、开释后应用、反复开释、堆溢出、栈溢出等等问题。
- LSAN/LeakSanitizer:查看内存透露问题。它是集成在 Address Sanitizer 中的一个绝对独立的工具,它工作在查看过程的最初阶段。
- MSAN/MemorySanitizer:查看应用未初始化内存的问题。
- TSAN/ThreadSanitizer:查看线程数据竞争和死锁问题。
- UBSAN/UndefinedBehaviorSanitizer:检测未定义行为(应用空指针、有符号整数溢出等)。
- gcov:代码覆盖率测试。
- gprof:性能剖析工具。
- 单元测试
- 压力测试
本文将逐个介绍对各个工具的反对状况。
补充:
MariaDB 曾经很好的反对了以上工具集:
Compile and Using MariaDB with Sanitizers (ASAN, UBSAN, TSAN, MSAN) – MariaDB Knowledge Base – MariaDB Knowledge Base”)
Compiling MariaDB for Debugging – MariaDB Knowledge Base(反对 valgrind)
MySQL 编译选项
首先阐明一下与本文相干的 MySQL 编译选项:
-
-DCMAKE_BUILD_TYPE=type
The type of build to produce:
RelWithDebInfo
: default value。Enable optimizations and generate debugging information. This is the default MySQL build type.Release
: Enable optimizations but omit debugging information to reduce the build size. This build type was added in MySQL 8.0.13 (MySQL 5.7 is not supported).Debug
: Disable optimizations and generate debugging information. This build type is also used if the WITH_DEBUG option is enabled. That is, -DWITH_DEBUG=1 has the same effect as -DCMAKE_BUILD_TYPE=Debug.
-
-DWITH_DEBUG=bool
Whether to include debugging support. The default is
OFF
. -
-DWITH_ASAN=bool
Whether to enable the AddressSanitizer, for compilers that support it. The default is off.
-
-DWITH_ASAN_SCOPE=bool
Whether to enable the AddressSanitizer
-fsanitize-address-use-after-scope
Clang flag for use-after-scope detection. The default is off. To use this option,-DWITH_ASAN
must also be enabled. -
-DWITH_LSAN=bool
Whether to run LeakSanitizer, without AddressSanitizer. The default is
OFF
.This option was added in MySQL 8.0.16.
-
-DWITH_MSAN=bool
Whether to enable MemorySanitizer, for compilers that support it. The default is off.
For this option to have an effect if enabled, all libraries linked to MySQL must also have been compiled with the option enabled.
-
-DWITH_TSAN=bool
Whether to enable the ThreadSanitizer, for compilers that support it. The default is off.
-
-DWITH_UBSAN=bool
Whether to enable the Undefined Behavior Sanitizer, for compilers that support it. The default is off.
-
-DWITH_UNIT_TESTS={ON|OFF}
If enabled, compile MySQL with unit tests. The default is ON unless the server is not being compiled.
-
-DWITH_VALGRIND=bool
Whether to compile in the Valgrind header files, which exposes the Valgrind API to MySQL code. The default is
OFF
.To generate a Valgrind-aware debug build, -DWITH_VALGRIND=1 normally is combined with -DWITH_DEBUG=1. See Building Debug Configurations.
-
-DENABLE_GCOV=bool
Whether to include
gcov
support (Linux only). -
-DENABLE_GPROF=bool
Whether to enable
gprof
(optimized Linux builds only). The default isOFF
. -
-DWITH_TEST_TRACE_PLUGIN=bool
Whether to build the test protocol trace client plugin (see Using the Test Protocol Trace Plugin). By default, this option is disabled. Enabling this option has no effect unless the WITH_CLIENT_PROTOCOL_TRACING option is enabled. If MySQL is configured with both options enabled, the
libmysqlclient
client library is built with the test protocol trace plugin built in, and all the standard MySQL clients load the plugin. However, even when the test plugin is enabled, it has no effect by default. Control over the plugin is afforded using environment variables; see Using the Test Protocol Trace Plugin.Note
Do not enable the WITH_TEST_TRACE_PLUGIN option if you want to use your own protocol trace plugins because only one such plugin can be loaded at a time and an error occurs for attempts to load a second one. If you have already built MySQL with the test protocol trace plugin enabled to see how it works, you must rebuild MySQL without it before you can use your own plugins.
For information about writing trace plugins, see Writing Protocol Trace Plugins.
-
-DWITH_CLIENT_PROTOCOL_TRACING=bool
Whether to build the client-side protocol tracing framework into the client library. By default, this option is enabled.
For information about writing protocol trace client plugins, see Writing Protocol Trace Plugins.
See also the WITH_TEST_TRACE_PLUGIN option.
-
-DWITH_KEYRING_TEST=bool
Whether to build the test program that accompanies the
keyring_file
plugin. The default isOFF
. Test file source code is located in theplugin/keyring/keyring-test
directory. -
-DWITH_NDB_TEST={ON|OFF}
If enabled, include a set of NDB API test programs. The default is OFF.
详见:MySQL :: MySQL 8.0 Reference Manual :: 2.8.7 MySQL Source-Configuration Options
以下是各组件或测试类型的具体介绍。
valgrind
简介
valgrind 是一个工具集,次要集成了:
- memcheck:内存谬误检测器。
- cachegrind:缓存和分支预测分析器。
- callgrind:可生成缓存分析器的调用图。
- helgrind:线程谬误检测器。
- DRD:也是线程谬误检测器。
- massif:堆分析器,它能够帮忙程序应用更少的内存。
- DHAT:一种不同类型的堆分析器。应用它能够理解块寿命,块利用率和布局效率低下的问题。
选项
mtr 提供的 valgrind 选项如下:
Options for valgrind
callgrind Instruct valgrind to use callgrind.
helgrind Instruct valgrind to use helgrind.
valgrind Run the "mysqltest" and "mysqld" executables using
valgrind with default options.
valgrind-all Synonym for --valgrind.
valgrind-clients Run clients started by .test files with valgrind.
valgrind-mysqld Run the "mysqld" executable with valgrind.
valgrind-mysqltest Run the "mysqltest" and "mysql_client_test" executable
with valgrind.
valgrind-option=ARGS Option to give valgrind, replaces default option(s), can
be specified more then once.
valgrind-options=ARGS Deprecated, use --valgrind-option.
valgrind-path=<EXE> Path to the valgrind executable.
从代码看:
- 反对的工具不仅限于 callgrind、helgrind,还反对 memcheck、massif。
- 只有启用
--valgrind
或--valgrind-mysqltest
选项,才会用到mysql_client_test
。
应用办法
编译选项:-DWITH_DEBUG=1 -DWITH_VALGRIND=1
应用倡议:
1、举荐指令可参考 mysql-test/collections/
目录下的文件 default.daily-valgrind
、default.push-valgrind
、default.weekly-valgrind
。
2、据我实测,如需测试 valgrind 所有性能,需在原有指令根底上增加 --valgrind
选项。比方:
# 官网 collections 中的示例指令:perl mysql-test-run.pl --timer --force --skip-rpl --comment=all_default_valgrind --vardir=var-all_default_valgrind --skip-ndb
# 增加 --valgrind
perl mysql-test-run.pl --timer --force --skip-rpl --comment=all_default_valgrind --vardir=var-all_default_valgrind --skip-ndb --valgrind
3、在同时启用 ASAN 和 valgrind 的状况下,并在运行 mtr 时增加 --valgrind
选项,mtr 会因 valgrind memcheck 与 asan 抵触而 crash,因而,valgrind 与 asan 不倡议同时启用。
Sanitizier
Sanitizers 是谷歌发动的开源工具集,曾经成为动态和动静代码剖析的利器,能够查看 内存谬误、未初始化的读取、线程平安和未定义的行为等 相干的问题。
长处:与同类型剖析工具相比,Sanitizers 带来的性能损失通常要小得多,而且往往提供的信息更具体。
毛病:代码(可能还有工具链的一部分)须要应用附加的标记从新编译。
Sanitizers 包含如下组件:
-
AddressSanitizer/ASAN
查看内存地址相干问题,包含内存透露、开释后应用、反复开释、堆溢出、栈溢出等问题。
通过 编译插桩(CTI),可能发现此堆 / 栈 / 全局变量读写溢出,内存泄露等问题,并将信息间接打印到日志中。
ASAN 是一个疾速的内存谬误检测工具。它十分快,只拖慢程序两倍左右(比起 Valgrind 快多了)。
它包含一个编译器 instrumentation 模块和一个提供
malloc()/free()
代替项的运行时库。 -
LeakSanitizer/LSAN
查看内存透露问题。它是 集成在 Address Sanitizer 中 的一个绝对独立的工具,它工作在查看过程的最初阶段。
-
UndefinedBehaviorSanitizer/UBSAN
检测未定义行为(应用空指针、有符号整数溢出等)。
-
ThreadSanitizer/TSAN
查看线程数据竞争和死锁问题。
-
MemorySanitizer/MSAN
查看应用未初始化内存问题。
- 内核 Sanitizer包含 KASAN 和KMSAN
Sanitizers 我的项目本是 LLVM 我的项目的一部分,但 GNU 也将该系列工具退出到了自家的 GCC 编译器中(clang 当然也反对)。
- GCC 4.8 版本开始反对 Address Sanitizer和 Thread Sanitizer。
- GCC 4.9 版本开始反对 Leak Sanitizer 和 UndefinedBehaviorSanitizer。
ASAN
简介
ASAN/AddressSanitizer 能检测很多种内存谬误,次要蕴含如下类别:
- Out-of-bounds accesses to heap, stack and globals
- Use-after-free
-
Use-after-return (clang flag
-fsanitize-address-use-after-return=(never|runtime|always)
default:runtime
)- Enable with:
ASAN_OPTIONS=detect_stack_use_after_return=1
(already enabled on Linux). - Disable with:
ASAN_OPTIONS=detect_stack_use_after_return=0
.
- Enable with:
- Use-after-scope (clang flag
-fsanitize-address-use-after-scope
) - Double-free, invalid free
- Memory leaks (experimental)
更具体的示例 case:参考 https://learn.microsoft.com/zh-cn/cpp/sanitizers/asan-error-examples?view=msvc-170
- alloc-dealloc-mismatch
- allocation-size-too-big
- calloc-overflow
- container-overflow
- double-free
- dynamic-stack-buffer-overflow
- global-buffer-overflow
- heap-buffer-overflow
- heap-use-after-free
- invalid-allocation-alignment
- memcpy-param-overlap
- new-delete-type-mismatch
- stack-buffer-overflow
- stack-buffer-underflow
- stack-use-after-return
- stack-use-after-scope
- strncat-param-overlap
- use-after-poison
性能影响:应用 ASAN 后,性能会升高 2 倍。
应用办法
装置:有一个独自的动静库libasan6.so
,会随 gcc 装置。
编译选项:-DWITH_DEBUG=1 -DWITH_ASAN=1
,可抉择启用-DWITH_ASAN_SCOPE=1
验证版本:8.0.29
MTR 选项:--sanitize
应用倡议:
1、ASAN 功能强大,相较于 valgrind,对性能影响小很多,倡议作为次要的内存检测工具。
2、因为 mtr 须要用到 /usr/bin/perl
,因而,有可能呈现 perl 本身某些函数的内存透露问题被 Leak Sanitizer 检测到,导致 mtr 测试失败,此时,将问题函数增加到lsan.supp
文件即可解决。比方 Ubuntu 22.04 perl v5.34.0 会遇到内存透露,同样的,Ubuntu 20.04 perl v5.30.0 就无该问题。
mysql-test/asan.supp 示例
# 语法
interceptor_via_fun:NameOfCFunctionToSuppress
interceptor_via_fun:-[ClassName objCMethodToSuppress:]
interceptor_via_lib:NameOfTheLibraryToSuppress
# 示例
interceptor_via_fun:Perl_safesyscalloc
interceptor_via_fun:Perl_safesysmalloc
参考:
- asan 文档
- gcc/asan_suppressions.cpp at master · gcc-mirror/gcc · GitHub
mysql-test/lsan.supp 示例
# LSAN suppressions for gcc/clang
leak:Perl_safesyscalloc
leak:Perl_safesysmalloc
leak:Perl_safesysrealloc
leak:Perl_savesharedpv
leak:Perl_Slab_Alloc
leak:Perl_newUNOP_AUX
leak:Perl_newSTATEOP
leak:Perl_pmruntime
leak:/usr/bin/perl
leak:/lib64/libperl.so.*
leak:/bin/bash
leak:/usr/bin/zip
# OpenLDAP bug 9081
# Fixed in 2.4.49, we build with 2.4.48
leak:ldap_initialize
# sasl_client_init will load all available plugins with _sasl_load_plugins().
# It seems some of the SASL plugin have leaks.
# Both LSAN and Valgrind report leaks.
leak:sasl_client_add_plugin
该内容来源于源码文件,可见官网通晓 Perl 高版本的内存透露问题,以此形式来疏忽。
指令示例
在Ubuntu 22.04 X86_64
运行:
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --vardir=var-rpl --suite=rpl --sanitize
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize
LSAN
简介
LSAN/LeakSanitizer 用于内存透露检测。
性能影响:应用 LSAN 后,除了执行的最初阶段会有一个内存透露检测之外,简直没有性能开销。
应用办法
装置:有一个独自的动静库liblsan0.so
,会随 gcc 装置。
编译选项:-DWITH_DEBUG=1 -DWITH_LSAN=1
验证版本:8.0.29
MTR 选项:--sanitize
应用倡议:
- 因为 ASAN 集成了 LSAN,因而,只有不应用 ASAN、仅应用 LSAN 的状况下才须要设置该选项。
lsan.supp
格局见「ASAN」大节。
指令示例
只有编译时启用 ASAN 或 LSAN,在运行时增加--sanitize
选项即可。
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize
UBSAN
简介
UBSAN/UndefinedBehaviorSanitizer 是针对未定义行为的检测器,速度很快。
UBSAN 须要在编译时批改程序,以捕捉程序执行期间的各种未定义行为。比方:
- 数组下标越界:Array subscript out of bounds, where the bounds can be statically determined
- 位移位超过数据类型边界:Bitwise shifts that are out of bounds for their data type
- 解除对未对齐指针或空指针的关联:Dereferencing misaligned or null pointers
- 有符号整数溢出:Signed integer overflow
- 浮点数类型转换导致的溢出:Conversion to, from, or between floating-point types which would overflow the destination
更多行为详见:
UndefinedBehaviorSanitizer — Clang 17.0.0git documentation (llvm.org)”)
性能影响:UBSAN 的运行时老本很小,对地址空间布局或 ABI 没有影响。
应用办法
装置:有一个独自的动静库libubsan1.so
,会随 gcc 装置。
编译选项:-DWITH_DEBUG=1 -DWITH_UBSAN=1
验证版本:8.0.29
MTR 选项:--sanitize
应用倡议:
- 与 ASAN、TSAN、gcov、gprof 都兼容,可一起启用。
- 若想要某些 case 跳过 UBSAN 的查看,可援用
include/not_ubsan.inc
。目前只有如下 case 会跳过 UBSAN:
./t/innodb_tmp_table_heap_to_disk.test
./t/ssl-big.test
./t/count_distinct3.test
./t/multi_update2.test
./t/ds_mrr-big.test
./suite/gis/t/gis_not_ubsan.test
./suite/binlog_gtid/t/binlog_warning_same_server_id.test
指令示例
只有编译时启用 UBSAN,在运行时增加--sanitize
选项即可。
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize
TSAN
简介
TSAN/ThreadSanitizer 是用于检测数据竞争和线程死锁的工具。
性能影响:引入 TSAN 后,会升高 5-15 倍性能,同时,内存占用率会晋升 5-10 倍。
应用办法
装置:有一个独自的动静库libtsan0.so
,会随 gcc 装置。
编译选项:-DWITH_DEBUG=1 -DWITH_TSAN=1
验证版本:8.0.29、8.0.32
MTR 选项:--sanitize
应用倡议:
1、TSAN 与 ASAN 不兼容(一起应用 cmake 会报错"No mysys timer support detected"
),但TSAN 与 UBSAN、VALGRIND 兼容。
2、对 TSAN 的反对是实验性的,尚不成熟,不倡议应用。
-- Performing Test HAVE_SANITIZE_SCOPE
-- Performing Test HAVE_SANITIZE_SCOPE - Success
CMake Warning at CMakeLists.txt:1101 (MESSAGE):
Thread sanitizer support is currently experimental.
-- Performing Test C_LD_LLD_RESULT
-- Performing Test C_LD_LLD_RESULT - Failed
-- Performing Test CXX_LD_LLD_RESULT
-- Performing Test CXX_LD_LLD_RESULT - Failed
-- Performing Test C_LD_GOLD_RESULT
-- Performing Test C_LD_GOLD_RESULT - Failed
-- Performing Test CXX_LD_GOLD_RESULT
-- Performing Test CXX_LD_GOLD_RESULT - Failed
-- Local boost dir /data/work/mysql/boost_1_77_0
-- Found /data/work/mysql/boost_1_77_0/boost/version.hpp
-- BOOST_VERSION_NUMBER is #define BOOST_VERSION 107700
-- BOOST_INCLUDE_DIR /data/work/mysql/boost_1_77_0
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Could NOT find Threads (missing: Threads_FOUND)
......
-- Looking for timer_create # 因为 timer_create/timer_settime 函数的确存在,尝试调整过 cmake,后续会报一系列谬误,该问题不太好调。-- Looking for timer_create - not found
-- Looking for timer_settime
-- Looking for timer_settime - not found
-- Looking for kqueue
-- Looking for kqueue - not found
-- Performing Test HAVE_SETNS
-- Performing Test HAVE_SETNS - Failed
-- Looking for EVFILT_TIMER
-- Looking for EVFILT_TIMER - not found
CMake Error at configure.cmake:334 (MESSAGE):
No mysys timer support detected!
Call Stack (most recent call first):
CMakeLists.txt:1487 (INCLUDE)
3、如果某些数据竞争或死锁状况是合乎预期的,能够通过 mysql-test/tsan.supp
跳过。
mysql-test/tsan.supp 示例
#
# Blacklist for Thread Sanitizer.
# Thread Sanitizer can be enabled with -DWITH_TSAN=1
#
# Suppression syntax is documented here:
# https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
#
race:innobase
race:client/dump/
deadlock:client/dump/
race:perfschema
race:plugin_vars_free_values
race:log_builtins_filter_run
race:MY_LOCALE_ERRMSGS::destroy
race:get_one_variable_ext
race:mysql_set_character_set_with_default_collation
race:ngs::Scheduler_dynamic::wait_if_idle_then_delete_worker
race:ngs::Socket_events::break_loop
deadlock:find_sys_var_ex
deadlock:Persisted_variables_cache::lock
signal:my_print_stacktrace
指令示例
只有编译时启用 TSAN,在运行时增加--sanitize
选项即可。
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize
存在问题
测试时,在 install database 阶段,线程之间就会有大量 data race,报错示例如下:
// 报错信息位于 mysql-test/var-main-tsan/log/bootstrap.log
==================
WARNING: ThreadSanitizer: data race (pid=65314)
Read of size 4 at 0x555ae20c13b0 by thread T4:
#0 fil_validate_skip /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:1953 (mysqld+0x541018e)
#1 fil_aio_wait(unsigned long) /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:8234 (mysqld+0x54109ca)
#2 io_handler_thread /data/work/mysql/mysql-server/storage/innobase/srv/srv0start.cc:279 (mysqld+0x5143b04)
......
#12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211 (mysqld+0x5159b59)
#13 <null> <null> (libstdc++.so.6+0xdc2b2)
Previous write of size 4 at 0x555ae20c13b0 by thread T3:
#0 fil_validate_skip /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:1953 (mysqld+0x54101a7)
#1 fil_aio_wait(unsigned long) /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:8234 (mysqld+0x54109ca)
#2 io_handler_thread /data/work/mysql/mysql-server/storage/innobase/srv/srv0start.cc:279 (mysqld+0x5143b04)
......
#12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211 (mysqld+0x5159b59)
#13 <null> <null> (libstdc++.so.6+0xdc2b2)
Location is global 'fil_validate_skip()::fil_validate_count' of size 4 at 0x555ae20c13b0 (mysqld+0x000007a5c3b0)
Thread T4 (tid=65320, running) created by thread T1 at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xdc388)
......
#8 handle_bootstrap /data/work/mysql/mysql-server/sql/bootstrap.cc:327 (mysqld+0x387778f)
#9 pfs_spawn_thread /data/work/mysql/mysql-server/storage/perfschema/pfs.cc:2942 (mysqld+0x56751fb)
Thread T3 (tid=65319, running) created by thread T1 at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xdc388)
......
#8 handle_bootstrap /data/work/mysql/mysql-server/sql/bootstrap.cc:327 (mysqld+0x387778f)
#9 pfs_spawn_thread /data/work/mysql/mysql-server/storage/perfschema/pfs.cc:2942 (mysqld+0x56751fb)
SUMMARY: ThreadSanitizer: data race /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:1953 in fil_validate_skip
==================
......
install database 阶段,相似的报错有 200 多个。尽管能够通过 tsan.supp
文件跳过,但毕竟报错波及较多函数,若全副跳过,可能会影响对失常状况下数据竞争的判断。因而, 集体暂不倡议应用。
MSAN
简介
MSAN/MemorySanitizer 用于检测对未初始化内存的读取(uninitialized reads)问题。
性能影响:引入 MSAN 后,性能会升高 3 倍。
应用办法
编译选项:-DWITH_DEBUG=1 -DWITH_MSAN=1
验证版本:8.0.29
MTR 选项:--sanitize
应用倡议:
- 对 MSAN 的反对是实验性的,尚不成熟,且与 ASAN 不兼容,思考到 ASAN 的弱小,因而,倡议应用 ASAN,不倡议应用 MSAN。
CMake Warning at CMakeLists.txt:1080 (MESSAGE):
Memory sanitizer support is currently experimental.
CMake Error at CMakeLists.txt:1107 (MESSAGE):
Cannot use AddressSanitizer and MemorySanitizer together
指令示例
只有编译时启用 MSAN,在运行时增加--sanitize
选项即可。
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize
代码覆盖率测试
简介
gcov 用于剖析代码覆盖率,gprof 用于剖析 gcov 生成的统计数据,二者个别一起应用。
gprof 只反对 linux 操作系统,不反对 MacOS。
官网手册:
Gcov (Using the GNU Compiler Collection (GCC)))”)
gprof(1) – Linux manual page (man7.org) – Linux manual page (man7.org)”)
超级不便的 Linux 自带性能剖析工具!gprof 介绍、装置、应用及实际 – 知乎 (zhihu.com)”)
应用办法
编译选项:-DWITH_DEBUG=1 -DENABLE_GCOV=1 -DENABLE_GPROF=1
应用倡议:
- 与 ASAN、UBSAN 兼容,倡议与 ASAN、UBSAN 同时启用(未验证与 valgrind、MSAN、TSAN 的兼容性)。
-
不能在
make install
的装置目录测试,必须在 执行编译的源码目录 测试。源码根目录 /build-debug/mysql-test/mysql-test-run.pl
(自己的编译目录是 build-debug)只是封装了一层对源码根目录 /mysql-test/mysql-test-run.pl
的调用:
[apps@node6 mysql-test]$ pwd
/home/apps/mtr/mysql-oracle/mysql-8.0.26/build-debug/mysql-test
[apps@node6 mysql-test]$ cat mysql-test-run.pl
#!/usr/bin/perl
# Call mtr in out-of-source build
$ENV{MTR_BINDIR} = '/home/apps/mtr/mysql-oracle/mysql-8.0.26/build-debug';
chdir('/home/apps/mtr/mysql-oracle/mysql-8.0.26/mysql-test');
exit(system($^X, '/home/apps/mtr/mysql-oracle/mysql-8.0.26/mysql-test/mysql-test-run.pl', @ARGV) >> 8);
留神:
- gcov、gprof 运行须要较大内存,尤其是开启
-big-test
的状况下。 - 官网
collections/
中没有 gcov 的举荐用法。 mysql-test/README.gcov
文件的最初批改日期是 2006 年,已过期,没有参考价值。
示例
间接在装置目录(make install
)执行测试,报错:
wslu@ubuntu:/data/work/mysql/mysql80-install.bak_asan_ubsan_gcov_gprof/mysql-test$ ./mtr --gcov
Logging: ./mtr --gcov
MySQL Version 8.0.29
mysql-test-run: *** ERROR: Coverage test needs the source - please use source dist
在 执行编译的目录(比方 console-build-debug/mysql-test
)执行:
wslu@ubuntu:/data/work/mysql/mysql-server/console-build-debug/mysql-test$ ./mtr --gcov -big-test
Logging: /data/work/mysql/mysql-server/mysql-test/mysql-test-run.pl --gcov -big-test
MySQL Version 8.0.29
Checking supported features
- Binaries are debug compiled
Purging gcov information from '/data/work/mysql/mysql-server'...
Using suite(s): auth_sec,binlog,binlog_gtid,binlog_nogtid,clone,collations,component_keyring_file,connection_control,encryption,federated,funcs_2,gcol,gis,information_schema,innodb,innodb_fts,innodb_gis,innodb_undo,innodb_zip,interactive_utilities,json,main,opt_trace,parts,perfschema,query_rewrite_plugins,rpl,rpl_gtid,rpl_nogtid,secondary_engine,service_status_var_registration,service_sys_var_registration,service_udf_registration,sys_vars,sysschema,test_service_sql_api,test_services,x
Collecting tests
- Adding combinations for binlog
- Adding combinations for binlog_gtid
- Adding combinations for binlog_nogtid
- Adding combinations for rpl
- Adding combinations for rpl_gtid
- Adding combinations for rpl_nogtid
Checking leftover processes
Removing old var directory
Creating var directory '/data/work/mysql/mysql-server/console-build-debug/mysql-test/var'
Installing system database
Using parallel: 1
==============================================================================
TEST NAME RESULT TIME (ms) COMMENT
------------------------------------------------------------------------------
[0%] binlog_gtid.binlog_xa_select_gtid_executed_explicitly_crash [disabled] Bug#28588717 Fails both on FreeBSD and other platforms
[0%] binlog_nogtid.binlog_gtid_next_xa [disabled] BUG#33650776 Failure of XA COMMIT of prepared txn, can result in txn rollback
[0%] sys_vars.innodb_log_writer_threads_basic [disabled] Bug#32129814 SYS_VARS.INNODB_LOG_WRITER_THREADS_BASIC TIMES OUT SPORADICALLY ON PB2
[0%] sysschema.v_wait_classes_global_by_avg_latency [disabled] BUG#21550054 Test fails too often.
[0%] binlog_gtid.binlog_gtid_mix_ok_packet_all_gtids 'mix' [pass] 770
[0%] binlog_gtid.binlog_gtid_mix_response_packet 'mix' [pass] 6474
[0%] binlog_gtid.binlog_xa_trx_gtid_response_packet 'mix' [pass] 683
......
[0%] binlog_gtid.binlog_gtid_errors 'mix' [pass] 1583
......
如果想执行测试后剖析 gmon.out
,则可增加 -gprof
参数(仅反对 linux):
wslu@ubuntu:/data/work/mysql/mysql-server/console-build-debug/mysql-test$ ./mtr --gcov -gprof -big-test
那么,在 gcov 执行实现后,mtr 就会主动调用 gprof 解析 gmon.out
文件。
存在问题
但在 CentOS 7.6(云服务器 4C8G SSD)实测时遇到问题——gprof 解析 gmon.out 时特地耗时,尽管该过程 CPU 占用率 100%,看起来还在运行,但并无任何输入。
比方,在编译的源码目录中执行:
➜ mysql-test git:(heads/mysql-8.0.26) ✗ ps -axf | grep mysql
30604 ? SN 0:00 \_ perl mysql-test-run.pl --parallel=4 --timer --debug-server --force --testcase-timeout=180 --suite-timeout=1800 --comment=all-default-debug --vardir=var-all-default --skip-combinations --unit-tests-report --no-skip --exclude-platform=windows --skip-ndb --max-test-fail=0 --suite=rpl -gcov -gprof
30611 ? SN 0:30 \_ /usr/bin/perl /home/wslu/work/mysql/mysql-server/mysql-test/mysql-test-run.pl --parallel=4 --timer --debug-server --force --testcase-timeout=180 --suite-timeout=1800 --comment=all-default-debug --vardir=var-all-default --skip-combinations --unit-tests-report --no-skip --exclude-platform=windows --skip-ndb --max-test-fail=0 --suite=rpl -gcov -gprof
32759 ? SN 0:44 \_ /usr/bin/perl /home/wslu/work/mysql/mysql-server/mysql-test/mysql-test-run.pl --parallel=4 --timer --debug-server --force --testcase-timeout=180 --suite-timeout=1800 --comment=all-default-debug --vardir=var-all-default --skip-combinations --unit-tests-report --no-skip --exclude-platform=windows --skip-ndb --max-test-fail=0 --suite=rpl -gcov -gprof
2829 ? SN 0:00 | \_ sh -c gprof /home/wslu/work/mysql/mysql-server/build-debug/runtime_output_directory/mysqld /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out 2 > /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.err > /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.msg
2830 ? RN 7:07 | \_ gprof /home/wslu/work/mysql/mysql-server/build-debug/runtime_output_directory/mysqld /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out 2
gmon.out 文件只有 61MB,但 gprof 在解析 gmon.out
时,长达 23 小时无任何输入。
➜ mysql-test git:(heads/mysql-8.0.26) ✗ ls /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out -lh
-rw-r--r-- 1 wslu wslu 61M Mar 27 20:21 /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out
➜ mysql-test git:(heads/mysql-8.0.26) ✗ ll /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.err
-rw-r--r-- 1 wslu wslu 0 Mar 28 09:23 /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.err
➜ mysql-test git:(heads/mysql-8.0.26) ✗ ll /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.msg
-rw-r--r-- 1 wslu wslu 0 Mar 28 09:23 /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.msg
单元测试
简介
MySQL 应用 TAP(Test Anything Protocol)和 Google Test Framework 来实现单元测试。
MyTAP
TAP 是 Perl 与测试模块之间所应用的简略的基于文本的接口,次要用于开发 Perl 和 PHP 模块。示例如下:
TAP version 13
ok 1 - testNewArrayIsEmpty(ArrayTest)
ok 2 - testArrayContainsAnElement(ArrayTest)
not ok 3 - Failure: testFailure(FailureErrorTest)
---
message: 'Failed asserting that <integer:2> matches expected value <integer:1>.'
severity: fail
data:
got: 2
expected: 1
...
not ok 4 - Error: testError(FailureErrorTest)
1..4
为了实现 C/C++ 的单元测试,MySQL 开发了一个用于生成 TAP 文本的库libmytap.a
,源码门路位于unittest/mytap/
。
Google Test Framework
Google Test Framework,与 MyTAP 相似,也是一个单元测试框架,但提供了更丰盛的性能:
- A rich set of predicates
- User-defined predicates and assertions
- Automatic test registration
- Nice error reporting when a predicate fails (with line number, expected and actual values, and additional comments)
- Test fixtures, and setup/teardown logic
- Death tests
- Disabled tests
- Test filtering and shuffling
参考:
-
MySQL: Creating and Executing Unit Tests
- Home – Test Anything Protocol
- GoogleTest User’s Guide | GoogleTest
应用办法
编译选项:-DWITH_DEBUG=1 -DWITH_UNIT_TESTS={ON|OFF}
,默认是 ON
。
执行门路:必须在编译的源码目录中执行。
应用办法:
- 编译后,在 执行编译(cmake)的目录 执行
make test
或make test-unit
指令,尽管按手册形容两个指令都能实现单元测试成果,但实测make test-unit
会输入更具体的信息,因而,倡议应用make test-unit
。 - 编译后,在
编译目录 /mysql-test
中执行 mtr 指令时,增加--unit-tests-report
选项。
注意事项:
若启用了 ASAN:
- 间接在编译目录执行
make test-unit
,可能会因 ASAN 检测到单元测试代码有内存谬误(RUN_ALL_TESTS()
的子函数)而导致 case 失败。 -
通过 mtr 指令来运行单元测试时,也可能会遇到 ASAN 检测到内存谬误或内存透露,即便按如下形式批改
.supp
文件,也无奈跳过:- 若是 AddressSanitizer 领域中的谬误,比方下表中的 heap-buffer-overflow,在
asan.supp
文件增加interceptor_via_fun:RUN_ALL_TESTS
,无奈跳过该谬误。 - 同理,如果是 ASAN 中的
LeakSanitizer
检测到内存透露,在lsan.supp
文件增加leak:RUN_ALL_TESTS
,无奈跳过该谬误。
- 若是 AddressSanitizer 领域中的谬误,比方下表中的 heap-buffer-overflow,在
==228225==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x620000098e90 at pc 0x5570c34c26fb bp 0x7ffe1d0d0590 sp 0x7ffe1d0d0580
READ of size 2 at 0x620000098e90 thread T0
#0 0x5570c34c26fa in modify_all_zh_pages /data/work/mysql/mysql-server/strings/ctype-uca.cc:4178
#1 0x5570c34c4d89 in init_weight_level /data/work/mysql/mysql-server/strings/ctype-uca.cc:4287
......
#16 0x5570c36724b8 in testing::UnitTest::Run() /data/work/mysql/mysql-server/extra/googletest/googletest-release-1.11.0/googletest[表情]c/gtest.cc:5438
#17 0x5570c3285445 in RUN_ALL_TESTS() /data/work/mysql/mysql-server/extra/googletest/googletest-release-1.11.0/googletest/include/gtest/gtest.h:2490
#18 0x5570c3284f94 in main /data/work/mysql/mysql-server/unittest/gunit/gunit_test_main.cc:150
#19 0x7f680a423d8f in __libc_start_call_main ../sysdeps/nptl[表情]bc_start_call_main.h:58
#20 0x7f680a423e3f in __libc_start_main_impl ..[表情]u[表情]bc-start.c:392
#21 0x5570c2682f34 in _start (/data/work/mysql/mysql-server/console-build-debug/runtime_output_directory/merge_small_tests-t+0x26a9f34)
综上,运行单元测试时,不倡议同时启用 ASAN。
指令示例
make test 示例
➜ console-build-debug git:(my_learn_8.0.29) ✗ make test
Running tests...
Test project /Users/wslu/work/mysql/mysql-server-8.0/console-build-debug
Start 1: hp_test1
1/223 Test #1: hp_test1 ................................................... Passed 0.49 sec
Start 2: hp_test2
2/223 Test #2: hp_test2 ................................................... Passed 0.74 sec
Start 3: pfs_instr_class
3/223 Test #3: pfs_instr_class ............................................ Passed 0.54 sec
......
Start 206: routertest_component_rest_metadata_cache
206/223 Test #206: routertest_component_rest_metadata_cache ...................***Failed 13.37 sec
Start 207: routertest_component_rest_mock_server
207/223 Test #207: routertest_component_rest_mock_server ...................... Passed 23.60 sec
Start 208: routertest_component_rest_router
208/223 Test #208: routertest_component_rest_router ........................... Passed 3.87 sec
Start 209: routertest_component_rest_routing
209/223 Test #209: routertest_component_rest_routing ..........................***Failed 145.64 sec
Start 210: routertest_component_rest_connection_pool
210/223 Test #210: routertest_component_rest_connection_pool .................. Passed 5.71 sec
Start 211: routertest_component_router_configuration_errors
211/223 Test #211: routertest_component_router_configuration_errors ........... Passed 5.15 sec
Start 212: routertest_component_routing
212/223 Test #212: routertest_component_routing ...............................***Failed 39.04 sec
Start 213: routertest_component_routing_connection
213/223 Test #213: routertest_component_routing_connection ....................***Failed 117.32 sec
Start 214: routertest_component_routing_strategy
214/223 Test #214: routertest_component_routing_strategy ...................... Passed 83.88 sec
Start 215: routertest_component_sd_notify
215/223 Test #215: routertest_component_sd_notify ............................. Passed 22.63 sec
Start 216: routertest_component_shutdown
216/223 Test #216: routertest_component_shutdown .............................. Passed 4.86 sec
Start 217: routertest_component_state_file
217/223 Test #217: routertest_component_state_file ............................ Passed 22.92 sec
Start 218: routertest_component_user_option
218/223 Test #218: routertest_component_user_option ........................... Passed 0.74 sec
Start 219: routertest_component_metadata_http_auth_backend
219/223 Test #219: routertest_component_metadata_http_auth_backend ............***Failed 76.95 sec
Start 220: routertest_component_socket_close
220/223 Test #220: routertest_component_socket_close .......................... Passed 162.52 sec
Start 221: routertest_component_routing_splicer
221/223 Test #221: routertest_component_routing_splicer ....................... Passed 287.30 sec
Start 222: routertest_component_mock_server
222/223 Test #222: routertest_component_mock_server ........................... Passed 20.44 sec
Start 223: routertest_integration_routing_reuse
223/223 Test #223: routertest_integration_routing_reuse ....................... Passed 440.36 sec
97% tests passed, 6 tests failed out of 223
Total Test time (real) = 2940.80 sec
The following tests FAILED:
14 - merge_small_tests (Failed)
206 - routertest_component_rest_metadata_cache (Failed)
209 - routertest_component_rest_routing (Failed)
212 - routertest_component_routing (Failed)
213 - routertest_component_routing_connection (Failed)
219 - routertest_component_metadata_http_auth_backend (Failed)
Errors while running CTest
Output from these tests are in: /Users/wslu/work/mysql/mysql-server-8.0/console-build-debug/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [test] Error 8
make test-unit 示例
wslu@ubuntu:/data/work/mysql/mysql-server/console-build-debug$ head -n 100 /tmp/maketestunit.out
Test project /data/work/mysql/mysql-server/console-build-debug
Start 14: merge_small_tests
Start 15: merge_large_tests
1/223 Test #14: merge_small_tests ..........................................***Failed 70.09 sec
[==========] Running 2386 tests from 132 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from BoundsCheckedArrayDeathTest
[RUN] BoundsCheckedArrayDeathTest.BoundsCheckRead
[OK] BoundsCheckedArrayDeathTest.BoundsCheckRead (616 ms)
[RUN] BoundsCheckedArrayDeathTest.BoundsCheckAssign
[OK] BoundsCheckedArrayDeathTest.BoundsCheckAssign (329 ms)
[RUN] BoundsCheckedArrayDeathTest.BoundsCheckPopFront
[OK] BoundsCheckedArrayDeathTest.BoundsCheckPopFront (276 ms)
[RUN] BoundsCheckedArrayDeathTest.BoundsCheckResize
[OK] BoundsCheckedArrayDeathTest.BoundsCheckResize (277 ms)
[RUN] BoundsCheckedArrayDeathTest.BoundsCheckResizeAssign
[OK] BoundsCheckedArrayDeathTest.BoundsCheckResizeAssign (290 ms)
[----------] 5 tests from BoundsCheckedArrayDeathTest (1794 ms total)
[----------] 1 test from DebugDeathTest
[RUN] DebugDeathTest.Suicide
[OK] DebugDeathTest.Suicide (178 ms)
[----------] 1 test from DebugDeathTest (178 ms total)
......
216/223 Test #1: hp_test1 ................................................... Passed 0.96 sec
Start 12: pfs_misc
217/223 Test #10: pfs_noop ................................................... Passed 0.76 sec
Start 83: basic
218/223 Test #83: basic ...................................................... Passed 0.12 sec
Start 79: skip
219/223 Test #79: skip ....................................................... Passed 0.07 sec
Start 80: todo
220/223 Test #12: pfs_misc ................................................... Passed 0.91 sec
Start 81: skip_all
221/223 Test #80: todo ....................................................... Passed 0.06 sec
Start 82: no_plan
222/223 Test #81: skip_all ................................................... Passed 0.07 sec
223/223 Test #82: no_plan .................................................... Passed 0.03 sec
94% tests passed, 14 tests failed out of 223
Total Test time (real) = 4334.59 sec
The following tests FAILED:
14 - merge_small_tests (Failed)
15 - merge_large_tests (Failed)
56 - gcs_xcom_xcom_cache (Subprocess killed)
57 - gcs_xcom_control_interface (Failed)
65 - merge_temptable_tests-t (Failed)
187 - routertest_component_bootstrap (Subprocess aborted)
189 - routertest_component_bootstrap_clusterset (Subprocess aborted)
191 - routertest_component_bootstrap_tls_endpoint (Subprocess aborted)
192 - routertest_component_clusterset (Subprocess aborted)
197 - routertest_component_gr_notifications (Failed)
205 - routertest_component_rest_api_enable (Subprocess aborted)
213 - routertest_component_routing_connection (Subprocess aborted)
221 - routertest_component_routing_splicer (Timeout)
223 - routertest_integration_routing_reuse (Failed)
Errors while running CTest
通过 mtr 执行单元测试
在 编译目录 /mysql-test
执行如下指令:
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --unit-tests-report
mtr 会首先运行 binlog suite 的所有 case,之后才会运行单元测试。
压力测试
简介
波及压力测试的有两局部:
压力测试 suites
只有两个:
- stress
- innodb_stress
如须要增加新 case,参考对应 suite 已有 case 照猫画虎即可,后续文章会详解介绍语法。
mysql-stress-test.pl
被 mysql-test-run.pl
调用,参数是--stress
。
应用阐明位于 mysql-test/README.stress
文件:
The stress script is designed to perform testing of the MySQL server in
a multi-threaded environment.
All functionality regarding stress testing is implemented in the
mysql-stress-test.pl script.
The stress script allows:
- To stress test the mysqltest binary test engine.
- To stress test the regular test suite and any additional test suites
(such as mysql-test-extra-5.0).
- To specify files with lists of tests both for initialization of
stress db and for further testing itself.
- To define the number of threads to be concurrently used in testing.
- To define limitations for the test run. such as the number of tests or
loops for execution or duration of testing, delay between test
executions, and so forth.
- To get a readable log file that can be used for identification of
errors that occur during testing.
There are two ways to run the mysql-stress-test.pl script:
- For most cases, it is enough to use the options below for starting
the stress test from the mysql-test-run wrapper. In this case, the
server is run automatically, all preparation steps are performed,
and after that the stress test is started.
- In advanced case, you can run the mysql-stress-test.pl script directly.
But this requires that you perform some preparation steps and to specify
a bunch of options as well, so this invocation method may be a bit
complicated.
可见,有两种用法:
- 大部分状况下,通过
mysql-test-run.pl --stress=[option1,option2,...]
运行即可,该脚本实现了筹备阶段、压力测试阶段所需的工作。 -
更高级的用法是间接执行
mysql-stress-test.pl
脚本,这就须要自行实现筹备阶段、测试阶段所需的工作。次要包含:-
--stress-init-file[=path]
file_name is the location of the file that contains the list of tests to be run once to initialize the database for the testing. If missing, the default file is stress_init.txt in the test suite directory.
-
--stress-tests-file[=file_name]
Use this option to run the stress tests. file_name is the location of the file that contains the list of tests. If file_name is omitted, the default file is stress-test.txt in the stress suite directory. (See
--stress-suite-basedir
).
-
其余参数见手册 MySQL: mysql-stress-test.pl — Server Stress Test Program
指令示例
独自运行压力测试 suites
没找到手册阐明,据我了解,只有未被动敞开单元测试标记(-DWITH_UNIT_TESTS={ON|OFF}
选项,默认是开启的),就必定会编译生成 stress suite。
在 Ubuntu 22.04 X86_64 执行测试,胜利。
wslu@ubuntu:/data/work/mysql/mysql80-install.bak_asan_ubsan/mysql-test$ perl mysql-test-run.pl --force --timer --comment=stress --vardir=var-stress --suite=stress --no-skip --max-test-fail=30
Logging: mysql-test-run.pl --force --timer --comment=stress --vardir=var-stress --suite=stress --no-skip --max-test-fail=30
MySQL Version 8.0.29
##############################################################################
# stress
##############################################################################
Checking supported features
- Binaries are debug compiled
Using suite(s): stress
Collecting tests
Removing old var directory
Creating var directory '/data/work/mysql/mysql80-install.bak_asan_ubsan/mysql-test/var-stress'
Installing system database
Using parallel: 1
==============================================================================
TEST NAME RESULT TIME (ms) COMMENT
------------------------------------------------------------------------------
[16%] stress.ddl_myisam [pass] 88171
[33%] stress.ddl_archive [pass] 11868
[50%] stress.ddl_csv [pass] 8007
[66%] stress.ddl_innodb [pass] 163638
[83%] stress.ddl_memory [pass] 84721
[100%] shutdown_report [pass]
------------------------------------------------------------------------------
The servers were restarted 1 times
The servers were reinitialized 0 times
Spent 356.405 of 423 seconds executing testcases
Completed: All 6 tests were successful.
mysql-stress-test.pl 应用示例
指令示例:
perl mysql-stress-test.pl
--stress-suite-basedir=/opt/qa/mysql-test-extra-5.0/mysql-test
--stress-basedir=/opt/qa/test
--server-logs-dir=/opt/qa/logs
--test-count=20
--stress-tests-file=innodb-tests.txt
--stress-init-file=innodb-init.txt
--threads=5
--suite=funcs_1
--mysqltest=/opt/mysql/mysql-5.0/client/mysqltest
--server-user=root
--server-database=test
--cleanup
官网举荐的压力测试用法
#### 提交代码时执行
perl mysql-test-run.pl --force --timer --comment=stress --vardir=var-stress --suite=stress --no-skip --max-test-fail=30
#### 每天执行
perl mysql-test-run.pl --force --timer --big-test --comment=stress --vardir=var-stress --suite=stress --no-skip
##### 每周执行 basic
# 相较于提交代码时执行的测试指令,多了 --debug-server 选项
perl mysql-test-run.pl --debug-server --force --timer --comment=stress --vardir=var-stress --suite=stress --no-skip
# 相较于上一条多了 --big-test
perl mysql-test-run.pl --debug-server --force --timer --big-test --comment=stress --vardir=var-stress --suite=stress --no-skip
#### 每天执行 valgrind
perl mysql-test-run.pl --force --timer --comment=stress --vardir=var-stress --suite=stress
#### 每周执行 valgrind
# 指定了 --big-test
perl mysql-test-run.pl --force --timer --big-test --testcase-timeout=60 --debug-server --comment=stress-debug-big --vardir=var-stress-debug-big --suite=stress
# 其余
perl mysql-test-run.pl --force --timer --comment=stress --vardir=var-stress --suite=stress --no-skip --max-test-fail=30
perl mysql-test-run.pl --force --timer --comment=innodb-stress --vardir=var-innodb-stress --suite=innodb_stress --no-skip --max-test-fail=30
留神:mysql-test/README.stress
文件的最初批改日期是 2006 年,已过期,没有参考价值。
论断
mtr 执行门路
- 代码覆盖率、单元测试只能在
编译的源码目录 /mysql-test
执行。 - 其余测试在
编译的源码目录 /mysql-test
和装置目录 /mysql-test
都能够执行。 - 如无非凡需要,更倡议在装置目录执行 mtr 测试(目录构造更清晰)。
测试后果及 兼容性
名称 | 对其反对是否是实验性的 | 编译兼容性(同时启用可编译胜利,则为兼容) | mtr 测试后果 | 论断 |
---|---|---|---|---|
ASAN | 否 | 与 UBSAN、Valgrind 兼容。 | 若与 Valgrind 同时启用(mtr 指定 --valgrind 等选项),执行 mtr 测试时,会导致很多 case 因 valgrind memcheck 报错而失败。比方 binlog_gtid.binlog_gtid_mix_ok_packet_all_gtids |
ASAN 与 Valgrind 不可同时启用,但可与 UBSAN 同时启用。 |
LSAN | 否 | 已集成到 ASAN,未测试。 | 已集成到 ASAN,无需独自启用。 | |
UBSAN | 否 | 与 ASAN、valgrind、TSAN 都兼容。 | 倡议与 ASAN 同时启用。 | |
MSAN | 是 | 与 ASAN 不兼容,若同时启用,编译会报错。 独自启用 MSAN,cmake 失败,报错。 |
MySQL 对其的反对是实验性的,暂不倡议应用。 | |
TSAN | 是 | 与 ASAN 不兼容,若同时启用,编译会报错。 与 UBSAN、Valgrind 兼容。 |
即便只启用 TSAN,在运行 mtr 时,install database 阶段仍然有大片的 data race。 | MySQL 对其的反对是实验性的,暂不倡议应用。 |
valgrind | 否 | 与 ASAN、UBSAN、TSAN 兼容。 | 独自启用 valgrind,mtr 可失常执行 齐全体 valgrind 测试(mtr 指定 --valgrind 选项),memcheck 未报错。与 ASAN 同时启用时,若执行齐全体 valgrind 测试,mtr 会因 valgrind memcheck 与 asan 抵触而 crash 。 |
若要应用,不倡议与 ASAN 同时启用。 |
gcov/gprof | 否 | 与 ASAN、UBSAN 兼容。 gprof 只反对 linux,不反对 MacOS/Windows。 |
须要在 执行 cmake 的源码目录 测试。 | 倡议与 ASAN、UBSAN 一起启用。 |
单元测试 | 否 | 只有是 DEBUG 版本,就会默认启用。 与 ASAN、UBSAN 兼容,其余组件未验证,实践上也应该兼容。 |
须要在 执行 cmake 的源码目录 测试,运行make test-unit 指令或./mtr --unit-tests-report 。ASAN 会检测到局部测试 case 本身存在内存透露,导致以后 case 失败。 |
做单元测试时倡议启用 ASAN/UBSAN/LSAN。 |
压力测试 | 否 | 只有编译,就会产生 stress 、innodb_stress 两个 suite。mysql-stress-test.pl 须要自定义 初始化和运行的 SQL 语句,不倡议应用。 |
测试胜利。 | 失常运行 mtr 全量 suite 或独自运行 stress 、innodb_stress suites。 |
注意事项:
- 因为 mtr case 特地多,运行全副 case 工夫过长,因而,该表中所说「失常执行」是执行一部分 suites(而不是全副)未报错。
- 从执行时长来看,如需在 hyper 运行官网举荐 collections,至多要开启 32 并发。
- 对于单元测试、代码覆盖率测试、内存谬误检测,倡议在 X86_64 平台运行,某些选项对 ARM 平台不兼容(编译失败)。
- 局部 perl 版本有内存透露,会被 ASAN 检测进去,导致 mtr 测试终止。目前验证 Ubuntu 22.04 所用的 perl 5.34.0 存在内存透露,而 Ubuntu 20.04 所用的 perl 5.30.0 不存在内存透露。
- 因为我次要是在虚拟机进行验证的,而 mtr 运行太过耗时,因而,本文章节波及的 mtr 指令,大部分并未残缺运行(进度小于 10%)。
举荐用法
本局部是集体依据官网 collections,结合实际状况给出的倡议,仅供参考。
在代码开发阶段,对立应用 debug 版本(编译选项-DWITH_DEBUG=1
)提前发现问题:
- push 代码到 dev 分支时,可参考
default.push
中的指令集。 - merge 代码到 main 分支时,可参考
mysql-trunk-stage.push
(与mysql-8.0-stage.push
完全相同,是default.push
的超集)中的指令集。
内核在公布 alpha 版本前,也要用 debug 版本来验证稳定性:
-
单元测试:
- 编译选项:
-DWITH_DEBUG=1 -DWITH_UNIT_TESTS={ON|OFF}
,默认是 ON。 - 执行门路:只能在编译后的源码目录执行。
-
应用办法:
- 执行
make test
或make test-unit
指令,尽管按手册形容两个指令都能实现单元测试成果,但实测make test-unit
会输入更具体的信息,因而,倡议应用make test-unit
。 - 执行 mtr 指令时增加
--unit-tests-report
选项也有同样成果。
- 执行
- 注意事项:在执行单元测试时,不倡议启用 ASAN。
- 编译选项:
-
内存谬误检测:
- 工具抉择:因为 valgrind 运行很慢,倡议应用 ASAN + UBSAN 来测试。
- 编译选项:
-DWITH_DEBUG=1 -DWITH_ASAN=1 -DUBSAN=1
,可抉择启用-DWITH_ASAN_SCOPE=1
- 倡议指令:官网并未提供举荐指令集,倡议在
default.daily
的指令根底上,增加--sanitize
选项。 - 指令示例:
perl mysql-test-run.pl --timer --max-test-fail=0 --force --comment=var-rpl --vardir=var-rpl --suite=rpl --sanitize
-
代码覆盖率测试:
- 编译选项:
-DWITH_DEBUG=1 -DENABLE_GCOV=1 -DENABLE_GPROF=1
- 特殊要求:必须在
编译的源码目录
执行测试。 -
指令示例:
./mtr --gcov --gprof -big-test --force --max-test-fail=0 --comment=gcov-gprof --vardir=var-gcov-gprof --no-skip
- 在 gcov 执行胜利后,会将代码覆盖率相干信息写到
gmon.out
,之后,mtr 会主动调用 gprof 解析该文件。
- 在 gcov 执行胜利后,会将代码覆盖率相干信息写到
- 编译选项:
-
压力测试 :尽管 mtr 整合了
mysql-stress-test.pl
脚本,但应用该脚本须要自行编写 stress-init、stress-test 文件,因而, 倡议间接测试 stress、innodb_stress 这两个 suites。perl mysql-test-run.pl --force --timer --big-test --comment=stress --vardir=var-stress --suite=stress,innodb_stress --no-skip
-
线程竞争:参考
mysql-test/collections/mysql-trunk-tsan.push
。- 编译选项:
-DWITH_DEBUG=1 -DWITH_TSAN=1
-
指令示例:
perl mysql-test-run.pl --timer --debug-server --force --comment=main-tsan --vardir=var-main-tsan --suite=main
- 在 install database 阶段会检测出大面积线程竞争,因而,以后版本无奈应用。
- 注意事项:TSAN/ThreadSanitizer 运行速度很慢,因而,只倡议运行 main suite。
- 编译选项:
如果须要验证 release 版本稳定性(实用于 QA、研发),可参考 default.daily
中的指令集。
- 该指令集笼罩了单元测试、压力测试等。
编译组合倡议
举荐:
- 一般 debug 版,运行 SQL 兼容性测试 + 单元测试 + 压力测试(stress/innodb_stress suite)
-
内存谬误 + 代码覆盖率测试:asan/ubsan + gcov/gprof
- 当然 二者也可离开编译、测试。
- ASAN 与 valgrind 不可同时启用,执行 mtr 时如果增加
--valgrind
参数,asan 会与 memcheck 抵触导致 crash,测试终止。
可选:
- valgrind:次要用于检测内存问题,但运行速度很慢,更倡议应用 ASAN。
- 编译选项:
-DWITH_VALGRIND=1
继续跟踪后续版本改良状况:
- TSAN:MySQL 对其的反对尚不成熟。
- MSAN:与 ASAN 性能重叠,且 MySQL 对其的反对尚不成熟。
参考链接:
llvm 工具集:
- AddressSanitizer — Clang 17.0.0git documentation (llvm.org)”)
- LeakSanitizer — Clang 17.0.0git documentation (llvm.org)”)
- UndefinedBehaviorSanitizer — Clang 17.0.0git documentation (llvm.org)”)
- ThreadSanitizer — Clang 17.0.0git documentation (llvm.org)”)
- MemorySanitizer — Clang 17.0.0git documentation (llvm.org)”)
- AddressSanitizer | Microsoft Learn
- AddressSanitizer · google/sanitizers Wiki · GitHub
linux kernel 工具集:
- The Kernel Address Sanitizer (KASAN) — The Linux Kernel documentation — The Linux Kernel documentation”)
- The Kernel Memory Sanitizer (KMSAN) — The Linux Kernel documentation — The Linux Kernel documentation”)
MySQL:
- MySQL :: MySQL 8.0 Reference Manual :: 2.8.7 MySQL Source-Configuration Options
-
MySQL: Creating and Executing Unit Tests
- Home – Test Anything Protocol
- GoogleTest User’s Guide | GoogleTest
- GitHub – google/googletest: GoogleTest – Google Testing and Mocking Framework
欢送关注我的微信公众号【数据库内核】:分享支流开源数据库和存储引擎相干技术。
题目 | 网址 |
---|---|
GitHub | https://dbkernel.github.io |
知乎 | https://www.zhihu.com/people/dbkernel/posts |
思否(SegmentFault) | https://segmentfault.com/u/dbkernel |
掘金 | https://juejin.im/user/5e9d3ed251882538083fed1f/posts |
CSDN | https://blog.csdn.net/dbkernel |
博客园(cnblogs) | https://www.cnblogs.com/dbkernel |