关于数据库:从零开始学习MySQL调试跟踪2

35次阅读

共计 9937 个字符，预计需要花费 25 分钟才能阅读完成。

  • GreatSQL 社区原创内容未经受权不得随便应用，转载请分割小编并注明起源。
  • GreatSQL 是 MySQL 的国产分支版本，应用上与 MySQL 统一。
  • 作者：Yejinrong/ 叶金荣
  • 文章起源：GreatSQL 社区投稿
  1. 启用 coredump
  2. 制作一个 coredump 场景
  3. 实在故障场景剖析跟踪

上一篇文档介绍了如何构建 gdb 跟踪调试环境，本文介绍如何依据谬误日志信息，跟踪定位问题可能的起因，以及如何利用 coredump 文件查找问题线索。

1. 启用 coredump

程序运行过程中可能会异样终止或解体，OS 会把程序挂掉时的内存状态记录下来，写入 core 文件，这就叫 coredump，通过 gdb 联合 core 文件能够不便地进行调试。

利用 core 文件中保留的异样堆栈文件，可能帮忙研发同学更快定位问题。因而，如果某些故障断断续续会呈现，倡议阶段性开启 coredump 性能。

想要开启 coredump，须要先批改 OS 层的几个设置：

$ ulimit -c unlimited
$ sysctl -w fs.suid_dumpable=2
$ echo "core.%p.%e.%s" > /proc/sys/kernel/core_pattern

同时，将这些批改长久化到相应文件中（假设 MySQL/GreatSQL 服务过程的属主用户是 mysql）：

$ echo "mysql  -  core   unlimited" >> /etc/security/limits.conf
$ echo "fs.suid_dumpable=2" >> /etc/sysctl.conf
$ echo "kernel.core_pattern=core.%e.%p.%t" >> /etc/sysctl.conf
$ sysctl -p

接下来，批改 my.cnf 配置文件，减少以下两行内容：

core_file
innodb_buffer_pool_in_core_file=OFF

而后重启 GreatSQL 服务过程，即可失效，查问确认下：

mysql> show global variables like '%core%';
+---------------------------------+-------+
| Variable_name                   | Value |
+---------------------------------+-------+
| core_file                       | ON    |
| innodb_buffer_pool_in_core_file | OFF   |
+---------------------------------+-------+

这样设置实现后，需要的话会在 datadir 目录下生成 core 文件。

2. 制作一个 coredump 场景

咱们能够给 mysqld 过程发送 SIGSEGV(11) 信号，即可模拟出 coredump 的场景，例如：

$ kill -s SIGSEGV `pidof mysqld`

这时查看 GreatSQL 谬误日志文件，以及 core 文件，就会发现有 coredump：

$ls -la 
...
-rw-------   1 mysql mysql 1081147392 Feb 20 22:36 core.mysqld-debug.2658134.1676903816
...

$ less error.log
...
14:36:56 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: 1f4232b893100742b7c519df2fa714648c2d76d9
Server Version: 8.0.25-16-debug Source distribution

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x80000
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4b04
d26]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(handle_fatal_signal+0x2cb) [0x39a7d22]
/lib64/libpthread.so.0(+0x12c20) [0x7fc3e669ac20]
/lib64/libc.so.6(__poll+0x51) [0x7fc3e45c4a41]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(Mysqld_socket_listener::listen_for_connection_event()+0x57) [0x3995195]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(Connection_acceptor<Mysqld_socket_listener>::connection_event_loop()+0
x30) [0x355a024]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(mysqld_main(int, char**)+0x27d2) [0x354e4a6]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(main+0x20) [0x32de906]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc3e44f6493]
/usr/local/GreatSQL-8.0.25-16-Linux-glibc2.28-x86_64/bin/mysqld-debug(_start+0x2e) [0x32de82e]
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/
...

在一线的同学，如果须要向研发寻求反对或报告故障时，能够先参考这篇文章 MySQL 报障之 coredump 收集解决流程，须要采集其余几个信息：

  • 故障时刻的 error log。
  • 故障产生的 core 文件。
  • 如果有 general log 的话，也采集起来（故障时刻往前约 1 小时或 10 万行日志）。
  • 导致 core 产生波及到的表 DDL 以及相应的 SQL 语句，有必要的话，可能还要同时提供实在数据（或样例数据）。

3. 实在故障场景剖析跟踪

在 GreatSQL 8.0.25-15 版本（上一个版本）中，InnoDB 并行查问性能在特定场景下存在 bug，会导致 crash，相应的日志见下：

mysqld-debug: /opt/greatsql-8.0.25/sql/item.cc:6047: virtual void Item_field::make_field(Send_field*): Assertion `item_name.is_set()' failed.
01:59:20 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: 1f4232b893100742b7c519df2fa714648c2d76d9
Server Version: 8.0.25-debug Source distribution

Thread pointer: 0x7fb4a9a0b000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fb4f7aa53b0 thread_stack 0x80000
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4b04d26]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(handle_fatal_signal+0x2cb) [0x39a7d22]
/lib64/libpthread.so.0(+0x12c20) [0x7fb5146cac20]
/lib64/libc.so.6(gsignal+0x10f) [0x7fb51253a37f]
/lib64/libc.so.6(abort+0x127) [0x7fb512524db5]
/lib64/libc.so.6(+0x21c89) [0x7fb512524c89]
/lib64/libc.so.6(+0x2fa76) [0x7fb512532a76]  #<-- 从这里网上，都是谬误信息处理逻辑
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Item_field::make_field(Send_field*)+0x9e) [0x3338758]  #<-- 从这里往下，才是真正触发故障的地位，并记住 "0x3338758" 这个指针
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(THD::send_result_metadata(mem_root_deque<Item*> const&, unsigned int)+0x19d) [0x36977ab]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Query_result_send::send_result_set_metadata(THD*, mem_root_deque<Item*> const&, unsigned int)+0x2d) [0x35f3ff9]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Query_expression::ExecuteIteratorQuery(THD*)+0x1f1) [0x38d057b]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Query_expression::execute(THD*)+0xed) [0x38d0d7d]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Sql_cmd_dml::execute_inner(THD*)+0x1c1) [0x381db25]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(Sql_cmd_dml::execute(THD*)+0x5c7) [0x381cfab]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(mysql_execute_command(THD*, bool)+0x565c) [0x37a1a2b]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(dispatch_sql_command(THD*, Parser_state*, bool)+0x769) [0x37a3a1d]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1491) [0x3799819]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug(do_command(THD*)+0x51c) [0x3797c48]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug() [0x3991168]
/usr/local/GreatSQL-8.0.25-Linux-glibc2.28-x86_64/bin/mysqld-debug() [0x52e4b22]
/lib64/libpthread.so.0(+0x817a) [0x7fb5146c017a]
/lib64/libc.so.6(clone+0x43) [0x7fb5125ffdc3]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fb4a9a65028): SELECT  ...  FROM t1 WHERE ...  #<-- 这是触发 bug 的 SQL 语句
Connection ID (thread ID): 8
Status: NOT_KILLED

依照下面所说的办法，咱们采集了所有相干信息，并能在测试环境重现上述故障。

接下来，咱们利用 gdb 来定位剖析问题起因：

$ gdb path/bin/mysqld-debug path/core.mysqld-debug.2657287.1657270311
GNU gdb (GDB) Red Hat Enterprise Linux 9.2-4.el8
...
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bin/mysqld-debug...
...
[New LWP 2675795]
[New LWP 2675825]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `./bin/mysqld-debug --defaults-extra-file=./my.cnf'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56        return (INTERNAL_SYSCALL_ERROR_P (val, err)
[Current thread is 1 (Thread 0x7fb4f7aa7700 (LWP 2676055))]
(gdb)
(gdb) b *0x3338758  #<-- 下面记下的指针值，后面加个 "*" 号，在这里打上断点
Breakpoint 1 at 0x3338758: file /opt/greatsql-8.0.25/sql/item.cc, line 6048.  #<-- 指向可能触发问题的源码地位
(gdb)
(gdb) bt  #<-- 打印具体 backtrace 信息
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x0000000004b04f1d in my_write_core (sig=6) at /opt/greatsql-8.0.25/mysys/stacktrace.cc:409
#2  0x00000000039a7f84 in handle_fatal_signal (sig=6) at /opt/greatsql-8.0.25/sql/signal_handler.cc:199
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  0x00007fb512524db5 in __GI_abort () at abort.c:79
#6  0x00007fb512524c89 in __assert_fail_base (fmt=0x7fb51268d698 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x57a1835"item_name.is_set()",
    file=0x57a1400 "/opt/greatsql-8.0.25/sql/item.cc", line=6047, function=<optimized out>) at assert.c:92
#7  0x00007fb512532a76 in __GI___assert_fail (assertion=0x57a1835 "item_name.is_set()", file=0x57a1400 "/opt/greatsql-8.0.25/sql/item.cc", line=6047,
    function=0x57a3a40 "virtual void Item_field::make_field(Send_field*)") at assert.c:101
#8  0x0000000003338758 in Item_field::make_field (this=0x7fb4a9b5bcf8, tmp_field=0x7fb4f7aa2380) at /opt/greatsql-8.0.25/sql/item.cc:6047
#9  0x00000000036977ab in THD::send_result_metadata (this=0x7fb4a9a0b000, list=..., flags=5) at /opt/greatsql-8.0.25/sql/sql_class.cc:2824
#10 0x00000000035f3ff9 in Query_result_send::send_result_set_metadata (this=0x7fb4a9a0fda0, thd=0x7fb4a9a0b000, list=..., flags=5)
    at /opt/greatsql-8.0.25/sql/query_result.cc:76
#11 0x00000000038d057b in Query_expression::ExecuteIteratorQuery (this=0x7fb4a9a65178, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_union.cc:1150
#12 0x00000000038d0d7d in Query_expression::execute (this=0x7fb4a9a65178, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_union.cc:1321
#13 0x000000000381db25 in Sql_cmd_dml::execute_inner (this=0x7fb4a9a0fd68, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_select.cc:814
#14 0x000000000381cfab in Sql_cmd_dml::execute (this=0x7fb4a9a0fd68, thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_select.cc:585
#15 0x00000000037a1a2b in mysql_execute_command (thd=0x7fb4a9a0b000, first_level=true) at /opt/greatsql-8.0.25/sql/sql_parse.cc:4684
#16 0x00000000037a3a1d in dispatch_sql_command (thd=0x7fb4a9a0b000, parser_state=0x7fb4f7aa41d0, update_userstat=false)
    at /opt/greatsql-8.0.25/sql/sql_parse.cc:5284
#17 0x0000000003799819 in dispatch_command (thd=0x7fb4a9a0b000, com_data=0x7fb4f7aa5370, command=COM_QUERY) at /opt/greatsql-8.0.25/sql/sql_parse.cc:1940
#18 0x0000000003797c48 in do_command (thd=0x7fb4a9a0b000) at /opt/greatsql-8.0.25/sql/sql_parse.cc:1388
#19 0x0000000003991168 in handle_connection (arg=0x7fb4ba094500) at /opt/greatsql-8.0.25/sql/conn_handler/connection_handler_per_thread.cc:307
#20 0x00000000052e4b22 in pfs_spawn_thread (arg=0x7fb511e44320) at /opt/greatsql-8.0.25/storage/perfschema/pfs.cc:2899
#21 0x00007fb5146c017a in start_thread (arg=<optimized out>) at pthread_create.c:479
#22 0x00007fb5125ffdc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

有了这些信息，研发同学再去跟踪定位问题本源就会不便很多。

本文简略演示了如何利用 core 文件去跟踪定位剖析可能导致 crash 的起因，更多乏味实用的办法还有待进一步开掘，一起摸索新世界吧。

Enjoy GreatSQL :)

## 对于 GreatSQL

GreatSQL 是由万里数据库保护的 MySQL 分支，专一于晋升 MGR 可靠性及性能，反对 InnoDB 并行查问个性，是实用于金融级利用的 MySQL 分支版本。

相干链接：GreatSQL 社区 Gitee GitHub Bilibili

GreatSQL 社区：

社区博客有奖征稿详情：https://greatsql.cn/thread-100-1-1.html

技术交换群：

微信：扫码增加 GreatSQL 社区助手 微信好友，发送验证信息 加群。

正文完
数据库
发表至： 数据库
2023-04-17
 0
关于数据库:基于-StarRocks-构建广告数据中心的实践
关于数据库:SphereEx-创始人张亮云咖访谈回顾构建数据服务的新思路
关于数据库:十八般武艺玩转GaussDBDWS性能调优三好味道表定义
关于数据库:知乎运营分析平台-舰桥平台如何通过-Apache-Doris-实现查询速度-10-倍提升