前言
为了解决 LVS ksoftirqd CPU 使用率 100% 导致网卡软中断丢包，我和同事们一起搜索了大量的资料去分析问题，特别是感谢美团技术团队的分享帮助我们快速梳理优化思路，最后明确了如何重构 RPS 和 RFS 网卡多队列的优化脚本。个人认为这是一个大家可能普遍会遇到的问题，文章内的分析思路和解决方案未必是最优解，也欢迎各位分享自己的解决方法。
RPS 和 RFS 网卡多队列性能调优实践
更新历史
2019 年 07 月 03 日 – 初稿
阅读原文 – https://wsgzao.github.io/post…
扩展阅读
Redis 高负载下的中断优化 – https://tech.meituan.com/2018…
事故复盘
我们遇到的问题属于计划外的 incident，现象是某产品用户在线率突然降低，LVS Master 同时收到 CPU High Load 告警，检查发现该节点出现网卡大量断开重连和丢包情况，应急切换到 LVS Slave 也出现上述问题，在排除掉流量异常和外部攻击后选择切换 DNS 到背后的 Nginx Real Servers 后服务逐步恢复。
复盘核心原因在于系统初始化时 rps 优化脚本没有成功执行，这个脚本起初是因为早期 DBA 团队遇到过 CPU 负载较高导致网卡异常，这个优化脚本也一直传承至今，却已经没有人知道为什么添加。现在大多数服务器没有执行成功而被大家一直所忽视显然也是 post check 没有做到位。在早期大家都停留在 Bash Shell 运维的阶段，没有专职的团队来管理确实容易失控，好在现在可以基于 Ansible 来做初始化和检查，运维的压力也减轻了一部分。
通过 Google 搜索相关知识的过程中，我们也发现在不少人都会遇到这样类似的问题。比如这篇文章提到 lvs/irq
lvs 的性能问题，软中断耗尽 CPU 单核后到达处理极限
-
瓶颈现象：当压力较大时，Lvs 服务器 CPU 的其中一个核使用率达到 100%（处理软中断）。
- 当 Lvs 服务器处理软中断的那个核使用率达到 100%，就到达系统处理上限。
- 占用 CPU 的是进程“ksoftirqd”，它未能使用到多核。
- 做双网卡绑定，然后调试内核 SMP，中断主要是来自网卡的，不是 LVS 本身。需要把 2 个网卡来的 IRQ 均衡在双核 CPU 上面。
和华为的工程师们在交换经验的时候对方分享了一个关于 RSS 和 RPS 关系图，之后的内容还会引用美团技术团队的分析
我们遇到的情况是缺少可用服务器资源选择把用户外部请求流量和 Codis Cache Cluster 内部流量临时混在了同一个 LVS 上，虽然看上去 CPU 和 traffic 的整体压力都不算高，但是 CPU 的处理压力可能恰好集中在了和外网 Bond1 网卡相同的 Core 上最后引起了 ksoftirqd 软中断，而内网 Bond0 网卡就没有监控到任何丢包。虽然我们也有正常开启 irqbalance，但不清楚是不是因为受到cpupower performance 和NUMA的影响最后也没能阻止事故的发生，最终的优化方案主要是手动开启 RPS 和 RFS，大致步骤如下：
- set cpupower
cpupower frequency-set -g performance，CPU 优化建议使用 cpupower 设置 CPU Performance 模式 - activate rps/rfs by script: rps.sh
- double ring buffer size:
ethtool -G p1p1 [rx|tx] 4096, checkethtool -g p1p1 - double NAPI poll budget:
sysctl -w net.core.netdev_budget=600 - add zabbix monitor on net.if.in[eth0,errors,dropped,overruns]
#!/bin/bash
# chkconfig: 2345 90 60
### BEGIN INIT INFO
# Provides: rps
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: enable rps config for ubuntu
# Description: enabele rps which is a kernel tweak for network performance
### END INIT INFO
NAME=rps
DESC=rps
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# cpupower frequency-set -g performance
# activate rps/rfs by script: https://gist.github.com/wsgzao/18828f69147635f3e38a14690a633daf
# double ring buffer size: ethtool -G p1p1 [rx|tx] 4096, ethtool -g p1p1
# double NAPI poll budget: sysctl -w net.core.netdev_budget=600
rps() {net_interface=`ip link show | grep "state UP" | awk '{print $2}' | egrep -v '^docker|^veth' | tr ":\n" " "`
for em in ${net_interface[@]}
do
rq_count=`ls /sys/class/net/$em/queues/rx-* -d | wc -l`
rps_flow_cnt_value=`expr 32768 / $rq_count`
for ((i=0; i< $rq_count; i++))
do
echo $rps_flow_cnt_value > /sys/class/net/$em/queues/rx-$i/rps_flow_cnt
done
flag=0
while [-f /sys/class/net/$em/queues/rx-$flag/rps_cpus]
do
echo `cat /sys/class/net/$em/queues/rx-$flag/rps_cpus | sed 's/0/f/g' ` > /sys/class/net/$em/queues/rx-$flag/rps_cpus
flag=$(($flag+1))
done
done
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
sysctl -p
}
check_rps() {ni_list=`ip link show | grep "state UP" | awk '{print $2}' | egrep -v "^docker|^veth" | tr ":\n" " "`
for n in $ni_list
do
rx_queues=`ls /sys/class/net/$n/queues/ | grep "rx-[0-9]"`
for q in $rx_queues
do
rps_cpus=`cat /sys/class/net/$n/queues/$q/rps_cpus`
rps_flow_cnt=`cat /sys/class/net/$n/queues/$q/rps_flow_cnt`
echo "[$n]" $q "--> rps_cpus =" $rps_cpus ", rps_flow_cnt =" $rps_flow_cnt
done
done
rps_sock_flow_entries=`cat /proc/sys/net/core/rps_sock_flow_entries`
echo "rps_sock_flow_entries =" $rps_sock_flow_entries
}
case "$1" in
start)
echo -n "Starting $DESC:"
rps
check_rps
;;
stop)
echo -n "Stop is not supported."
;;
restart|reload|force-reload)
echo -n "Restart is not supported."
;;
status)
check_rps
;;
*)
echo "Usage: $0 [start|status]"
;;
esac
exit 0
Scaling in the Linux Networking Stack
了解 RSS、RPS、RFS 等基础知识很重要
This document describes a set of complementary techniques in the Linux
networking stack to increase parallelism and improve performance for
multi-processor systems.
The following technologies are described:
- RSS: Receive Side Scaling
- RPS: Receive Packet Steering
- RFS: Receive Flow Steering
- Accelerated Receive Flow Steering
- XPS: Transmit Packet Steering
https://www.kernel.org/doc/Do…
RPS
RECEIVE PACKET STEERING (RPS)
Receive Packet Steering (RPS) is similar to RSS in that it is used to direct packets to specific CPUs for processing. However, RPS is implemented at the software level, and helps to prevent the hardware queue of a single network interface card from becoming a bottleneck in network traffic.
RPS has several advantages over hardware-based RSS:
- RPS can be used with any network interface card.
- It is easy to add software filters to RPS to deal with new protocols.
- RPS does not increase the hardware interrupt rate of the network device. However, it does introduce inter-processor interrupts.
RPS is configured per network device and receive queue, in the /sys/class/net/*device*/queues/*rx-queue*/rps_cpus file, where device is the name of the network device (such as eth0) and rx-queue is the name of the appropriate receive queue (such as rx-0).
The default value of the rps_cpus file is zero. This disables RPS, so the CPU that handles the network interrupt also processes the packet.
To enable RPS, configure the appropriate rps_cpus file with the CPUs that should process packets from the specified network device and receive queue.
The rps_cpus files use comma-delimited CPU bitmaps. Therefore, to allow a CPU to handle interrupts for the receive queue on an interface, set the value of their positions in the bitmap to 1. For example, to handle interrupts with CPUs 0, 1, 2, and 3, set the value of rps_cpus to 00001111 (1+2+4+8), or f (the hexadecimal value for 15).
For network devices with single transmit queues, best performance can be achieved by configuring RPS to use CPUs in the same memory domain. On non-NUMA systems, this means that all available CPUs can be used. If the network interrupt rate is extremely high, excluding the CPU that handles network interrupts may also improve performance.
For network devices with multiple queues, there is typically no benefit to configuring both RPS and RSS, as RSS is configured to map a CPU to each receive queue by default. However, RPS may still be beneficial if there are fewer hardware queues than CPUs, and RPS is configured to use CPUs in the same memory domain.
RFS
RECEIVE FLOW STEERING (RFS)
Receive Flow Steering (RFS) extends RPS behavior to increase the CPU cache hit rate and thereby reduce network latency. Where RPS forwards packets based solely on queue length, RFS uses the RPS backend to calculate the most appropriate CPU, then forwards packets based on the location of the application consuming the packet. This increases CPU cache efficiency.
RFS is disabled by default. To enable RFS, you must edit two files:
/proc/sys/net/core/rps_sock_flow_entries
Set the value of this file to the maximum expected number of concurrently active connections. We recommend a value of 32768 for moderate server loads. All values entered are rounded up to the nearest power of 2 in practice.
/sys/class/net/*device*/queues/*rx-queue*/rps_flow_cnt
Replace device with the name of the network device you wish to configure (for example, eth0), and rx-queue with the receive queue you wish to configure (for example, rx-0).
Set the value of this file to the value of rps_sock_flow_entries divided by N, where N is the number of receive queues on a device. For example, if rps_flow_entries is set to 32768 and there are 16 configured receive queues, rps_flow_cnt should be set to 2048. For single-queue devices, the value of rps_flow_cnt is the same as the value of rps_sock_flow_entries.
Data received from a single sender is not sent to more than one CPU. If the amount of data received from a single sender is greater than a single CPU can handle, configure a larger frame size to reduce the number of interrupts and therefore the amount of processing work for the CPU. Alternatively, consider NIC offload options or faster CPUs.
Consider using numactl or taskset in conjunction with RFS to pin applications to specific cores, sockets, or NUMA nodes. This can help prevent packets from being processed out of order.
了解接收数据包的流程
这里摘取了美团技术团队的分析，在此表示感谢
接收数据包是一个复杂的过程，涉及很多底层的技术细节，但大致需要以下几个步骤：
- 网卡收到数据包。
- 将数据包从网卡硬件缓存转移到服务器内存中。
- 通知内核处理。
- 经过 TCP/IP 协议逐层处理。
- 应用程序通过
read()从socket buffer读取数据。
将网卡收到的数据包转移到主机内存（NIC 与驱动交互）
NIC 在接收到数据包之后，首先需要将数据同步到内核中，这中间的桥梁是 rx ring buffer。它是由 NIC 和驱动程序共享的一片区域，事实上，rx ring buffer 存储的并不是实际的 packet 数据，而是一个描述符，这个描述符指向了它真正的存储地址，具体流程如下：
- 驱动在内存中分配一片缓冲区用来接收数据包，叫做
sk_buffer； - 将上述缓冲区的地址和大小（即接收描述符），加入到
rx ring buffer。描述符中的缓冲区地址是 DMA 使用的物理地址； - 驱动通知网卡有一个新的描述符；
- 网卡从
rx ring buffer中取出描述符，从而获知缓冲区的地址和大小； - 网卡收到新的数据包；
- 网卡将新数据包通过 DMA 直接写到
sk_buffer中。
当驱动处理速度跟不上网卡收包速度时，驱动来不及分配缓冲区，NIC 接收到的数据包无法及时写到 sk_buffer，就会产生堆积，当 NIC 内部缓冲区写满后，就会丢弃部分数据，引起丢包。这部分丢包为 rx_fifo_errors，在 /proc/net/dev 中体现为 fifo 字段增长，在 ifconfig 中体现为 overruns 指标增长。
通知系统内核处理（驱动与 Linux 内核交互）
这个时候，数据包已经被转移到了 sk_buffer 中。前文提到，这是驱动程序在内存中分配的一片缓冲区，并且是通过 DMA 写入的，这种方式不依赖 CPU 直接将数据写到了内存中，意味着对内核来说，其实并不知道已经有新数据到了内存中。那么如何让内核知道有新数据进来了呢？答案就是中断，通过中断告诉内核有新数据进来了，并需要进行后续处理。
提到中断，就涉及到硬中断和软中断，首先需要简单了解一下它们的区别：
- 硬中断：由硬件自己生成，具有随机性，硬中断被 CPU 接收后，触发执行中断处理程序。中断处理程序只会处理关键性的、短时间内可以处理完的工作，剩余耗时较长工作，会放到中断之后，由软中断来完成。硬中断也被称为上半部分。
- 软中断：由硬中断对应的中断处理程序生成，往往是预先在代码里实现好的，不具有随机性。（除此之外，也有应用程序触发的软中断，与本文讨论的网卡收包无关。）也被称为下半部分。
当 NIC 把数据包通过 DMA 复制到内核缓冲区 sk_buffer 后，NIC 立即发起一个硬件中断。CPU 接收后，首先进入上半部分，网卡中断对应的中断处理程序是网卡驱动程序的一部分，之后由它发起软中断，进入下半部分，开始消费 sk_buffer 中的数据，交给内核协议栈处理。
通过中断，能够快速及时地响应网卡数据请求，但如果数据量大，那么会产生大量中断请求，CPU 大部分时间都忙于处理中断，效率很低。为了解决这个问题，现在的内核及驱动都采用一种叫 NAPI（new API）的方式进行数据处理，其原理可以简单理解为 中断 + 轮询，在数据量大时，一次中断后通过轮询接收一定数量包再返回，避免产生多次中断。
什么是中断？
由于接收来自外围硬件 (相对于 CPU 和内存) 的异步信号或者来自软件的同步信号，而进行相应的硬件、软件处理；发出这样的信号称为进行中断请求 (interrupt request, IRQ)
[](# 硬中断与软中断 “ 硬中断与软中断?”)硬中断与软中断？
- 硬中断：外围硬件发给 CPU 或者内存的异步信号就称之为硬中断
- 软中断：由软件系统本身发给操作系统内核的中断信号，称之为软中断。通常是由硬中断处理程序或进程调度程序对操作系统内核的中断，也就是我们常说的系统调用 (System Call)
[](# 硬中断与软中断之区别与联系？“ 硬中断与软中断之区别与联系？”)硬中断与软中断之区别与联系？
- 硬中断是有外设硬件发出的，需要有中断控制器之参与。其过程是外设侦测到变化，告知中断控制器，中断控制器通过 CPU 或内存的中断脚通知 CPU，然后硬件进行程序计数器及堆栈寄存器之现场保存工作（引发上下文切换），并根据中断向量调用硬中断处理程序进行中断处理
- 软中断则通常是由硬中断处理程序或者进程调度程序等软件程序发出的中断信号，无需中断控制器之参与，直接以一个 CPU 指令之形式指示 CPU 进行程序计数器及堆栈寄存器之现场保存工作 (亦会引发上下文切换)，并调用相应的软中断处理程序进行中断处理 (即我们通常所言之系统调用)
- 硬中断直接以硬件的方式引发，处理速度快。软中断以软件指令之方式适合于对响应速度要求不是特别严格的场景
- 硬中断通过设置 CPU 的屏蔽位可进行屏蔽，软中断则由于是指令之方式给出，不能屏蔽
- 硬中断发生后，通常会在硬中断处理程序中调用一个软中断来进行后续工作的处理
- 硬中断和软中断均会引起上下文切换 (进程 / 线程之切换)，进程切换的过程是差不多的
查看中断情况
1.top 按下数字键 1
- hi : time spent servicing hardware interrupts
- si : time spent servicing software interrupts
2.mpstat -P ALL 2
- irq 为硬中断
- soft 为软中断
mpstat 使用介绍和输出参数详解 – https://wsgzao.github.io/post…
[root@localhost wangao]# top
top - 19:17:57 up 7 days, 7:11, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 338 total, 1 running, 337 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 2.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 1.3 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st
%Cpu12 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu14 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st
%Cpu16 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 st
%Cpu17 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu18 : 0.3 us, 0.7 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu19 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu20 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu21 : 0.0 us, 0.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
%Cpu22 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu23 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st
%Cpu24 : 0.0 us, 0.0 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu25 : 0.0 us, 0.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
%Cpu26 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu27 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu28 : 0.0 us, 0.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 3.3 si, 0.0 st
%Cpu29 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st
%Cpu30 : 0.0 us, 0.0 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st
%Cpu31 : 0.0 us, 0.0 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 2.6 si, 0.0 st
KiB Mem : 65688788 total, 62127224 free, 1742128 used, 1819436 buff/cache
KiB Swap: 66060284 total, 66060284 free, 0 used. 63291844 avail Mem
[root@localhost wangao]# mpstat -P ALL 2
Linux 3.10.0-957.21.3.el7.x86_64 (localhost.localdomain) 07/04/2019 _x86_64_ (32 CPU)
07:18:35 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:18:37 PM all 0.02 0.00 0.02 0.00 0.00 2.14 0.00 0.00 0.00 97.83
07:18:37 PM 0 0.00 0.00 0.00 0.00 0.00 2.50 0.00 0.00 0.00 97.50
07:18:37 PM 1 0.00 0.00 0.00 0.00 0.00 4.41 0.00 0.00 0.00 95.59
07:18:37 PM 2 0.00 0.00 0.00 0.00 0.00 2.51 0.00 0.00 0.00 97.49
07:18:37 PM 3 0.00 0.00 0.00 0.00 0.00 2.96 0.00 0.00 0.00 97.04
07:18:37 PM 4 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99
07:18:37 PM 5 0.00 0.00 0.00 0.00 0.00 2.99 0.00 0.00 0.00 97.01
07:18:37 PM 6 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99
07:18:37 PM 7 0.00 0.00 0.00 0.00 0.00 2.99 0.00 0.00 0.00 97.01
07:18:37 PM 8 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50
07:18:37 PM 9 0.00 0.00 0.00 0.00 0.00 2.01 0.00 0.00 0.00 97.99
07:18:37 PM 10 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50
07:18:37 PM 11 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50
07:18:37 PM 12 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99
07:18:37 PM 13 0.00 0.00 0.00 0.00 0.00 2.49 0.00 0.00 0.00 97.51
07:18:37 PM 14 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99
07:18:37 PM 15 0.00 0.00 0.00 0.00 0.00 3.94 0.00 0.00 0.00 96.06
07:18:37 PM 16 0.00 0.00 0.00 0.00 0.00 2.50 0.00 0.00 0.00 97.50
07:18:37 PM 17 0.00 0.00 0.00 0.00 0.00 2.97 0.00 0.00 0.00 97.03
07:18:37 PM 18 0.00 0.00 0.00 0.00 0.00 2.02 0.00 0.00 0.00 97.98
07:18:37 PM 19 0.00 0.00 0.00 0.00 0.00 1.49 0.00 0.00 0.00 98.51
07:18:37 PM 20 0.00 0.00 0.00 0.00 0.00 1.01 0.00 0.00 0.00 98.99
07:18:37 PM 21 0.00 0.00 0.00 0.00 0.00 2.50 0.00 0.00 0.00 97.50
07:18:37 PM 22 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50
07:18:37 PM 23 0.00 0.00 0.00 0.00 0.00 3.48 0.00 0.00 0.00 96.52
07:18:37 PM 24 0.00 0.00 0.00 0.00 0.00 0.51 0.00 0.00 0.00 99.49
07:18:37 PM 25 0.00 0.00 0.00 0.00 0.00 2.97 0.00 0.00 0.00 97.03
07:18:37 PM 26 0.00 0.00 0.00 0.00 0.00 1.99 0.00 0.00 0.00 98.01
07:18:37 PM 27 0.00 0.00 0.00 0.00 0.00 0.51 0.00 0.00 0.00 99.49
07:18:37 PM 28 0.00 0.00 0.00 0.00 0.00 1.51 0.00 0.00 0.00 98.49
07:18:37 PM 29 0.00 0.00 0.00 0.00 0.00 3.45 0.00 0.00 0.00 96.55
07:18:37 PM 30 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 98.50
07:18:37 PM 31 0.00 0.00 0.00 0.00 0.00 2.97 0.00 0.00 0.00 97.03
优化建议和效果演示
- 查看网卡支持多队列的情况，
ethtool -l eth0 - 开启 irqbalance 服务，让系统自动调整网络中断在多个 CPU 核上的分配,
systemctl start irqbalance - 开启 RPS/RFS 特性，软件方式实现 CPU 均衡，接收包中断的优化
查看 RPS/RFS 优化前后变化
# 优化前
[root@localhost wangao]# service rps status
[em1] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[em1] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[em1] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[em1] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-16 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-17 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-18 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-19 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-20 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-21 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-22 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-23 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-24 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-25 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-26 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-27 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-28 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-29 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-30 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-31 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-16 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-17 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-18 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-19 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-20 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-21 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-22 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-23 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-24 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-25 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-26 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-27 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-28 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-29 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-30 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-31 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
rps_sock_flow_entries = 0
# 优化后
[root@localhost wangao]# service rps status
[em1] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[em1] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[em1] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[em1] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[p1p1] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-16 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-17 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-18 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-19 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-20 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-21 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-22 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-23 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-24 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-25 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-26 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-27 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-28 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-29 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-30 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-31 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-16 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-17 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-18 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-19 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-20 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-21 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-22 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-23 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-24 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-25 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-26 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-27 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-28 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-29 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-30 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-31 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[bond0] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
rps_sock_flow_entries = 32768查看软中断变化
# 优化前
[root@localhost wangao]# egrep 'cpu|p1p1' /proc/interrupts
68: 14737707 0 7843 0 571920 0 265157 0 0 0 1610 0 15028 0 314078 0 54904 0 449958 0 126160 0 338804 0 668880 0 122459 0 174803 0 26147 0 IR-PCI-MSI-edge p1p1-TxRx-0
69: 56960 0 115829 0 2669071 0 61425 0 25161 0 4573083 0 72772 0 30975 0 61705 0 189408 0 13134 0 212159 0 49345 0 104657 0 1475007 0 82145 0 IR-PCI-MSI-edge p1p1-TxRx-1
70: 73053 0 118657 0 95018 0 8645414 0 60962 0 83141 0 53948 0 7465 0 99615 0 119273 0 26191 0 14160 0 45703 0 335306 0 4014649 0 94692 0 IR-PCI-MSI-edge p1p1-TxRx-2
71: 46293 0 75419 0 57967 0 6088 0 44337 0 88437 0 22859 0 51370 0 45510 0 120119 0 53276 0 1105728 0 92556 0 7627687 0 374344 0 79394 0 IR-PCI-MSI-edge p1p1-TxRx-3
74: 55261 0 738486 0 14860 0 76202 0 173074 0 215494 0 6385081 0 2691319 0 167374 0 60585 0 17804 0 63837 0 1267656 0 25655 0 94509 0 115839 0 IR-PCI-MSI-edge p1p1-TxRx-4
75: 59182 0 58481 0 89362 0 49916 0 53663 0 40476 0 81847 0 48582 0 7661643 0 74951 0 81414 0 37247 0 58610 0 244719 0 1449948 0 53745 0 IR-PCI-MSI-edge p1p1-TxRx-5
76: 46993 0 290826 0 58661 0 31146 0 112260 0 51531 0 124002 0 47309 0 74370 0 5474439 0 8948212 0 97511 0 184151 0 141408 0 60222 0 19645 0 IR-PCI-MSI-edge p1p1-TxRx-6
77: 44716 0 30601 0 111528 0 61197 0 29883 0 1262486 0 44401 0 19157 0 55785 0 80069 0 143110 0 67065 0 8581759 0 75214 0 201489 0 116415 0 IR-PCI-MSI-edge p1p1-TxRx-7
78: 68542 0 46913 0 60848 0 61158 0 66591 0 8619198 0 1284598 0 68180 0 88156 0 3904413 0 32320 0 158036 0 74480 0 36877 0 91269 0 66078 0 IR-PCI-MSI-edge p1p1-TxRx-8
79: 45256 0 70914 0 57025 0 58231 0 70398 0 24195 0 29084 0 55756 0 8559005 0 91255 0 105623 0 66336 0 34652 0 161266 0 1555160 0 63201 0 IR-PCI-MSI-edge p1p1-TxRx-9
80: 81984 0 71120 0 8853871 0 4829736 0 38789 0 66915 0 37601 0 55277 0 52352 0 226681 0 64967 0 68163 0 100074 0 48442 0 106219 0 120024 0 IR-PCI-MSI-edge p1p1-TxRx-10
81: 69573 0 55072 0 98255 0 49468 0 11084926 0 50300 0 67870 0 91907 0 61667 0 69887 0 83711 0 80970 0 87310 0 47258 0 1438136 0 80672 0 IR-PCI-MSI-edge p1p1-TxRx-11
82: 41874 0 141338 0 40619 0 9095415 0 2744017 0 63918 0 203991 0 314563 0 1205451 0 124044 0 112602 0 308959 0 43454 0 128518 0 46659 0 80356 0 IR-PCI-MSI-edge p1p1-TxRx-12
83: 33080 0 8793595 0 53807 0 46637 0 62925 0 54755 0 72271 0 60842 0 93343 0 47785 0 82287 0 73409 0 745000 0 832425 0 102907 0 90783 0 IR-PCI-MSI-edge p1p1-TxRx-13
84: 52045 0 52518 0 83353 0 64534 0 32414 0 35972 0 45190 0 36925 0 24817 0 79178 0 1443964 0 3712758 0 9480150 0 109740 0 82856 0 58387 0 IR-PCI-MSI-edge p1p1-TxRx-14
85: 63117 0 22871 0 220711 0 75206 0 19498 0 70962 0 52117 0 97241 0 123345 0 67058 0 115891 0 7140767 0 1527076 0 230527 0 52956 0 36916 0 IR-PCI-MSI-edge p1p1-TxRx-15
86: 341182 0 25306 0 436494 0 79810 0 336125 0 24769 0 1015036 0 56332 0 996708 0 46704 0 44728 0 171661 0 73625 0 112345 0 595442 0 1249696 0 IR-PCI-MSI-edge p1p1-TxRx-16
87: 27329 0 55574 0 348849 0 43122 0 333589 0 68221 0 146119 0 264597 0 42572 0 84245 0 217853 0 446305 0 47318 0 202998 0 423085 0 961201 0 IR-PCI-MSI-edge p1p1-TxRx-17
88: 270648 0 46600 0 217253 0 31500 0 38469 0 3095476 0 84290 0 52591 0 26967 0 34871 0 38145 0 87283 0 67059 0 270541 0 210215 0 3993171 0 IR-PCI-MSI-edge p1p1-TxRx-18
89: 148447 0 48653 0 326406 0 53179 0 123863 0 66748 0 391198 0 159050 0 98765 0 54322 0 141506 0 173276 0 1397478 0 133225 0 78736 0 119593 0 IR-PCI-MSI-edge p1p1-TxRx-19
90: 56503 0 135502 0 687704 0 380045 0 469732 0 37274 0 281894 0 1552891 0 121443 0 47009 0 173177 0 650010 0 104931 0 131578 0 217383 0 633766 0 IR-PCI-MSI-edge p1p1-TxRx-20
91: 214823 0 61146 0 339177 0 49928 0 327002 0 64010 0 318068 0 541504 0 45756 0 20992 0 176829 0 57846 0 148528 0 1162846 0 94836 0 433683 0 IR-PCI-MSI-edge p1p1-TxRx-21
92: 106435 0 26816 0 555204 0 31009 0 391325 0 125873 0 290335 0 567300 0 40470 0 341570 0 61671 0 192984 0 221575 0 3245714 0 80733 0 596397 0 IR-PCI-MSI-edge p1p1-TxRx-22
93: 115951 0 70761 0 381334 0 143441 0 872170 0 118644 0 517816 0 60193 0 100746 0 41460 0 70866 0 102305 0 134298 0 79193 0 79264 0 1835037 0 IR-PCI-MSI-edge p1p1-TxRx-23
94: 146101 0 35816 0 733832 0 29613 0 313268 0 202986 0 340662 0 100681 0 1993051 0 335538 0 71931 0 23162 0 246612 0 556509 0 224855 0 559908 0 IR-PCI-MSI-edge p1p1-TxRx-24
95: 150668 0 36783 0 251091 0 36610 0 452604 0 118424 0 197907 0 471898 0 916471 0 25373 0 43497 0 170329 0 504482 0 504129 0 61751 0 349911 0 IR-PCI-MSI-edge p1p1-TxRx-25
96: 175244 0 85547 0 149382 0 25817 0 27945 0 55507 0 297689 0 1877839 0 72682 0 294648 0 34613 0 54712 0 184015 0 5097111 0 62440 0 893488 0 IR-PCI-MSI-edge p1p1-TxRx-26
97: 188255 0 79222 0 196770 0 93331 0 62542 0 197921 0 499815 0 1489952 0 685652 0 127194 0 39956 0 48082 0 46078 0 444262 0 89668 0 197128 0 IR-PCI-MSI-edge p1p1-TxRx-27
98: 113119 0 44914 0 418302 0 17345 0 51941 0 187466 0 37948 0 572466 0 54668 0 27553 0 580539 0 366022 0 2375605 0 49572 0 73136 0 1132828 0 IR-PCI-MSI-edge p1p1-TxRx-28
99: 94404 0 56968 0 407066 0 45791 0 302591 0 106122 0 261980 0 583169 0 32802 0 50997 0 47888 0 99704 0 129685 0 315057 0 38119 0 1544062 0 IR-PCI-MSI-edge p1p1-TxRx-29
100: 98958 0 178434 0 354708 0 111329 0 159026 0 156721 0 224555 0 188416 0 224497 0 30343 0 497511 0 121221 0 280827 0 354135 0 84543 0 1650501 0 IR-PCI-MSI-edge p1p1-TxRx-30
101: 199414 0 34177 0 506856 0 92717 0 246991 0 118922 0 123314 0 261694 0 145339 0 730269 0 397150 0 26237 0 541393 0 357194 0 174133 0 520104 0 IR-PCI-MSI-edge p1p1-TxRx-31
102: 5 0 0 0 0 0 130 0 0 0 4 0 5 0 2 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 57 0 IR-PCI-MSI-edge p1p1
# 优化后
[root@localhost wangao]# egrep 'cpu|enp175s0f0' /proc/interrupts
545: 0 0 0 0 0 0 0 0 78728441 66114965 52520832 41158901 65698825 55803106 20888551 23716200 0 0 0 0 0 0 0 0 41296940 19654307 42367407 68035795 31971316 73083489 109360661 60151057 IR-PCI-MSI-edge enp175s0f0-TxRx-0
546: 0 0 0 0 0 0 0 0 293334701 57784003 57947637 279907263 83003310 59328994 14582717 10594077 0 0 0 0 0 0 0 0 33844346 53425312 104463040 121835272 48115201 11875829 149517303 27888367 IR-PCI-MSI-edge enp175s0f0-TxRx-1
547: 0 0 0 0 0 0 0 0 65087834 74714070 70617455 50729188 50629957 53222216 29320338 43600286 0 0 0 0 0 0 0 0 78790098 21664768 53133919 104396382 28219169 98109898 106275780 6075631 IR-PCI-MSI-edge enp175s0f0-TxRx-2
548: 0 0 0 0 0 0 0 0 108290552 62484073 55417923 64030173 55771760 74127972 63180417 62512564 0 0 0 0 0 0 0 0 127950399 63170356 20177522 106969671 21966002 107295156 138414635 3139534 IR-PCI-MSI-edge enp175s0f0-TxRx-3
549: 0 0 0 0 0 0 0 0 175701987 134359768 118283034 93078181 56579608 43699849 83211249 72178065 0 0 0 0 0 0 0 0 67170073 21723454 56826677 103798934 14026848 39506854 136483492 11700838 IR-PCI-MSI-edge enp175s0f0-TxRx-4
550: 0 0 0 0 0 0 0 0 161298050 70517371 30581739 186350377 170422634 226218127 55471402 95255735 0 0 0 0 0 0 0 0 158450530 36931783 14121926 68432292 26086904 5561980 46129837 10039 IR-PCI-MSI-edge enp175s0f0-TxRx-5
551: 0 0 0 0 0 0 0 0 59810450 85668650 74748733 66956510 53509915 67475372 37312571 27187643 0 0 0 0 0 0 0 0 42998302 18434014 26932131 54190593 16723043 46515600 59603049 11897912 IR-PCI-MSI-edge enp175s0f0-TxRx-6
552: 0 0 0 0 0 0 0 0 37068253 137459908 126379700 132069647 44577185 92132779 18951865 48826631 0 0 0 0 0 0 0 0 168471567 18399908 9609939 156723119 45300251 47356473 71094409 13049049 IR-PCI-MSI-edge enp175s0f0-TxRx-7
553: 0 0 0 0 0 0 0 0 9148701 15294255 337628836 28379485 21729246 18839950 5383431 5064436 0 0 0 0 0 0 0 0 45773035 6898143 3359206 4497486 569915849 5649581 659950071 561102436 IR-PCI-MSI-edge enp175s0f0-TxRx-8
554: 0 0 0 0 0 0 0 0 45919755 648235162 90862075 82511196 121536855 775131101 5059996 2245523 0 0 0 0 0 0 0 0 39913983 123180793 4250458 117911846 13217963 4291001 72759615 11901382 IR-PCI-MSI-edge enp175s0f0-TxRx-9
555: 0 0 0 0 0 0 0 0 17252329 12804726 202960565 226950817 436490841 62135664 286074659 2405684 0 0 0 0 0 0 0 0 1937801 2774288 198336045 26598791 3834288 620409019 7956742 0 IR-PCI-MSI-edge enp175s0f0-TxRx-10
556: 0 0 0 0 0 0 0 0 2543207 1589549 378184536 5537639 2432570 10278492 370935530 2557214 0 0 0 0 0 0 0 0 2084807 2817750 3214825 3352447 1633147902 3067434 3704468 0 IR-PCI-MSI-edge enp175s0f0-TxRx-11
557: 0 0 0 0 0 0 0 0 35578276 274609699 253038113 4759061 62653759 2988138 24435129 2729652 0 0 0 0 0 0 0 0 1276874078 3470696 522229642 4623985 3795035 3779577 4484373 26368 IR-PCI-MSI-edge enp175s0f0-TxRx-12
558: 0 0 0 0 0 0 0 0 3282953 3151370 13102970 15841447 2290018 191758517 253118172 3139260 0 0 0 0 0 0 0 0 2245766 6169610 1545045398 30596340 4355187 163581051 4176977 0 IR-PCI-MSI-edge enp175s0f0-TxRx-13
559: 0 0 0 0 0 0 0 0 88746890 2665364 81746473 477713921 4323711 87193204 74776878 3650446 0 0 0 0 0 0 0 0 2540382 403859013 3537468 997197460 189897196 4592593 5480200 0 IR-PCI-MSI-edge enp175s0f0-TxRx-14
560: 0 0 0 0 0 0 0 0 6646487 5048477 4430038 8497796 18754540 225111260 231266854 1499236783 0 0 0 0 0 0 0 0 138682422 4968757 27628949 22916070 3702479 28557972 20003328 0 IR-PCI-MSI-edge enp175s0f0-TxRx-15
561: 0 0 0 0 0 0 0 0 60867 79129 72520 56561 47863 58682 37248 30413 0 0 0 0 0 0 0 0 31560 22821 36684 41662 23009 40489 40755 10139346 IR-PCI-MSI-edge enp175s0f0-TxRx-16
562: 0 0 0 0 0 0 0 0 704280 2485475 1162836 815515 762812 681074 684065 578558 0 0 0 0 0 0 0 0 384355 706857 392981 350292 339121 403073 201587 19983 IR-PCI-MSI-edge enp175s0f0-TxRx-17
563: 0 0 0 0 0 0 0 0 629722 2106933 1485145 1179751 579723 864401 771639 621684 0 0 0 0 0 0 0 0 357663 563356 465863 413412 351007 341288 208065 25611 IR-PCI-MSI-edge enp175s0f0-TxRx-18
564: 0 0 0 0 0 0 0 0 823889 835158 898125 1042075 479060 649698 488703 431781 0 0 0 0 0 0 0 0 3198859 368537 361450 268475 279292 335527 175329 21693 IR-PCI-MSI-edge enp175s0f0-TxRx-19
565: 0 0 0 0 0 0 0 0 676732 1450852 1388628 704602 644689 423264 404306 365596 0 0 0 0 0 0 0 0 3038526 391864 405727 263275 191222 248226 136003 21112 IR-PCI-MSI-edge enp175s0f0-TxRx-20
566: 0 0 0 0 0 0 0 0 566595 1175664 2031641 958319 714165 319767 349309 318957 0 0 0 0 0 0 0 0 2958683 218495 239817 214993 179935 191555 106269 22241 IR-PCI-MSI-edge enp175s0f0-TxRx-21
567: 0 0 0 0 0 0 0 0 832674 1176655 1652999 789321 736090 853359 748998 577413 0 0 0 0 0 0 0 0 498610 570634 823144 451361 350612 336046 200696 23342 IR-PCI-MSI-edge enp175s0f0-TxRx-22
568: 0 0 0 0 0 0 0 0 486851 946241 1375282 525162 582995 609945 696507 440446 0 0 0 0 0 0 0 0 2986687 390852 513487 317316 283314 320359 191080 21907 IR-PCI-MSI-edge enp175s0f0-TxRx-23
569: 0 0 0 0 0 0 0 0 51211317 79773036 82956204 46567489 42244838 62523416 34798214 31871168 0 0 0 0 0 0 0 0 89233552 34741833 45475248 56079776 44626874 112466556 127468577 26730942 IR-PCI-MSI-edge enp175s0f0-TxRx-24
570: 0 0 0 0 0 0 0 0 84554805 62689462 101496218 40518441 34792544 140921853 28422396 29428544 0 0 0 0 0 0 0 0 23860917 73164249 50084153 72011910 24525127 548979217 332764098 50908113 IR-PCI-MSI-edge enp175s0f0-TxRx-25
571: 0 0 0 0 0 0 0 0 69909137 55776777 94776131 63355410 42321451 74321693 45528982 50604702 0 0 0 0 0 0 0 0 56618040 31516738 58332108 69357737 20168247 58852410 137885013 28088790 IR-PCI-MSI-edge enp175s0f0-TxRx-26
572: 0 0 0 0 0 0 0 0 97305670 87729287 88586433 59983480 55828246 79305722 51064394 44409948 0 0 0 0 0 0 0 0 41248135 60491280 43010548 99696137 32513292 64373823 74114831 49645455 IR-PCI-MSI-edge enp175s0f0-TxRx-27
573: 0 0 0 0 0 0 0 0 72108897 76540055 69546704 69542158 46167801 57170600 48419068 58439907 0 0 0 0 0 0 0 0 68912196 27371642 58904860 82050006 79475074 107799031 109654468 81082896 IR-PCI-MSI-edge enp175s0f0-TxRx-28
574: 0 0 0 0 0 0 0 0 69014002 80647341 62399685 97216594 87591440 103529815 72314579 45731155 0 0 0 0 0 0 0 0 89557655 50919154 68754628 109583064 33891553 76526249 188283182 29596358 IR-PCI-MSI-edge enp175s0f0-TxRx-29
575: 0 0 0 0 0 0 0 0 70240109 84468547 61070305 57381753 122569467 76404910 51871561 47071056 0 0 0 0 0 0 0 0 97676445 22451436 44986642 93854970 51734947 141860090 97976950 63786565 IR-PCI-MSI-edge enp175s0f0-TxRx-30
576: 0 0 0 0 0 0 0 0 66395628 99188539 63355633 82424649 110498163 119343763 129927329 49811047 0 0 0 0 0 0 0 0 76096869 78511976 32199289 122850446 108523248 49641666 346741057 101551776 IR-PCI-MSI-edge enp175s0f0-TxRx-31
577: 0 0 0 0 0 0 0 0 307 422 349 309 268 317 222 167 0 0 0 0 0 0 0 0 139 126 215 193 95 182 231 7349 IR-PCI-MSI-edge enp175s0f0
参考文章
Redis 高负载下的中断优化
网卡软中断过高问题优化总结
Linux 网络协议栈收消息过程 – TCP Protocol Layer
Monitoring and Tuning the Linux Networking Stack: Receiving Data
Performance Tuning Guide