poolboy max_overflow 引发的血案

问题这是个线上问题. 某个服务节点在较低的qps(每秒2000次数据库访问)下, 在worker进程数100, max_overflow进程数100的情况下. 突然性能下降, 每秒只能处理1500次数据库访问. 导致请求处理延时从几MS上升至几百MS, 之后又逐渐恢复.原因逐渐把范围缩小至 mongodb poolboy 进程池的 checkout:check outhandle_call({checkout, CRef, Block}, {FromPid, _} = From, State) -> #state{supervisor = Sup, workers = Workers, monitors = Monitors, overflow = Overflow, max_overflow = MaxOverflow} = State, case Workers of [Pid | Left] -> MRef = erlang:monitor(process, FromPid), true = ets:insert(Monitors, {Pid, CRef, MRef}), {reply, Pid, State#state{workers = Left}}; [] when MaxOverflow > 0, Overflow < MaxOverflow -> {Pid, MRef} = new_worker(Sup, FromPid), true = ets:insert(Monitors, {Pid, CRef, MRef}), {reply, Pid, State#state{overflow = Overflow + 1}}; [] when Block =:= false -> {reply, full, State}; [] -> MRef = erlang:monitor(process, FromPid), Waiting = queue:in({From, CRef, MRef}, State#state.waiting), {noreply, State#state{waiting = Waiting}} end;可以看到, 当max_overflow不为0时, 瞬间过载会创建新的worker, 而这些worker, 都会去链接mongodb, 耗时1-2MS. 创建的消耗会阻塞master process.check in而归还时, 又会将worker销毁, 导致链接一直创建/销毁, 而且都卡在master process, 这导致所有的请求, 都会因master process的链接创建和销毁而阻塞, 导致qps雪崩下降.handle_checkin(Pid, State) -> #state{supervisor = Sup, waiting = Waiting, monitors = Monitors, overflow = Overflow, strategy = Strategy} = State, case queue:out(Waiting) of {{value, {From, CRef, MRef}}, Left} -> true = ets:insert(Monitors, {Pid, CRef, MRef}), gen_server:reply(From, Pid), State#state{waiting = Left}; {empty, Empty} when Overflow > 0 -> ok = dismiss_worker(Sup, Pid), State#state{waiting = Empty, overflow = Overflow - 1}; {empty, Empty} -> Workers = case Strategy of lifo -> [Pid | State#state.workers]; fifo -> State#state.workers ++ [Pid] end, State#state{workers = Workers, waiting = Empty, overflow = 0} end.结论不要使用 poolboy 的 max_overflow, 若创建/销毁 children process时有一定消耗, 很容易阻塞 poolboy master进程, 频繁创建/销毁 worker 导致雪崩.每次查BUG, 回头看来都是理所当然. 追查时却要费一番心思, 监控数据不便在个人blog给出. 不免省掉很多推断过程, 希望这个结论对大家有帮助.