关于lock:Python3-的进程池与锁qbit

前言

• 测试环境

Windows 10
Python 3.8.2
loguru 0.5.3

• Python 3.8 多过程官网文档: https://docs.python.org/3.8/l…
• loguru 0.5.3 不是多过程平安的，正好能够用来做多过程与锁的测试
• 参考文章：Python sharing a lock between processes

代码与阐明

无锁多过程

• 代码

# encoding: utf-8
# author: qbit
# date: 2021-01-14
# summary: 无锁多过程测试
from multiprocessing import Pool
from loguru import logger
def ProcOne(ch):
    logger.info(ch*20)
    return ch
def Main():
    line = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    with Pool(4) as p:
        results = p.map(func=ProcOne, iterable=line)
        print(repr(results))
if __name__ == "__main__":
    Main()

• 运行后果（乱）

multiprocessing.Lock

• 代码

# encoding: utf-8
# author: qbit
# date: 2021-01-14
# summary: multiprocessing.Lock 测试
import os
from multiprocessing import Pool, Lock
from functools import partial
from loguru import logger
def ProcOne(lock, ch):
    with lock:
        logger.info(f'{os.getpid()}-{id(lock)}-{ch*20}')
    return ch
def Main():
    line = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    lock = Lock()
    func = partial(ProcOne, lock)
    with Pool(4) as p:
        results = p.map(func=func, iterable=line)
        print(repr(results))
if __name__ == "__main__":
    Main()

• 运行后果（报错）

• 报错的原始是 multiprocessing.Lock 不能被序列化（pickle）

multiprocessing.Manager

• 代码

# encoding: utf-8
# author: qbit
# date: 2021-01-14
# summary: multiprocessing.Manager 测试
import os
from multiprocessing import Pool, Manager
from functools import partial
from loguru import logger
def ProcOne(lock, ch):
    with lock:
        logger.info(f'{os.getpid()}-{id(lock)}-{ch*20}')
    return ch
def Main():
    line = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    lock = Manager().Lock()
    func = partial(ProcOne, lock)
    with Pool(4) as p:
        results = p.map(func=func, iterable=line)
        print(repr(results))
if __name__ == "__main__":
    Main()

• 运行后果（工整）

• 从图中能够看到，锁的对象 id 在不同的过程中是不同的

multiprocessing.Lock 改进版

• 代码

# encoding: utf-8
# author: qbit
# date: 2021-01-14
# summary: multiprocessing.Lock 改良测试
import os
from multiprocessing import Pool, Lock
from loguru import logger
def ProcOne(ch):
    with glock:
        logger.info(f'{os.getpid()}-{id(glock)}-{ch*20}')
    return ch
def init(lock: Lock):
    global glock
    glock = lock
def Main():
    line = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    lock = Lock()
    with Pool(4, initializer=init, initargs=(lock,)) as p:
        results = p.map(func=ProcOne, iterable=line)
        print(repr(results))
if __name__ == "__main__":
    Main()

• 运行后果（工整）

• 从图中能够看到，锁的对象 id 在不同的过程中还是不同，但从运行后果来看 lock 对象是同一个
• 如果简略的将 lock 申明为全局对象在 Windows 下并不可行，因为 Windows 不反对 fork，每个过程将会失去不同的 lock 对象
• 实测在 Linux 中对象 id 雷同

本文出自 qbit snap