問題描述

我想把進程池封裝在裝飾器里，但是它既沒生效也沒報錯

# coding:utf-8import multiprocessingimport tornadofrom tornado.httpclient import AsyncHTTPClientprocess_num = 20 # 進程數url = 'https://www.baidu.com'def handle_request(response): print str(response)def run_in_process(process_num): def _run_in_process(f):def __run_in_process(*args, **kwargs): pool = multiprocessing.Pool(processes=process_num) for i in range(process_num):pool.apply_async(f, args=args, kwds=kwargs, callback=kwargs.get('callback')) pool.close() pool.join()return __run_in_process return _run_in_process@run_in_process(process_num)def main(): http_client = AsyncHTTPClient() http_client.fetch(url, callback=handle_request) global loop loop = tornado.ioloop.IOLoop.instance() if loop._running is False:loop.start()if __name__ == ’__main__’: main()

結果如下

/usr/bin/python2.7 /home/xxx/workspace/py_project/crawler/center/sample.pyProcess finished with exit code 0

但是奇怪的是，我用多進程的方式重寫一次，發現是可以生效的

# coding:utf-8import multiprocessingimport tornadofrom tornado.httpclient import AsyncHTTPClientprocess_num = 20 # 進程數url = 'https://www.baidu.com'def handle_request(response): print str(response)def run_in_process(process_num): def _run_in_process(f):def __run_in_process(*args, **kwargs): _processes = [] for i in xrange(process_num):p = multiprocessing.Process(target=f, args=args, kwargs=kwargs)p.start()_processes.append(p) for p in _processes:p.join()return __run_in_process return _run_in_process@run_in_process(process_num)def main(): http_client = AsyncHTTPClient() http_client.fetch(url, callback=handle_request) global loop loop = tornado.ioloop.IOLoop.instance() if loop._running is False:loop.start()if __name__ == ’__main__’: main()

日志如下

/usr/bin/python2.7 /home/shufeng/workspace/private_project/jobscrawler/center/sample.pyHTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url=’http://www.baidu.com’,error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa425d0>,reason=’OK’,request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa42250>,request_time=0.014312028884887695,time_info={})HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url=’http://www.baidu.com’,error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa43450>,reason=’OK’,request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa430d0>,request_time=0.02327895164489746,time_info={})HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url=’http://www.baidu.com’,error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa43510>,reason=’OK’,request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa43190>,request_time=0.026951074600219727,time_info={})HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url=’http://www.baidu.com’,error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa42690>,reason=’OK’,request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa42310>,request_time=0.0552978515625,time_info={})HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa24ef0>,code=200,effective_url=’http://www.baidu.com’,error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa39e10>,reason=’OK’,request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa39a90>,request_time=0.05612993240356445,time_info={})

同樣的情況也會出現在線程池跟協程的使用上，有誰知道這是怎么回事嗎？

問題解答

知乎靈劍大神已回答此問題：https://www.zhihu.com/questio...

在linux下運行, 會得到下面的報錯:

PicklingError: Can’t pickle <type ’function’>: attribute lookup __builtin__.function failed

而這個報錯是因為, 傳入不可序列化的對象進進程池時, 報錯導致的, 而這個對象就是實例方法, 可以試下用py3運行下, 因為3的實例方法已經可以支持序列化

參考資料: https://virusdefender.net/ind...