keras tensorflow 实现在python下多进程运行
如下所示：
frommultiprocessingimportProcess importos deftraining_function(...): importkeras#此处需要在子进程中 ... if__name__=='__main__': p=Process(target=training_function,args=(...,)) p.start()
原文地址：https://stackoverflow.com/questions/42504669/keras-tensorflow-and-multiprocessing-in-python
1、DONOTLOADKERASTOYOURMAINENVIRONMENT.IfyouwanttoloadKeras/Theano/TensorFlowdoitonlyinthefunctionenvironment.E.g.don'tdothis:
importkeras deftraining_function(...): ...
butdothefollowing:
deftraining_function(...): importkeras ...
Runworkconnectedwitheachmodelinaseparateprocess:I'musuallycreatingworkerswhicharemakingthejob(likee.g.training,tuning,scoring)andI'mrunningtheminseparateprocesses.Whatisniceaboutitthatwholememoryusedbythisprocessiscompletelyfreedwhenyourprocessisdone.Thishelpsyouwithloadsofmemoryproblemswhichyouusuallycomeacrosswhenyouareusingmultiprocessingorevenrunningmultiplemodelsinoneprocess.Sothislookse.g.likethis:
def_training_worker(train_params): importkeras model=obtain_model(train_params) model.fit(train_params) send_message_to_main_process(...) deftrain_new_model(train_params): training_process=multiprocessing.Process(target=_training_worker,args=train_params) training_process.start() get_message_from_training_process(...) training_process.join()
Differentapproachissimplypreparingdifferentscriptsfordifferentmodelactions.Butthismaycausememoryerrorsespeciallywhenyourmodelsarememoryconsuming.NOTEthatduetothisreasonit'sbettertomakeyourexecutionstrictlysequential.
以上这篇kerastensorflow实现在python下多进程运行就是小编分享给大家的全部内容了，希望能给大家一个参考，也希望大家多多支持毛票票。
声明：本文内容来源于网络，版权归原作者所有，内容由互联网用户自发贡献自行上传，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任。如果您发现有涉嫌版权的内容，欢迎发送邮件至：czq8825#qq.com（发邮件时，请将#更换为@）进行举报，并提供相关证据，一经查实，本站将立刻删除涉嫌侵权内容。