Python3如何使用多线程升程序运行速度
优化前后新老代码如下:
fromgit_tools.git_toolimportget_collect_projects,QQNews_Git fromthreadingimportThread,Lock importdatetime base_url="http://git.xx.com" project_members_commits_lang_info={} lock=Lock() threads=[] ''' Author:zenkilan ''' defcount_time(func): deftook_up_time(*args,**kwargs): start_time=datetime.datetime.now() ret=func(*args,**kwargs) end_time=datetime.datetime.now() took_up_time=(end_time-start_time).total_seconds() print(f"{func.__name__}executiontookuptime:{took_up_time}") returnret returntook_up_time defget_project_member_lang_code_lines(git,member,begin_date,end_date): globalproject_members_commits_lang_info globallock member_name=member["username"] r=git.get_user_info(member_name) ifnotr["id"]: return user_commits_lang_info=git.get_commits_user_lang_diff_between(r["id"],begin_date,end_date) iflen(user_commits_lang_info)==0: return lock.acquire() project_members_commits_lang_info.setdefault(git.project,dict()) project_members_commits_lang_info[git.project][member_name]=user_commits_lang_info lock.release() defget_project_lang_code_lines(project,begin_date,end_date): globalthreads git=QQNews_Git(project[1],base_url,project[0]) project_members=git.get_project_members() iflen(project_members)==0: return formemberinproject_members: thread=Thread(target=get_project_member_lang_code_lines,args=(git,member,begin_date,end_date)) threads.append(thread) thread.start() @count_time defget_projects_lang_code_lines(begin_date,end_date): """ 获取项目代码行语言相关统计——新方法(提升效率) 应用多线程替代for循环 并发访问共享外部资源 :return: """ globalproject_members_commits_lang_info globalthreads forprojectinget_collect_projects(): thread=Thread(target=get_project_lang_code_lines,args=(project,begin_date,end_date)) threads.append(thread) thread.start() @count_time defget_projects_lang_code_lines_old(begin_date,end_date): """ 获取项目代码行语言相关统计——老方法(耗时严重) 使用最基本的思路进行编程 双层for循环嵌套并且每层都包含耗时操作 :return: """ project_members_commits_lang_info={} forprojectinget_collect_projects(): git=QQNews_Git(project[1],base_url,project[0]) project_members=git.get_project_members() user_commits_lang_info_dict={} iflen(project_members)==0: continue formemberinproject_members: member_name=member["username"] r=git.get_user_info(member_name,debug=False) ifnotr["id"]: continue try: user_commits_lang_info=git.get_commits_user_lang_diff_between(r["id"],begin_date,end_date) iflen(user_commits_lang_info)==0: continue user_commits_lang_info_dict[member_name]=user_commits_lang_info project_members_commits_lang_info[git.project]=user_commits_lang_info_dict except: pass returnproject_members_commits_lang_info deftest_results_equal(resultA,resultB): """ 测试方法 :paramresultA: :paramresultB: :return: """ print(resultA) print(resultB) assertlen(str(resultA))==len(str(resultB)) if__name__=='__main__': fromgit_tools.configimportbegin_date,end_date get_projects_lang_code_lines(begin_date,end_date) fortinthreads: t.join() old_result=get_projects_lang_code_lines_old(begin_date,end_date) test_results_equal(old_result,project_members_commits_lang_info)
老方法里外层for循环和内层for循环里均存在耗时操作:
1)git.get_project_members()
2)git.get_user_info(member_name,debug=False)
分两步来优化,先里后外或先外后里都行。用多线程替换for循环,并发共享外部资源,加锁避免写冲突。
测试结果通过,函数运行时间装饰器显示(单位秒):
get_projects_lang_code_linesexecutiontookuptime:1.85294
get_projects_lang_code_lines_oldexecutiontookuptime:108.604177
速度提升了约58倍
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持毛票票。
声明:本文内容来源于网络,版权归原作者所有,内容由互联网用户自发贡献自行上传,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任。如果您发现有涉嫌版权的内容,欢迎发送邮件至:czq8825#qq.com(发邮件时,请将#更换为@)进行举报,并提供相关证据,一经查实,本站将立刻删除涉嫌侵权内容。