详解python并发获取snmp信息及性能测试
python&snmp
用python获取snmp信息有多个现成的库可以使用，其中比较常用的是netsnmp和pysnmp两个库。网上有较多的关于两个库的例子。
本文重点在于如何并发的获取snmp的数据，即同时获取多台机器的snmp信息。
netsnmp
先说netsnmp。python的netsnmp，其实是来自于net-snmp包。
python通过一个c文件调用net-snmp的接口获取数据。
因此，在并发获取多台机器的时候，不能够使用协程获取。因为使用协程，在get数据的时候，协程会一直等待net-snmp接口返回数据，而不会像socket使用时那样在等待数据时把CPU切换给其他协程使用。从这点上来说，使用协程和串行获取没有区别。
那么如何解决并发获取的问题呢？可以使用线程，多线程获取(当然也可以使用多进程)。多个线程同时调用net-snmp的接口获取数据，然后cpu在多个线程之间不停切换。当一个线程获取一个结果后，可以继续调用接口获取下一个snmp数据。
这里我写了一个样例程序。首先把所有的host和oid做成任务放到队列里，然后启动多个线程，去执行获取任务。程序样例如下：
importthreading
importtime
importnetsnmp
importQueue
start_time=time.time()
hosts=["192.20.150.109","192.20.150.110","192.20.150.111","192.20.150.112","192.20.150.113","192.20.150.114",
"192.20.150.115","192.20.150.116","192.20.150.117","192.20.150.118","192.20.150.119","192.20.150.120",
"192.20.150.121","192.20.80.148","192.20.80.149","192.20.96.59","192.20.82.14","192.20.82.15",
"192.20.82.17","192.20.82.19","192.20.82.12","192.20.80.139","192.20.80.137","192.20.80.136",
"192.20.80.134","192.20.80.133","192.20.80.131","192.20.80.130","192.20.81.141","192.20.81.140",
"192.20.82.26","192.20.82.28","192.20.82.23","192.20.82.21","192.20.80.128","192.20.80.127",
"192.20.80.122","192.20.81.159","192.20.80.121","192.20.80.124","192.20.81.151","192.20.80.118",
"192.20.80.119","192.20.80.113","192.20.80.112","192.20.80.116","192.20.80.115","192.20.78.62",
"192.20.81.124","192.20.81.125","192.20.81.122","192.20.81.121","192.20.82.33","192.20.82.31",
"192.20.82.32","192.20.82.30","192.20.81.128","192.20.82.39","192.20.82.37","192.20.82.35",
"192.20.81.130","192.20.80.200","192.20.81.136","192.20.81.137","192.20.81.131","192.20.81.133",
"192.20.81.134","192.20.82.43","192.20.82.45","192.20.82.41","192.20.79.152","192.20.79.155",
"192.20.79.154","192.25.76.235","192.25.76.234","192.25.76.233","192.25.76.232","192.25.76.231",
"192.25.76.228","192.25.20.96","192.25.20.95","192.25.20.94","192.25.20.93","192.24.163.14",
"192.24.163.21","192.24.163.29","192.24.163.6","192.18.136.22","192.18.136.23","192.24.193.2",
"192.24.193.19","192.24.193.18","192.24.193.11","192.20.157.132","192.20.157.133","192.24.212.232",
"192.24.212.231","192.24.212.230"]
oids=[".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1",
".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0",
".1.3.6.1.4.1.2021.4.15.0"]
myq=Queue.Queue()
rq=Queue.Queue()
#把host和oid组成任务
forhostinhosts:
foroidinoids:
myq.put((host,oid))
defpoll_one_host():
whileTrue:
try:
#死循环从队列中获取任务，直到队列任务为空
host,oid=myq.get(block=False)
session=netsnmp.Session(Version=2,DestHost=host,Community="cluster",Timeout=3000000,Retries=0)
var_list=netsnmp.VarList()
var_list.append(netsnmp.Varbind(oid))
ret=session.get(var_list)
rq.put((host,oid,ret,(time.time()-start_time)))
exceptQueue.Empty:
break
thread_arr=[]
#开启多线程
num_thread=50
foriinrange(num_thread):
t=threading.Thread(target=poll_one_host,kwargs={})
t.setDaemon(True)
t.start()
thread_arr.append(t)
#等待任务执行完毕
foriinrange(num_thread):
thread_arr[i].join()
whileTrue:
try:
info=rq.get(block=False)
printinfo
exceptQueue.Empty:
printtime.time()-start_time
break
netsnmp除了支持get操作之外，还支持walk操作，即遍历某个oid。
但是walk使用的时候需要谨慎，以免导致高延时等问题，具体可以参见之前的一篇snmpwalk高延时问题分析的博客。
pysnmp
pysnmp是用python实现的一套snmp协议的库。其自身提供了对于异步的支持。
importtime
importQueue
frompysnmp.hlapi.asyncoreimport*
t=time.time()
myq=Queue.Queue()
#回调函数。在有数据返回时触发
defcbFun(snmpEngine,sendRequestHandle,errorIndication,errorStatus,errorIndex,varBinds,cbCtx):
myq.put((time.time()-t,varBinds))
hosts=["192.20.150.109","192.20.150.110","192.20.150.111","192.20.150.112","192.20.150.113","192.20.150.114",
"192.20.150.115","192.20.150.116","192.20.150.117","192.20.150.118","192.20.150.119","192.20.150.120",
"192.20.150.121","192.20.80.148","192.20.80.149","192.20.96.59","192.20.82.14","192.20.82.15",
"192.20.82.17","192.20.82.19","192.20.82.12","192.20.80.139","192.20.80.137","192.20.80.136",
"192.20.80.134","192.20.80.133","192.20.80.131","192.20.80.130","192.20.81.141","192.20.81.140",
"192.20.82.26","192.20.82.28","192.20.82.23","192.20.82.21","192.20.80.128","192.20.80.127",
"192.20.80.122","192.20.81.159","192.20.80.121","192.20.80.124","192.20.81.151","192.20.80.118",
"192.20.80.119","192.20.80.113","192.20.80.112","192.20.80.116","192.20.80.115","192.20.78.62",
"192.20.81.124","192.20.81.125","192.20.81.122","192.20.81.121","192.20.82.33","192.20.82.31",
"192.20.82.32","192.20.82.30","192.20.81.128","192.20.82.39","192.20.82.37","192.20.82.35",
"192.20.81.130","192.20.80.200","192.20.81.136","192.20.81.137","192.20.81.131","192.20.81.133",
"192.20.81.134","192.20.82.43","192.20.82.45","192.20.82.41","192.20.79.152","192.20.79.155",
"192.20.79.154","192.25.76.235","192.25.76.234","192.25.76.233","192.25.76.232","192.25.76.231",
"192.25.76.228","192.25.20.96","192.25.20.95","192.25.20.94","192.25.20.93","192.24.163.14",
"192.24.163.21","192.24.163.29","192.24.163.6","192.18.136.22","192.18.136.23","192.24.193.2",
"192.24.193.19","192.24.193.18","192.24.193.11","192.20.157.132","192.20.157.133","192.24.212.232",
"192.24.212.231","192.24.212.230"]
oids=[".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1",
".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0",
".1.3.6.1.4.1.2021.4.15.0"]
snmpEngine=SnmpEngine()
#添加任务
foroidinoids:
forhinhosts:
getCmd(snmpEngine,
CommunityData('cluster'),
UdpTransportTarget((h,161),timeout=3,retries=0,),
ContextData(),
ObjectType(ObjectIdentity(oid)),
cbFun=cbFun)
time1=time.time()-t
#执行异步获取snmp
snmpEngine.transportDispatcher.runDispatcher()
#打印结果
whileTrue:
try:
info=myq.get(block=False)
printinfo
exceptQueue.Empty:
printtime1
printtime.time()-t
break
pysnmp本身只支持最基础的get和getnext命令，因此如果想使用walk，需要自己进行实现。
性能测试
在同一个环境下，对两者进行了性能测试。两者对198个host，10个oid进行采集。
| 测试组 | 耗时(sec) | netsnmp(20线程) | 6.252 | netsnmp(50线程) | 3.269 | netsnmp(200线程) | 3.265 | pysnmp | 4.812 |
|---|
可以看到netsnmp的采集速度跟线程数有关。当线程数增大到一定程度，采集时间不再缩短。因为开辟线程同样会消耗时间。而已有的线程已经足够处理。
pysnmp性能较之略差一下。详细分析pysnmp在添加任务(执行getCmd时)消耗了约1.2s，之后的采集约消耗3.3秒。
在增加了oid数，在进行实验。host仍然是198个，oid是42个。
| 测试组 | 耗时(sec) | netsnmp(20线程) | 30.935 | netsnmp(50线程) | 12.914 | netsnmp(200线程) | 4.044 | pysnmp | 11.043 |
|---|
可以看到差距被进一步拉大。在线程足够多的情况下，netsnmp的效率要明显强于pysnmp。
因为二者都支持可以并行采集多个host，从易用性来说，netsnmp更为简单一些，且netsnmp支持walk功能。本文更加推荐netsnmp。
安装netsnmp需要安装net-snmp。如果centos，则使用yum会较为方便。
以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持毛票票。