Scrapy 配置动态代理IP的实现
应用Scrapy框架,配置动态IP处理反爬。
#settings配置中间件
DOWNLOADER_MIDDLEWARES={
'text.middlewares.TextDownloaderMiddleware':543,
#'text.middlewares.RandomUserAgentMiddleware':544,
#'text.middlewares.CheckUserAgentMiddleware':545,
'text.middlewares.ProxyMiddleware':546,
'text.middlewares.CheckProxyMiddleware':547
}
#settings配置可用动态IP
PROXIES=[
"http://101.231.104.82:80",
"http://39.137.69.6:8080",
"http://39.137.69.10:8080",
"http://39.137.69.7:80",
"http://39.137.77.66:8080",
"http://117.191.11.102:80",
"http://117.191.11.113:8080",
"http://117.191.11.113:80",
"http://120.210.219.103:8080",
"http://120.210.219.104:80",
"http://120.210.219.102:80",
"http://119.41.236.180:8010",
"http://117.191.11.80:8080"
]
#middlewares配置中间件
importrandom
classProxyMiddleware(object):
defprocess_request(self,request,spider):
ip=random.choice(spider.settings.get('PROXIES'))
print('测试IP:',ip)
request.meta['proxy']=ip
classCheckProxyMiddleware(object):
defprocess_response(self,request,response,spider):
print('代理IP:',request.meta['proxy'])
returnresponse
到此这篇关于Scrapy配置动态代理IP的实现的文章就介绍到这了,更多相关Scrapy动态代理IP内容请搜索毛票票以前的文章或继续浏览下面的相关文章希望大家以后多多支持毛票票!
热门推荐
1 训练狗狗憋尿
2 狗狗追踪训练
3 狗狗训练要多少钱
5 深圳狗狗训练
6 训练狗狗唤回
7 狗狗叫怎么训练
8 狗狗训练班
10 狗狗社会化训练
11 狗狗叫回训练
12 广州狗狗训练学校
13 狗狗训练躺下
14 狗狗随行训练
15 训练狗狗巡回
16 狗狗要怎么训练
17 狗狗喂食训练
18 训练狗狗躺下