平时爬虫一般都使用Scrapy框架,通常都是在一台机器上跑,爬取速度也不能达到预期效果,数据量小,而且很容易就会被封禁IP或者账号,这时候可以使用代理IP或者登录方式爬,然而代理IP很多时候都很鸡肋,除非使用付费版IP,但是和真实IP差别很大。这时候便有了Scrapy-redis分布式爬虫框架,它基于Scrapy改造,把Scrapy的调度器(scheduler)换成了Scrapy-redis的调度器,可以轻松达到目的,利用多台服务器来爬取数据,而且还可以自动去重,效率高。爬取的数据默认保存在redis缓存中,速度很快。
Scrapy工作原理:
Scrapy-redis工作原理:
中间的就是调度器
豆瓣电影简易分布式爬虫
我这里直接使用start_urls的方式,数据存入到Mysql中
class DoubanSpider(RedisSpider): name = 'douban' redis_key = 'douban:start_urls' allowed_domains = ['douban.com'] def start_requests(self): urls = get_urls() for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): # item_loader = MovieItemLoader(item=MovieItem, response=response) # # item_loader.add_xpath('title', '') item = MovieItem() print(response.url) item['movieId'] = int(response.url.split('subject/')[1].replace('/', '')) item['title'] = response.xpath('//h1/span/text()').extract()[0] item['year'] = response.xpath('//h1/span/text()').extract()[1].split('(')[1].split(')')[0] or '2019' item['url'] = response.url item['cover'] = response.xpath('//a[@class="nbgnbg"]/img/@src').extract()[0] try: item['director'] = response.xpath('//a[@rel="v:directedBy"]/text()').extract()[0] or '无' except Exception: item['director'] = '暂无' item['major'] = '/'.join(response.xpath('//a[@rel="v:starring"]/text()').extract()) item['category'] = ','.join(response.xpath('//span[@property="v:genre"]/text()').extract()) item['time'] = ','.join(response.xpath('//span[@property="v:initialReleaseDate"]/text()').extract()) try: item['duration'] = response.xpath('//span[@property="v:runtime"]/text()').extract()[0] except Exception: item['duration'] = '暂无' item['score'] = response.xpath('//strong[@property="v:average"]/text()').extract()[0] item['comment_nums'] = response.xpath('//span[@property="v:votes"]/text()').extract()[0] or 0 item['desc'] = response.xpath('//span[@property="v:summary"]/text()').extract()[0].strip() actor_list = response.xpath('//ul[@class="celebrities-list from-subject __oneline"]/li/a/@title').extract() actor_img_list = response.xpath('//ul[@class="celebrities-list from-subject __oneline"]/li/a/div/@style'). extract() actor_img_list = [i.split('url(')[1].replace(')', '') for i in actor_img_list] item['actor_name_list'] = '----'.join(actor_list) item['actor_img_list'] = '----'.join(actor_img_list) yield item
settings.py文件
BOT_NAME = 'MovieSpider' SPIDER_MODULES = ['MovieSpider.spiders'] NEWSPIDER_MODULE = 'MovieSpider.spiders' # REDIS_HOST = '127.0.0.1' # REDIS_PORT = 6379 REDIS_URL = 'redis://username:password@xxx.xxx.xxx.xxx:6379' # Obey robots.txt rules ROBOTSTXT_OBEY = False SCHEDULER = "scrapy_redis.scheduler.Scheduler" # Ensure all spiders share same duplicates filter through redis. DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" ITEM_PIPELINES = { 'scrapy_redis.pipelines.RedisPipeline': 300, 'MovieSpider.pipelines.MysqlPipeline': 200, }
这里只是为了多台服务器一起爬取,没有手动在redis中推入起始的URL
此时将爬虫项目上传到其他服务器上,一起开始
效果如下:
神龙|纯净稳定代理IP免费测试>>>>>>>>天启|企业级代理IP免费测试>>>>>>>>IPIPGO|全球住宅代理IP免费测试