一.scrapy环境搭建,参考我的博客–>爬虫框架虚拟环境搭建

二.scrapy设置配置

1.设置用户代理

进入页面并刷新,进入开发者模式,点击选中一个网页,在Network-Headers中找到USER_AGENT,并复制就可以了.

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'

2.是否遵守爬虫协议改为否(原因你懂的)

ROBOTSTXT_OBEY = False

3.一次允许的最大请求数

# Configure maximum concurrent requests performed by Scrapy (default: 16) CONCURRENT_REQUESTS = 2

4.设置下载延迟时间,因而使得爬虫更像是人的行为,避免IP被屏蔽

DOWNLOAD_DELAY = 3

5.设置下载中间键

DOWNLOADER_MIDDLEWARES = { 'xymtest.middlewares.XymtestDownloaderMiddleware': 543, }

6.设置管道

ITEM_PIPELINES = { 'xymtest.pipelines.XymtestPipeline': 300, }

7.取消最后几行的注释

HTTPCACHE_ENABLED = True HTTPCACHE_EXPIRATION_SECS = 0 HTTPCACHE_DIR = 'httpcache' HTTPCACHE_IGNORE_HTTP_CODES = [] HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

三.开始编写爬虫代码

1.设置要爬取得items

# define the fields for your item here like: title = scrapy.Field() content = scrapy.Field()

2.在spider下创建一个.py文件,编写爬虫代码.

class testInformation(scrapy.Spider): name = 'test' #域名后面跟的一串数字不要跟上来 allowed_domains = ['blog.test.net'] #the format of different page's address is https://blog.test.net/u42/article/list/ follows with a number,https://blog.test.net/u042/article/list/1 means the first page, https://blog.test.net/u014229742/article/list/2 is the second.so we can use a the same part plus a number,but we can not add a int type with a string,so we change the number to str

start_urls = ['https://blog.test.net/u42/article/list/' + str(x) for x in range(1, 4)]

#函数 def parse(self, response):

#get the xpath of the title: #the first title xpath is://*[@id="mainBox"]/main/div[2]/div[1]/h4/a #the seconde title xpath is://*[@id="mainBox"]/main/div[2]/div[2]/h4/a #the same part is://*[@id="mainBox"]/main/div[2],and from the next div everything is different.div[1] means the first title's xpath.div[2] means the second title's xpath. #so if when want to get all the xpath,we can use://*[@id="mainBox"]/main/div[2]/div #获取到所有标题的xpath li_list = response.xpath('//*[@id="mainBox"]/main/div[2]/div')

#we have to get all the title,so there must has a recycle,xq means one of the for xq in li_list: item = XymtestItem() #获取到标题内容//*[@id="mainBox"]/main/div[2]/div[1]/h4/a/text() #//*[@id="mainBox"]/main/div[2]/div[2]/h4/a/text() #获取到的标题去掉li_list中的公共部分 item_list = xq.xpath('h4/a/text()').extract() #因为获取到的item_list有空的内容,如果直接extract()[0],会报错,故先判断长度,长度不为空,开始取标题 if len(item_list) > 0: #strip()函数可以去除空格 item['title'] = item_list[1].strip() #获取到每个标题的href内容 url = xq.xpath('h4/a/@href').extract()[0] #Request(url, meta={'item': item}, callback = self.parse_detail)方法实现二层链接函数的调取 yield Request(url, meta={'item': item}, callback = self.parse_detail)

def parse_detail(self, response):

item = response.meta['item'] #获取到二层链接中要爬取的页面的xpath item['content'] = response.xpath('//*[@id="mainBox"]/main/div[1]/div[2]/div/div/span/text()').extract()[0]

yield item 好了,以上代码基本实现了一个二层链接的爬取,接下来要做的事将爬取到的数据存储到数据库中供我们使用.想知道更多,继续关注小姐姐!