在使用python爬虫进行网络页面爬取的过程中,第一步肯定是要爬取url,若是面对网页中很多url,,又该如何爬取所以url呢?本文介绍Python爬虫爬取网页中所有的url的三种实现方法:1、使用BeautifulSoup快速提取所有url;2、使用Scrapy框架递归调用parse;3、在get_next_url()函数中调用自身,递归循环爬取所有url。
方法一:使用BeautifulSoup快速提取所有url
BeautifulSoup是一种可以从html和xml中快速提取内容的python库def getAllUrl(self,url):
import urllib.request
from bs4 import BeautifulSoup
html = urllib.request.urlopen(url).read().decode("utf-8")
soup = BeautifulSoup(html, features='html.parser')
tags = soup.find_all('a')
for tag in tags:
print(str(tag.get('href')).strip())
方法二:使用Scrapy框架递归调用parse
递归调用parse直到每个页面爬取完class QiubaiSpider(scrapy.Spider):
name = 'qiubai'
# allowed_domains = ['www.qiushibaike.com/text']
start_urls = ['https://www.qiushibaike.com/text/']
# 设计一个url模板
url = 'https://www.qiushibaike.com/text/page/%d/'
pageNum = 1
def parse(self, response):
div_list = response.xpath("//div[@id='content-left']/div")
for div in div_list:
….
# 将item提交给管道
yield item
# 多url, 请求的手动发送
if self.pageNum <= 13: # 控制!否则无限递归了。。
self.pageNum += 1
print('爬第:%d 页' % self.pageNum)
new_url = self.url % self.pageNum
# callback 回调函数,页面进行解析
yield scrapy.Request(url=new_url, callback=self.parse)
方法三:在get_next_url()函数中调用自身,递归循环爬取所有url#!/usr/bin/env python
# -*- coding:utf-8 -*-
import requests
from bs4 import BeautifulSoup as Bs4
head_url = "http://www.xxx.com.cn"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/72.0.3626.121 Safari/537.36"
}
def get_first_url():
list_href = []
reaponse = requests.get(head_url, headers=headers)
soup = Bs4(reaponse.text, "lxml")
urls_li = soup.select("#mainmenu_top > div > div > ul > li")
for url_li in urls_li:
urls = url_li.select("a")
for url in urls:
url_href = url.get("href")
list_href.append(head_url+url_href)
out_url = list(set(list_href))
return out_url
def get_next_url(urllist):
url_list = []
for url in urllist:
response = requests.get(url,headers=headers)
soup = Bs4(response.text,"lxml")
urls = soup.find_all("a")
if urls:
for url2 in urls:
url2_1 = url2.get("href")
if url2_1:
if url2_1[0] == "/":
url2_1 = head_url + url2_1
url_list.append(url2_1)
if url2_1[0:24] == "http://www.xxx.com.cn":
url2_1 = url2_1
url_list.append(url2_1)
else:
pass
else:
pass
else:
pass
else:
pass
url_list2 = set(url_list)
for url_ in url_list2:
res = requests.get(url_)
if res.status_code ==200:
print(url_)
print(len(url_list2))
get_next_url(url_list2)
if __name__ == "__main__":
urllist = get_first_url()
get_next_url(urllist)
以上就是Python爬虫爬取网页中所有的url的三种实现方法,希望能对你有所帮助哟~更多python爬虫进阶推荐:python爬虫进阶教程。
神龙|纯净稳定代理IP免费测试>>>>>>>>天启|企业级代理IP免费测试>>>>>>>>IPIPGO|全球住宅代理IP免费测试