看过Robin的一篇文章，就是反爬虫的。他提到了几种反爬虫的方法：1.手工拒绝，即爬虫的并发量相当高，那么按照80端口进行并发排序，然后手动的把爬虫的IP给禁掉。2.根据User-Agent拒绝，比如如果我们用Java程序进行爬取时，如果没有设header的话，User-Agent就是java，那么就禁掉User-Agent不为浏览器那样的请求。3.根据流量统计和日志分析来屏蔽爬虫，封掉流量特别大的爬虫。4.实时屏蔽，即如果一个IP在一段时间内请求特别频繁，就为爬虫，加入黑名单，不再响应后续请求。

高并发的爬虫却是会对网站的服务器造成很大的压力，但是有时候我们需要从ITEYE或者CSDN上爬取一些东西时，也被拒绝掉了。（CSDN博客爬取时报403拒绝请求）

很明显，1,3,4，条对我们无效，因为我们的爬取不是高并发的频繁的；第2条，User-Agent的判别，才是封掉我们爬取的真正原因。那么，我们就只能加入头部，让自己的爬取像是一个浏览器请求的样子。那么浏览器请求时，发出的是怎样的数据包呢？

我们可以写个程序在10086端口监听一下（端口你自己随便取）：

package com.JavaUtil.IESimilator; import java.io.*; import java.net.*; import java.util.*; public class IEHeaderTest {

//在端口10086上监听,得到IE发送的数据包 public IEHeaderTest() {

int port = 10086; ServerSocket serverSocket = null; Socket client = null; BufferedInputStream bis = null;

try{

serverSocket = new ServerSocket(port); client = serverSocket.accept(); bis = new BufferedInputStream(client.getInputStream()); int index = -1; byte[] buffer = new byte[1024];

while((index=bis.read(buffer))!=-1){

System.out.println(new String(buffer,0,index)); } }catch(Exception ex){

ex.printStackTrace(); }finally{

if(bis!=null){ try { bis.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } if(client!=null){

try { client.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } if(serverSocket!=null){

try { serverSocket.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } }

public static void main(String[] args) {

IEHeaderTest headerTest = new IEHeaderTest(); } }

然后再在浏览器中输入：http://localhost:10086/ 我们就可以得到IE发送的信息：

GET / HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/msword, application/vnd.ms-excel, application/vnd.ms-powerpoint, */*
Accept-Language: zh-CN
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322; .NET4.0C)
Accept-Encoding: gzip, deflate
Host: localhost:10086
Connection: Keep-Alive

我们可以看到，只有我们把User-Agent设置好，就不会出现爬取被拒绝的问题了：

package com.JavaUtil.IESimilator; import java.io.BufferedInputStream; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.io.UnsupportedEncodingException; import java.util.ArrayList; import java.util.List; import org.apache.commons.httpclient.Header; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpException; import org.apache.commons.httpclient.cookie.CookiePolicy; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.params.DefaultHttpParams; /* * author:Tammy Pi */ public class IESimilatorFetchCSDN {

private HttpClient httpClient = new HttpClient(); private GetMethod getMethod = null; private BufferedReader bis = null; private String rtn = null;

//get page public String getPage(String url){

StringBuilder sb = new StringBuilder();

getMethod = new GetMethod(url); //set http header List<Header> headers = new ArrayList<Header>(); headers.add(new Header("Accept"," image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/msword, application/vnd.ms-excel, application/vnd.ms-powerpoint, */*")); headers.add(new Header("User-Agent","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322; .NET4.0C)")); headers.add(new Header("Connection","Keep-Alive"));

//设置Cookie，解决cookie reject问题 DefaultHttpParams.getDefaultParams().setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY);

httpClient.getHostConfiguration().getParams().setParameter("http.default-headers",headers);

try { //设置编码格式 int status = httpClient.executeMethod(getMethod); System.out.println("status:"+status);

bis = new BufferedReader(new InputStreamReader(getMethod.getResponseBodyAsStream(),getMethod.getResponseCharSet())); String line = null; while((line=bis.readLine())!=null){

sb.append(line); }

try { rtn = new String(sb.toString().getBytes(getMethod.getResponseCharSet()),"utf-8"); } catch (UnsupportedEncodingException e) { // TODO Auto-generated catch block e.printStackTrace(); } } catch (HttpException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally{

if(getMethod!=null){

getMethod.releaseConnection(); } }

return rtn; }

public static void main(String[] args){

IESimilatorFetchCSDN similator = new IESimilatorFetchCSDN(); String rtn = similator.getPage("http://blog.csdn.net/hdhtqq/article/details/6088461"); System.out.println(rtn); } }