cn.edu.hfut.dmic.webcollector.crawler

类 AutoParseCrawler

java.lang.Object

cn.edu.hfut.dmic.webcollector.crawler.Crawler

cn.edu.hfut.dmic.webcollector.crawler.AutoParseCrawler

所有已实现的接口:

Executor, Visitor, Requester

直接已知子类:

BreadthCrawler, RamCrawler

public abstract class AutoParseCrawler
extends Crawler
implements Executor, Visitor, Requester

作者:

hu

字段概要

字段

限定符和类型	字段和说明
protected boolean	autoParse 是否自动抽取符合正则的链接并加入后续任务
static org.slf4j.Logger	LOG
protected boolean	parseImg
protected RegexRule	regexRule URL正则约束
protected Requester	requester
protected Visitor	visitor

从类继承的字段 cn.edu.hfut.dmic.webcollector.crawler.Crawler

dbManager, executeInterval, executor, fetcher, forcedSeeds, maxExecuteCount, nextFilter, resumable, RUNNING, seeds, status, STOPED, threads, topN

构造器概要

构造器

构造器和说明
AutoParseCrawler(boolean autoParse)

方法概要

限定符和类型	方法和说明
void	addRegex(String urlRegex) 添加URL正则约束
protected void	afterParse(Page page, CrawlDatums next)
void	execute(CrawlDatum datum, CrawlDatums next)
RegexRule	getRegexRule() 获取正则规则
Requester	getRequester()
HttpResponse	getResponse(CrawlDatum crawlDatum)
Visitor	getVisitor() 获取Visitor
boolean	isAutoParse()
boolean	isParseImg()
protected void	parseLink(Page page, CrawlDatums next)
void	setAutoParse(boolean autoParse) 设置是否自动抽取符合正则的链接并加入后续任务
void	setParseImg(boolean parseImg)
void	setRegexRule(RegexRule regexRule) 设置正则规则
void	setRequester(Requester requester)
void	setVisitor(Visitor visitor) 设置Visitor

从类继承的方法 cn.edu.hfut.dmic.webcollector.crawler.Crawler

addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, addSeed, getDBManager, getExecuteInterval, getExecutor, getMaxExecuteCount, getNextFilter, getThreads, getTopN, inject, injectForcedSeeds, isResumable, setDBManager, setExecuteInterval, setExecutor, setMaxExecuteCount, setNextFilter, setResumable, setThreads, setTopN, start, stop, toString

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

从接口继承的方法 cn.edu.hfut.dmic.webcollector.fetcher.Visitor

visit

字段详细资料

LOG

public static final org.slf4j.Logger LOG

autoParse

protected boolean autoParse

是否自动抽取符合正则的链接并加入后续任务

parseImg

protected boolean parseImg

visitor

protected Visitor visitor

requester

protected Requester requester

regexRule

protected RegexRule regexRule

URL正则约束

构造器详细资料

AutoParseCrawler

public AutoParseCrawler(boolean autoParse)

方法详细资料

getResponse

public HttpResponse getResponse(CrawlDatum crawlDatum)
                         throws Exception

指定者:

getResponse 在接口中 Requester

抛出:

Exception

execute

public void execute(CrawlDatum datum,
                    CrawlDatums next)
             throws Exception

指定者:

execute 在接口中 Executor

抛出:

Exception

afterParse

protected void afterParse(Page page,
                          CrawlDatums next)

parseLink

protected void parseLink(Page page,
                         CrawlDatums next)

addRegex

public void addRegex(String urlRegex)

添加URL正则约束

参数:

urlRegex - URL正则约束

isAutoParse

public boolean isAutoParse()

返回:

返回是否自动抽取符合正则的链接并加入后续任务

setAutoParse

public void setAutoParse(boolean autoParse)

设置是否自动抽取符合正则的链接并加入后续任务

参数:

autoParse - 是否自动抽取符合正则的链接并加入后续任务

getRegexRule

public RegexRule getRegexRule()

获取正则规则

返回:

正则规则

setRegexRule

public void setRegexRule(RegexRule regexRule)

设置正则规则

参数:

regexRule - 正则规则

getVisitor

public Visitor getVisitor()

获取Visitor

返回:

Visitor

setVisitor

public void setVisitor(Visitor visitor)

设置Visitor

参数:

visitor - Visitor

getRequester

public Requester getRequester()

setRequester

public void setRequester(Requester requester)

isParseImg

public boolean isParseImg()

setParseImg

public void setParseImg(boolean parseImg)