us.codecraft.webmagic.scheduler

Class RedisScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.scheduler.RedisScheduler

All Implemented Interfaces:

DuplicateRemover, MonitorableScheduler, Scheduler

Direct Known Subclasses:

RedisPriorityScheduler

public class RedisScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler, DuplicateRemover

Use Redis as url scheduler for distributed crawlers.

Since:

0.2.0

Author:

code4crafter@gmail.com

Field Summary

Fields

Modifier and Type	Field and Description
protected redis.clients.jedis.JedisPool	pool

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor and Description
RedisScheduler(redis.clients.jedis.JedisPool pool)
RedisScheduler(String host)

Method Summary

Modifier and Type	Method and Description
protected String	getItemKey(Task task)
int	getLeftRequestsCount(Task task)
protected String	getQueueKey(Task task)
protected String	getSetKey(Task task)
int	getTotalRequestsCount(Task task)
boolean	isDuplicate(Request request, Task task)
Request	poll(Task task)
protected void	pushWhenNoDuplicate(Request request, Task task)
void	resetDuplicateCheck(Task task)

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

push

Field Detail

pool

protected redis.clients.jedis.JedisPool pool

Constructor Detail

RedisScheduler

public RedisScheduler(String host)

RedisScheduler

public RedisScheduler(redis.clients.jedis.JedisPool pool)

Method Detail

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

Specified by:

resetDuplicateCheck in interface DuplicateRemover

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

Specified by:

isDuplicate in interface DuplicateRemover

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)

Overrides:

pushWhenNoDuplicate in class DuplicateRemovedScheduler

poll

public Request poll(Task task)

Specified by:

poll in interface Scheduler

getSetKey

protected String getSetKey(Task task)

getQueueKey

protected String getQueueKey(Task task)

getItemKey

protected String getItemKey(Task task)

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

Specified by:

getLeftRequestsCount in interface MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Specified by:

getTotalRequestsCount in interface DuplicateRemover

Specified by:

getTotalRequestsCount in interface MonitorableScheduler