us.codecraft.webmagic.scheduler

Class BloomFilterDuplicateRemover

java.lang.Object

us.codecraft.webmagic.scheduler.BloomFilterDuplicateRemover

All Implemented Interfaces:

DuplicateRemover

public class BloomFilterDuplicateRemover
extends Object
implements DuplicateRemover

BloomFilterDuplicateRemover for huge number of urls.

Since:

0.5.1

Author:

code4crafer@gmail.com

Constructor Summary

Constructors

Constructor and Description
BloomFilterDuplicateRemover(int expectedInsertions)
BloomFilterDuplicateRemover(int expectedInsertions, double fpp)

Method Summary

Modifier and Type	Method and Description
int	getTotalRequestsCount(Task task)
protected String	getUrl(Request request)
boolean	isDuplicate(Request request, Task task)
protected com.google.common.hash.BloomFilter<CharSequence>	rebuildBloomFilter()
void	resetDuplicateCheck(Task task)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions)

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions,
                                   double fpp)

Parameters:

expectedInsertions - the number of expected insertions to the constructed

fpp - the desired false positive probability (must be positive and less than 1.0)

Method Detail

rebuildBloomFilter

protected com.google.common.hash.BloomFilter<CharSequence> rebuildBloomFilter()

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

Specified by:

isDuplicate in interface DuplicateRemover

getUrl

protected String getUrl(Request request)

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

Specified by:

resetDuplicateCheck in interface DuplicateRemover

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Specified by:

getTotalRequestsCount in interface DuplicateRemover