Package org.lionsoul.jcseg.segmenter

Class NGramSeg

java.lang.Object

org.lionsoul.jcseg.segmenter.NGramSeg

All Implemented Interfaces:

ISegment

public class NGramSeg
extends Object
implements ISegment

Jcseg n-gram tokenizer implementation

Since:

2.6.0

Author:

lionsoul

Nested Class Summary

Nested classes/interfaces inherited from interface org.lionsoul.jcseg.ISegment

ISegment.Type

Field Summary

Fields

Modifier and Type	Field	Description
SegmenterConfig	config
ADictionary	dic	the dictionary and task configuration
protected int	idx	the index value of the current input stream mainly for track the start position of the token
protected IStringBuffer	isb
protected byte	N	The N for n-gram, default to 1 and that is uni-gram
protected IPushbackReader	reader
protected LinkedList<IWord>	wordPool	CJK word cache pool, Reusable string buffer

Fields inherited from interface org.lionsoul.jcseg.ISegment

CHECK_CE_MASk, CHECK_CF_MASK, CHECK_EC_MASK, COMPLEX, COMPLEX_MODE, DELIMITER, DELIMITER_MODE, DETECT, DETECT_MODE, MOST, MOST_MODE, NGRAM, NGRAM_MODE, NLP, NLP_MODE, SIMPLE, SIMPLE_MODE, START_SS_MASK

Constructor Summary

Constructors

Constructor	Description
NGramSeg(SegmenterConfig config, ADictionary dic)	method to create a new ISegment

Method Summary

Modifier and Type	Method	Description
SegmenterConfig	getConfig()
ADictionary	getDic()
byte	getN()
protected String	getNextType(int c, int type, CharTypeFunction checker)	common interface to get the next n-gram word for the specified char type.
int	getStreamPosition()	get the current length of the stream
IWord	next()	segment a word from a char array from a specified position.
protected void	pushBack(int data)	push back the data to the stream
protected int	readNext()	read the next char from the current position
void	reset(Reader input)	reset the reader
void	setN(byte n)
protected void	streamResetTo(String str, int start)	reset the data back from the specified position
IWord	wordNewOrClone(int t, String str, int type)	check if the specified word is existed in a specified dictionary and if does clone it or create a new one.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

idx

protected int idx

the index value of the current input stream mainly for track the start position of the token

reader

protected IPushbackReader reader

wordPool

protected final LinkedList<IWord> wordPool

CJK word cache pool, Reusable string buffer

isb

protected final IStringBuffer isb

dic

public final ADictionary dic

the dictionary and task configuration

config

public final SegmenterConfig config

N

protected byte N

The N for n-gram, default to 1 and that is uni-gram

Constructor Detail

NGramSeg

public NGramSeg(SegmenterConfig config,
                ADictionary dic)

method to create a new ISegment

Parameters:

config -

dic -

Method Detail

reset

public void reset(Reader input)
           throws IOException

Description copied from interface: ISegment

reset the reader

Specified by:

reset in interface ISegment

Throws:

IOException

getStreamPosition

public int getStreamPosition()

Description copied from interface: ISegment

get the current length of the stream

Specified by:

getStreamPosition in interface ISegment

readNext

protected int readNext()
                throws IOException

read the next char from the current position

Returns:

int

Throws:

IOException

pushBack

protected void pushBack(int data)

push back the data to the stream

Parameters:

data -

streamResetTo

protected void streamResetTo(String str,
                             int start)

reset the data back from the specified position

next

public IWord next()
           throws IOException

Description copied from interface: ISegment

segment a word from a char array from a specified position.

Specified by:

next in interface ISegment

Throws:

IOException

getNextType

protected String getNextType(int c,
                             int type,
                             CharTypeFunction checker)
                      throws IOException

common interface to get the next n-gram word for the specified char type. For the basic Latin char this will automatically do the full-width to half-width uppercase to lowercase conversion.

Parameters:

c -

type -

checker -

Returns:

IWord

Throws:

IOException

wordNewOrClone

public IWord wordNewOrClone(int t,
                            String str,
                            int type)

check if the specified word is existed in a specified dictionary and if does clone it or create a new one.

Parameters:

t -

str -

type -

getDic

public ADictionary getDic()

getConfig

public SegmenterConfig getConfig()

getN

public byte getN()

setN

public void setN(byte n)