| Interface | Description |
|---|---|
| CoreTokenFactory<IN extends CoreMap> |
To make tokens like CoreMap or CoreLabel.
|
| LexedTokenFactory<T> |
Constructs a token (of arbitrary type) from a String and its position
in the underlying text.
|
| SerializableFunction<T1,T2> |
This interface is a conjunction of Function and Serializable, which is
a bad idea from the perspective of the type system, but one that seems
more palatable than other bad ideas until java's type system is flexible
enough to support type conjunctions.
|
| Tokenizer<T> |
Tokenizers break up text into individual Objects.
|
| TokenizerFactory<T> |
A TokenizerFactory is a factory that can build a Tokenizer (an extension of Iterator)
from a java.io.Reader.
|
| WordSegmenter |
An interface for segmenting strings into words
(in unwordsegmented languages).
|
| Class | Description |
|---|---|
| AbstractTokenizer<T> |
An abstract tokenizer.
|
| Americanize |
Takes a HasWord or String and returns an Americanized version of it.
|
| CoreLabelTokenFactory |
Constructs
CoreLabels from Strings optionally with
beginning and ending (character after the end) offset positions in
an original text. |
| LexerTokenizer | |
| LexerUtils |
This class contains various static utility methods invoked by our JFlex NL lexers.
|
| Morphology |
Morphology computes the base form of English words, by removing just
inflections (not derivational morphology).
|
| PTBTokenizer<T extends HasWord> |
A fast, rule-based tokenizer implementation, which produces Penn Treebank
style tokenization of English text.
|
| PTBTokenizer.PTBTokenizerFactory<T extends HasWord> |
This class provides a factory which will vend instances of PTBTokenizer
which wrap a provided Reader.
|
| TokenizerAdapter |
This class adapts between a
java.io.StreamTokenizer
and a edu.stanford.nlp.process.Tokenizer. |
| WhitespaceTokenizer<T extends HasWord> |
A WhitespaceTokenizer is a tokenizer that splits on and discards only
whitespace characters.
|
| WhitespaceTokenizer.WhitespaceTokenizerFactory<T extends HasWord> |
A factory which vends WhitespaceTokenizers.
|
| WordSegmentingTokenizer |
A tokenizer that works by calling a WordSegmenter.
|
| WordTokenFactory |
Constructs a Word from a String.
|
| Enum | Description |
|---|---|
| LexerUtils.DashesEnum | |
| LexerUtils.EllipsesEnum | |
| LexerUtils.QuotesEnum |
Document process(Document) method
which takes a document and returns another document, which may
be parsed, stoplisted, stemmed, etc.