Class: Ferret::Analysis::AsciiWhiteSpaceTokenizer

Summary

A WhiteSpaceTokenizer is a tokenizer that divides text at white-space. Adjacent sequences of non-WhiteSpace characters form tokens.

Example

  "Dave's résumé, at http://www.davebalmain.com/ 1234"
    => ["Dave's", "résumé,", "at", "http://www.davebalmain.com", "1234"]

Public Class Methods

AsciiWhiteSpaceTokenizer.new() → tokenizer

Create a new AsciiWhiteSpaceTokenizer 

/*
 *  call-seq:
 *     AsciiWhiteSpaceTokenizer.new() -> tokenizer
 *
 *  Create a new AsciiWhiteSpaceTokenizer 
 */
static VALUE
frb_a_whitespace_tokenizer_init(VALUE self, VALUE rstr) 
{
    return get_wrapped_ts(self, rstr, whitespace_tokenizer_new());
}