tokenizer.h File Reference

#include <zorba/config.h>
#include <zorba/locale.h>
#include <zorba/internal/unique_ptr.h>
#include <zorba/internal/ztd.h>

Include dependency graph for tokenizer.h:

Go to the source code of this file.

Classes
class	zorba::Tokenizer::Callback
	A Callback is called once per token. More...
struct	zorba::Tokenizer::Numbers
	A Numbers contains the current token, sentence, and paragraph numbers. More...
class	zorba::Tokenizer
	A Tokenizer breaks a string into a stream of word tokens. More...
class	zorba::TokenizerProvider
	A TokenizerProvider provides a Tokenizer for a given language. More...
Namespaces
namespace	zorba
	Copyright 2006-2011 The FLWOR Foundation.