Class TextFormat.Tokenizer

java.lang.Object

com.google.protobuf.TextFormat.Tokenizer

Enclosing class:

TextFormat

private static final class TextFormat.Tokenizer
extends Object

Represents a stream of tokens parsed from a String.

The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:

• java.io.StreamTokenizer: This almost does what we want -- or, at least, something that would get us close to what we want -- except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').
• java.util.Scanner: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly, Scanner requires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"), Scanner would recognize it only as a single token. Furthermore, Scanner provides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.

Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need Matcher.usePattern(), which is new in Java 1.5.) So, we can use that, at least. Unfortunately, this implies that we need to have the entire input in one contiguous string.

Field Summary

Fields

Modifier and Type	Field	Description
private int	column
private String	currentToken
private static final Pattern	DOUBLE_INFINITY
private static final Pattern	FLOAT_INFINITY
private static final Pattern	FLOAT_NAN
private int	line
private final Matcher	matcher
private int	pos
private int	previousColumn
private int	previousLine
private final CharSequence	text
private static final Pattern	TOKEN
private static final Pattern	WHITESPACE

Constructor Summary

Constructors

Modifier	Constructor	Description
private	Tokenizer(CharSequence text)	Construct a tokenizer that parses tokens from the given text.

Method Summary

Modifier and Type	Method	Description
boolean	atEnd()	Are we at the end of the input?
void	consume(String token)	If the next token exactly matches token, consume it.
boolean	consumeBoolean()	If the next token is a boolean, consume it and return its value.
ByteString	consumeByteString()	If the next token is a string, consume it, unescape it as a ByteString, and return it.
private void	consumeByteString(List<ByteString> list)	Like consumeByteString() but adds each token of the string to the given list.
double	consumeDouble()	If the next token is a double, consume it and return its value.
float	consumeFloat()	If the next token is a float, consume it and return its value.
String	consumeIdentifier()	If the next token is an identifier, consume it and return its value.
int	consumeInt32()	If the next token is a 32-bit signed integer, consume it and return its value.
long	consumeInt64()	If the next token is a 64-bit signed integer, consume it and return its value.
String	consumeString()	If the next token is a string, consume it and return its (unescaped) value.
int	consumeUInt32()	If the next token is a 32-bit unsigned integer, consume it and return its value.
long	consumeUInt64()	If the next token is a 64-bit unsigned integer, consume it and return its value.
private TextFormat.ParseException	floatParseException(NumberFormatException e)	Constructs an appropriate TextFormat.ParseException for the given NumberFormatException when trying to parse a float or double.
(package private) int	getColumn()
(package private) int	getLine()
(package private) int	getPreviousColumn()
(package private) int	getPreviousLine()
private TextFormat.ParseException	integerParseException(NumberFormatException e)	Constructs an appropriate TextFormat.ParseException for the given NumberFormatException when trying to parse an integer.
boolean	lookingAt(String text)	Returns true if the current token's text is equal to that specified.
boolean	lookingAtInteger()	Returns true if the next token is an integer, but does not consume it.
void	nextToken()	Advance to the next token.
TextFormat.ParseException	parseException(String description)	Returns a TextFormat.ParseException with the current line and column numbers in the description, suitable for throwing.
TextFormat.ParseException	parseExceptionPreviousToken(String description)	Returns a TextFormat.ParseException with the line and column numbers of the previous token in the description, suitable for throwing.
private void	skipWhitespace()	Skip over any whitespace so that the matcher region starts at the next token.
boolean	tryConsume(String token)	If the next token exactly matches token, consume it and return true.
boolean	tryConsumeDouble()	If the next token is a double, consume it and return true.
boolean	tryConsumeFloat()	If the next token is a float, consume it and return true.
boolean	tryConsumeIdentifier()	If the next token is an identifier, consume it and return true.
boolean	tryConsumeInt64()	If the next token is a 64-bit signed integer, consume it and return true.
boolean	tryConsumeString()	If the next token is a string, consume it and return true.
boolean	tryConsumeUInt64()	If the next token is a 64-bit unsigned integer, consume it and return true.
TextFormat.UnknownFieldParseException	unknownFieldParseExceptionPreviousToken(String unknownField, String description)	Returns a TextFormat.UnknownFieldParseException with the line and column numbers of the previous token in the description, and the unknown field name, suitable for throwing.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

text

private final CharSequence text

matcher

private final Matcher matcher

currentToken

private String currentToken

pos

private int pos

line

private int line

column

private int column

previousLine

private int previousLine

previousColumn

private int previousColumn

WHITESPACE

private static final Pattern WHITESPACE

TOKEN

private static final Pattern TOKEN

DOUBLE_INFINITY

private static final Pattern DOUBLE_INFINITY

FLOAT_INFINITY

private static final Pattern FLOAT_INFINITY

FLOAT_NAN

private static final Pattern FLOAT_NAN

Constructor Details

Tokenizer

private Tokenizer(CharSequence text)

Construct a tokenizer that parses tokens from the given text.

Method Details

getPreviousLine

int getPreviousLine()

getPreviousColumn

int getPreviousColumn()

getLine

int getLine()

getColumn

int getColumn()

atEnd

public boolean atEnd()

Are we at the end of the input?

nextToken

public void nextToken()

Advance to the next token.

skipWhitespace

private void skipWhitespace()

Skip over any whitespace so that the matcher region starts at the next token.

tryConsume

public boolean tryConsume(String token)

If the next token exactly matches token, consume it and return true. Otherwise, return false without doing anything.

consume

public void consume(String token)
             throws TextFormat.ParseException

If the next token exactly matches token, consume it. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

lookingAtInteger

public boolean lookingAtInteger()

Returns true if the next token is an integer, but does not consume it.

lookingAt

public boolean lookingAt(String text)

Returns true if the current token's text is equal to that specified.

consumeIdentifier

public String consumeIdentifier()
                         throws TextFormat.ParseException

If the next token is an identifier, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

tryConsumeIdentifier

public boolean tryConsumeIdentifier()

If the next token is an identifier, consume it and return true. Otherwise, return false without doing anything.

consumeInt32

public int consumeInt32()
                 throws TextFormat.ParseException

If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

consumeUInt32

public int consumeUInt32()
                  throws TextFormat.ParseException

If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

consumeInt64

public long consumeInt64()
                  throws TextFormat.ParseException

If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

tryConsumeInt64

public boolean tryConsumeInt64()

If the next token is a 64-bit signed integer, consume it and return true. Otherwise, return false without doing anything.

consumeUInt64

public long consumeUInt64()
                   throws TextFormat.ParseException

If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

tryConsumeUInt64

public boolean tryConsumeUInt64()

If the next token is a 64-bit unsigned integer, consume it and return true. Otherwise, return false without doing anything.

consumeDouble

public double consumeDouble()
                     throws TextFormat.ParseException

If the next token is a double, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

tryConsumeDouble

public boolean tryConsumeDouble()

If the next token is a double, consume it and return true. Otherwise, return false without doing anything.

consumeFloat

public float consumeFloat()
                   throws TextFormat.ParseException

If the next token is a float, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

tryConsumeFloat

public boolean tryConsumeFloat()

If the next token is a float, consume it and return true. Otherwise, return false without doing anything.

consumeBoolean

public boolean consumeBoolean()
                       throws TextFormat.ParseException

If the next token is a boolean, consume it and return its value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

consumeString

public String consumeString()
                     throws TextFormat.ParseException

If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

tryConsumeString

public boolean tryConsumeString()

If the next token is a string, consume it and return true. Otherwise, return false.

consumeByteString

public ByteString consumeByteString()
                             throws TextFormat.ParseException

If the next token is a string, consume it, unescape it as a ByteString, and return it. Otherwise, throw a TextFormat.ParseException.

Throws:

TextFormat.ParseException

consumeByteString

private void consumeByteString(List<ByteString> list)
                        throws TextFormat.ParseException

Like consumeByteString() but adds each token of the string to the given list. String literals (whether bytes or text) may come in multiple adjacent tokens which are automatically concatenated, like in C or Python.

Throws:

TextFormat.ParseException

parseException

public TextFormat.ParseException parseException(String description)

Returns a TextFormat.ParseException with the current line and column numbers in the description, suitable for throwing.

parseExceptionPreviousToken

public TextFormat.ParseException parseExceptionPreviousToken(String description)

Returns a TextFormat.ParseException with the line and column numbers of the previous token in the description, suitable for throwing.

integerParseException

private TextFormat.ParseException integerParseException(NumberFormatException e)

Constructs an appropriate TextFormat.ParseException for the given NumberFormatException when trying to parse an integer.

floatParseException

private TextFormat.ParseException floatParseException(NumberFormatException e)

Constructs an appropriate TextFormat.ParseException for the given NumberFormatException when trying to parse a float or double.

unknownFieldParseExceptionPreviousToken

public TextFormat.UnknownFieldParseException unknownFieldParseExceptionPreviousToken(String unknownField,
 String description)

Returns a TextFormat.UnknownFieldParseException with the line and column numbers of the previous token in the description, and the unknown field name, suitable for throwing.