is the quote character ch. You can have more than one quote character at a
time on a stream, but strings must start and end with the same quote
character. In other words, a string that starts with one quote character ends
when the next instance of that same quote character is found. If a different
quote character is found in between, it is simply part of the string.public voidparseNumbers()Specifies that numbers should be parsed as double-precision floating-point
numbers. When a number is found, the stream returns a type of TT_NUMBER,
leaving the value in nval. There is no way to turn off just this featureto turn
it off you must either invoke ordinaryChars for all the number-related
characters (don't forget the decimal point and minus sign) or invoke
resetSyntax.public voidresetSyntax()Resets the syntax table so that all characters are ordinary. If you do this and
then start reading the stream, nextToken always returns the next character
in the stream, just as when you invoke InputStream.read.There are no methods to ask the character class of a given character or to add new classes of characters. Here
are the default settings for a newly created StreamTokenizer object:
wordChars('a', 'z'); // lower case ASCII letters
wordChars('A', 'Z'); // upper case ASCII letters
wordChars(128 + 32, 255); // "high" non-ASCII values
whitespaceChars(0, ' '); // ASCII control codes
commentChar('/');
quoteChar('"');
quoteChar('\'');
parseNumbers();
This leaves the ordinary characters consisting of most of the punctuation and arithmetic characters (;, :, [, {,
+, =, and so forth).
The changes made to the character classes are cumulative, so, for example, invoking wordChars with two
different ranges of characters defines both ranges as word characters. To replace a range you must first mark
the old range as ordinary and then add the new range. Resetting the syntax table clears all settings, so if you
want to return to the default settings, for example, you must manually make the invocations listed above.
Other methods control the basic behavior of the tokenizer:
public voideolIsSignificant(boolean flag)If flag is true, ends of lines are significant and TT_EOL may be returned
by nextToken. If false, ends of lines are treated as whitespace and
TT_EOL is never returned. The default is false.public voidslashStarComments(boolean flag)If flag is true, the tokenizer recognizes /*...*/ comments. This occurs
independently of settings for any comment characters. The default is false.