Java The Complete Reference, Seventh Edition

IfeolFlagistrue, the end-of-line characters are returned as tokens. IfeolFlagisfalse, the end-
of-line characters are ignored.
ThewordChars( )method is used to specify the range of characters that can be used in
words. Its general form is shown here:

void wordChars(intstart, intend)

Here,startandendspecify the range of valid characters. In this program, characters in the
range 33 to 255 are valid word characters.
The whitespace characters are specified usingwhitespaceChars( ). It has this general form:

void whitespaceChars(intstart, intend)

Here,startandendspecify the range of valid whitespace characters.
The next token is obtained from the input stream by callingnextToken( ). It returns the
type of token.
StreamTokenizerdefines fourintconstants:TT_EOF,TT_EOL,TT_NUMBER, and
TT_WORD. There are three instance variables.nvalis a publicdoubleused to hold the
values of numbers as they are recognized.svalis a publicStringused to hold the value of
any words as they are recognized.ttypeis a publicintindicating the type of token that has
just been read by thenextToken( )method. If the token is a word,ttypeequalsTT_WORD.
If the token is a number,ttypeequalsTT_NUMBER. If the token is a single character,ttype
contains its value. If an end-of-line condition has been encountered,ttypeequalsTT_EOL.
(This assumes thateolIsSignificant( )was invoked with atrueargument.) If the end of the
stream has been encountered,ttypeequalsTT_EOF.
The word count program revised to use aStreamTokenizeris shown here:

// Enhanced word count program that uses a StreamTokenizer
import java.io.*;

class WordCount {
public static int words=0;
public static int lines=0;
public static int chars=0;

public static void wc(Reader r) throws IOException {

StreamTokenizer tok = new StreamTokenizer(r);

tok.resetSyntax();

tok.wordChars(33, 255);

tok.whitespaceChars(0, ' ');

tok.eolIsSignificant(true);

while (tok.nextToken() != tok.TT_EOF) {

switch (tok.ttype) {

case StreamTokenizer.TT_EOL:

lines++;

chars++;

break;

case StreamTokenizer.TT_WORD:

words++;

Chapter 19: Input/Output: Exploring java.io 591