naive about the string itself. This modular structure keeps the code relatively simple.
And it’s another example of the object-oriented programming (OOP) composition re-
lationship at work: parsers embed and delegate to scanners.
The module in Example 19-13 implements the lexical analysis task—detecting the ex-
pression’s basic tokens by scanning the text string left to right on demand. Notice that
this is all straightforward logic; such analysis can sometimes be performed with regular
expressions instead (described earlier), but the pattern needed to detect and extract
tokens in this example would be too complex and fragile for my tastes. If your tastes
vary, try recoding this module with re.
Example 19-13. PP4E\Lang\Parser\scanner.py
"""
###############################################################################
the scanner (lexical analyser)
###############################################################################
"""
import string
class SyntaxError(Exception): pass # local errors
class LexicalError(Exception): pass # used to be strings
class Scanner:
def init(self, text):
self.next = 0
self.text = text + '\0'
def newtext(self, text):
Scanner.init(self, text)
def showerror(self):
print('=> ', self.text)
print('=> ', (' ' * self.start) + '^')
def match(self, token):
if self.token != token:
raise SyntaxError(token)
else:
value = self.value
if self.token != '\0':
self.scan() # next token/value
return value # return prior value
def scan(self):
self.value = None
ix = self.next
while self.text[ix] in string.whitespace:
ix += 1
self.start = ix
if self.text[ix] in ['(', ')', '-', '+', '/', '*', '\0']:
self.token = self.text[ix]
ix += 1
1442 | Chapter 19: Text and Language