if exitstatus or errdata: # consider both failure
numfail += 1 # can get status+stderr
open(outpathbad, 'wb').write(outdata) # save output to view
elif not os.path.exists(outpath) or forcegen:
print('generating:', outpath) # create first output
open(outpath, 'wb').write(outdata)
else:
priorout = open(outpath, 'rb').read() # or compare to prior
if priorout == outdata:
print('passed:', testname)
else:
numfail += 1
print('FAILED output:', testname, outpathbad)
open(outpathbad, 'wb').write(outdata)
print('Finished:', time.asctime())
print('%s tests were run, %s tests failed.' % (len(testfiles), numfail))
We’ve seen the tools used by this script earlier in this part of the book—subprocess,
os.path, glob, files, and the like. This example largely just pulls these tools together to
solve a useful purpose. Its core operation is comparing new outputs to old, in order to
spot changes (“regressions”). Along the way, it also manages command-line arguments,
error messages, status codes, and files.
This script is also larger than most we’ve seen so far, but it’s a realistic and representative
system administration tool (in fact, it’s derived from a similar tool I actually used in the
past to detect changes in a compiler). Probably the best way to understand how it works
is to demonstrate what it does. The next section steps through a testing session to be
read in conjunction with studying the test script’s code.
Running the Test Driver
Much of the magic behind the test driver script in Example 6-9 has to do with its
directory structure. When you run it for the first time in a test directory (or force it to
start from scratch there by passing a second command-line argument), it:
- Collects scripts to be run in the Scripts subdirectory
- Fetches any associated script input and command-line arguments from the
Inputs and Args subdirectories - Generates initial stdout output files for tests that exit normally in the Outputs
subdirectory - Reports tests that fail either by exit status code or by error messages appearing in
stderr
On all failures, the script also saves any stderr error message text, as well as any
stdout data generated up to the point of failure; standard error text is saved to a file in
the Errors subdirectory, and standard output of failed tests is saved with a special
A Regression Test Script | 299