Chapter 4 ■ Data transformation

There is one thing that using a data aggregator like this makes a bit more complicated. Recall sometimes
we want to store the date and time when the data was read. If we implement a data aggregator like as shown
previously where we do not capture the date and time when the values were read, setting a timestamp
column in the database may result in inaccuracies in the date and time values.
For example, if it takes 10 seconds to read all three sets of data, the timestamp values will not only be
about 10 seconds delayed, the three rows will have nearly the same timestamp. That may be OK for some
solutions, but if the delay were more like 10 minutes or even an hour, the delay may be unacceptable.
Thus, if you want to store date and time information for data aggregators, you will have to collect that
data when the sensors either are read by the data collectors or are set by the data aggregator when it receives
the data.

Aggregate Calculations

The last form of aggregation you may encounter includes those situations where you need to do some
calculations on a set of data. This could be as simple as calculating an average for a sum, finding minimal
and maximum values, or performing any such formula or operation. You can write code to handle these
operations, and that is a valid solution. However, this is another area where the database server excels.
For example, consider the code excerpt from a Python script shown in Listing 4-13. Here we see code
for reading a number of rows from a file containing data with several columns. We use the power of Python
to decipher the file and then perform operations on the data.

Listing 4-13. Aggregate Calculations (Python)

file_log = open("data_log_python.txt", 'a')
temp_tot = 0;
temp_min = 999;
temp_max = 0;
for i in range(0,20):

read sensors

temp = read_sensor(1)
baro = read_sensor(2)

add to total

temp_tot = temp_tot + temp

find min/max

if (temp < temp_min):
temp_min = temp
if (temp > temp_max):
temp_max = temp
print(temp, baro)
file_log.write(strData.format(temp, baro))

display aggregate values

print "Average Temperature:", temp_tot/20.00
print "Min Temperature:", temp_min
print "Max Temperature:", temp_max
file_log.close()