Chapter 4
Data Transformation
Whether you have an existing IOT solution, you’re developing an idea for a new IOT solution, or you are
refining an existing IOT solution, data will be a major focus at some point. Having an excellent IOT solution
with near-perfect hardware, communication, and even display capabilities is no insurance against poorly
stored or poorly handled data. Getting the data right can be the most difficult task of all. Indeed, I have
seen IOT solutions that were good but failed miserably because the data was either not easily presented,
understood, or incomplete.
The struggle comes when you consider the type of data being generated and how that data is
interpreted. Since most IOT solutions are designed to observe the world around us, it is natural that most
of the data generated is sensor data in some form or another. Of course, there could be other data such as
images or video from cameras, user input, or even data from social media applications. Fortunately, this data
is easily understood. Storage may still be a challenge but typically easily solved.
However, sensor data can be much harder to get right. This is because, as you saw in Chapter 2 , sensors
can generate analog (for example, voltage) or digital values, which represent the observation. For example,
you may find a sensor that you want to use produces values between -5.0 and +5.0. But the sensor may be
measuring something like flow rate through a pipe (hose), which you may want to see as liters or gallons per
minute. The trick then, is converting the value from the sensor into the expected rate.
There are two classes of transformations that you may need to do with the data. You may need to
add additional information such as the units of measure, notes about the events, date or time, and so on.
This additional data is a form of annotation. You may also need to perform larger-scale operations on the
data where you are combining data from multiple sources (sensors) or even from multiple nodes. These
operations are aggregations of the data and can require more effort to implement.
You will discover a number of techniques for handling these and similar transformations in this
chapter. The examples are shown from two perspectives: excerpts of code from sketches for the Arduino or
Python scripts for use on low-cost computer nodes and suggestions for how to implement the technique in
a database. Presenting both techniques for use outside and inside^1 the database server provides you with
more breadth of knowledge and thus more tools to develop your IOT solutions.
The database examples will present suggestions rather than concrete examples. The point is to show
how it is done in code followed by suggestions for a database implementation so that when you read the next
two chapters, you will see how to realize some of these suggestions. I will present many of these suggestions
in more detail in Chapter 8. The takeaway then is you will see how much power is available in using a
database server.
(^1) Before database storage and during database storage.