Chapter 4 ■ Data transformation
■Note some examples may not have database alternatives. in these cases, i mention considerations for
storing the data. also, some of the examples use techniques we have discussed. for those examples, the focus
should be on the concepts illustrated.
We begin with a brief discussion and review of what IOT data is and the strategies for making sense of
the data, that is, to gain knowledge from the data.
Making Sense of IOT Data
Before we get into examples of how to implement annotation and aggregation of data, let’s discuss and
review some strategies for making sense of the data. It is important to focus on this topic so that we
understand how to make the design of our data storage successful. I present some of the important concepts
in the following sections in the form of questions you should ask during the database design.
What Is Being Observed?
The first thing to consider is what you are observing. Don’t focus on the data that the sensor is producing (which
is also important), but knowing what you are observing will allow you to plan for how the data will be used.
I like to make a list of the things I want to observe either from the perspective of the problem or when considering
the sensor itself. That is, if I am using a gas detector, I make a list of the gases it can measure and consider how
I would use that observation. While the form of that data—the data type from the sensor—may need to be
transformed, it is more important to keep the focus on what you are observing. Sometimes this isn’t so obvious.
For example, if you are using a sensor to monitor temperature inside an enclosure for a sensitive
device, what does the temperature tell you? Let’s assume you decided to use the sensor to capture when the
temperature reaches a certain threshold because you know that the device cannot withstand temperatures
over that threshold. Thus, it seems natural to store those events by saving the temperature measured and
perhaps the time and date of the event.
However, what if you want to know whether other temperature ranges can affect the device? In this case,
it may be that a certain temperature could be harmful to the device if it were to operate at that temperature
over a period of time. Is it then sufficient to simply store the temperature and signal or alert when the
temperature reaches the original threshold? It may not be. You may find after some time that there is a
correlation with certain temperatures that produces a negative effect on the functioning or accuracy of the
device. If this were true, recording the event of exceeding the threshold is not sufficient because the data
(temperatures over time) has been lost in favor of storing the event.
Thus, you must consider not only what you are observing but also any other way that the observation
could be interpreted and plan what you need to store for those uses. For example, you may find that
recording a certain event is sufficient for some use observations, but you may be able to learn more from the
data collected over time than a single event.
Is There Another Way to Make the Observation?
Closely related to what you expect to observe is how to make the observation. Sometimes it may not be
possible (or too costly) to make a direct observation with certain sensors. For example, suppose you have a
garden pond and want to determine when the filters need cleaning.
In some garden ponds, filters are used to remove debris floating in the water (such as leaves, waste, and
so on). Over time, these filters may become full of debris and thus lose the ability for water to flow through
the filter. Further, let’s assume there is no easy mechanism to measure the flow of water passing through the