Chapter 4 ■ Data transformation
■Tip make a plan for how long you want to store the data. implement any purge events as backup or
archival rather than deletion.
Now that we’ve seen some things you need to consider about making sense of the data, let’s see several
techniques for implementing these strategies starting with annotation.
Some of the code examples in the following sections are written for file-based storage. Other media
forms are similar, but the actual write methods may differ. I also simulate the collection of data from sensors.
You would typically use a method written to retrieve the data in a similar manner.
I also include considerations for using the database to add the annotations or implement the
aggregation. I use these sections to introduce the database concepts to prepare you for the more in-depth
tutorial on MySQL in the rest of this book.
Let’s begin with a look at some of the more common annotations you may want to make.
Annotation
Annotating data for your IOT solution is simply any additional data you add to, combine with, or calculate
from the data. For example, you may want to store a string to describe the event, sensor, node, and so on. Or
you may want to perform some transformation on the data and save it along with the original data.
■Tip it is always a good practice to save the original values.
I demonstrate a number of these transformations in this chapter, providing a code example that you
can use on your data aggregators or data nodes such as those hosted by an Arduino, Raspberry Pi, or similar
low-cost computer node. Thus, where appropriate, I present both an Arduino sketch excerpt and an example
using Python. While you may not be familiar with both languages, you can use these examples to help guide
you to write in your chosen language.
CaN YOU haVe tOO MUCh aNNOtatION?
annotation should be for the benefit of the user or to make data more informative. thus, you must
consider whether the annotation will add anything, but more importantly, the annotation should not
complicate or otherwise obscure the data. some of the examples in this chapter get close to that line of
benefit versus detriment, but this is for illustrative purposes. Use your own judgment as to how much
annotation is enough for your needs. too much and you risk obscuring knowledge and too little can
make the data harder to use.Now that you know what annotation is, let’s see a few examples beginning with the easiest forms of
transformation—adding text and recording notes.
■Note the following code examples are excerpts for brevity. Complete sketches or scripts are available from
the book source code download site.