Chapter 4 ■ Data transformationWhat Level of Accuracy Do You Need?
It may come as some surprise, but sensors are not always 100 percent accurate. In fact, most sensors that are
marketed for hobbyists and enthusiasts are only about 90 to 95 percent accurate. That is, 5 to 10 times out of
100, the sensor will not produce an accurate value.
If your sensors are only 95 percent accurate, you can only expect your data to be the same. Thus, you
must determine the accuracy required for your solution when choosing or analyzing data from your sensors.
Fortunately, if you need a higher level of accuracy, you can typically find sensors to match your expectations.
However, in my experience, the more accurate the sensor, the more expensive they become.
■Tip Balance accuracy of your sensors with expected accuracy of your data.
What Is the Lifetime of the Data?
Finally, consider how long you want or need to keep the data. Some IOT solutions, especially those that use
the cloud,^2 seem to keep the data for only a few hours, days, or weeks. Not only does this seem arbitrary, but
it also ensures you may never be able to perform any data analysis on historical data. That is, you may lose
knowledge because your sample set is too small.
When considering the lifetime of the data, you should consider all the previous questions and make
a plan for how long you want to keep the data, more specifically, whether data analysis of data over longer
periods of time will be beneficial.
One factor to consider is if larger amounts of data have an effect on the efficiency of your solution.
This could be a case where more data makes searches or even certain code functions slower because it has
to spend more time reading and comparing data. It could also be possible that your storage solution has a
limit to how much data can be stored. Thus, you should consider hardware and software limitations when
determining data lifetime.
Once you determine a lifetime of the data, you should then determine whether you need the data
accessible or not. If you do not need it accessible from your solution or its application, you can consider
removing the old data for safe keeping such as a file archive or backup. If it still must be accessible, you may
want to consider using an alternative or supplemental storage mechanism. For example, if you are storing
the data in a file, you can simply open a new file.
arChIVe, DON’t DeLete
if you determine you can purge data older than a certain date, don’t delete the data. instead, archive
it so that if you ever need to access the older data, you can. for databases, a database backup may
suffice or you can use a table of the same schema to store older data. for file-based storage, make a
copy of the file on removable media or another device.Once you have a plan for the lifetime of the data and what to do with the older data, record this plan in
your notebook so that you can write the code or implement the correct processes to periodically purge the
data.
(^2) Not always the case, but it is a trend I’ve noticed.