It’s a fact that data stored on magnetic media degrades over time. It cannot be stopped; it can only be managed. It’s a problem that you must address if you plan to create a data monetization system that stores data for many years. The good news is object storage systems deliver this functionality, natively.
Each bit stored on magnetic media has the possibility of flipping over time. That is, a 1 becomes a 0 and vice versa. Coercivity is the term that describes the degree to which this is a possibility, as in how easy is it to coerce the bit to flip. The formula that determines the coercivity of a medium is KuV/kT, where V is the size (or Volume) of the bit, k is its average temperature in kelvin, and T is the length of time the magnetic media will store the data. Since a bigger coercivity value means data will last longer, the bigger the magnetic bit the better, and the colder the media the better. And obviously, the shorter amount of time we store it, the less it will degrade.
Unfortunately for data monetization projects, the data that needs storage for a very long time is stored on magnetic disk, which uses very small magnetic bits and that operate a very high temperature. Specifically, it is commonly accepted that storing data on a disk drive for longer than five years has a very high possibility of bit rot, which is the colloquial term for magnetic degradation. Bit rot is an especially sensitive problem in a data monetization project, where corrupt data can render the project useless.
Another risk affecting the long-term storage of data is rogue employees or outside people attempting to corrupt the data. This is a less likely scenario than bit rot, but it is still a possibility that you must mitigate as well.
Typical NAS systems have no built-in features to detect or recover from bit rot, so using those systems for storing data for several years is risky. Object storage systems, on the other hand, have a few features that make them well suited for long-term retention of data and for mitigating the issues mentioned above. Each object is typically stored in multiple locations. And those objects have a cryptographic hash the object store creates based on the contents of the object. If any part of the object changes due to bit rot or malfeasance, the hash changes, giving the object the ability to detect the corruption.
The only good thing about bit rot is that if it corrupted a bit in one copy of an object, the other copy of the object would not be corrupted. All the object storage system has to do is detect the corruption by occasionally rechecking hash values against the content of each object. If it detects corruption in an object, it can simply replace the corrupted copy with a replicated copy of the same object.
Data that is going to be stored for long periods of time must be protected against bit rot and malfeasance. NAS systems have no built-in features for doing this, so storing data on them for long periods of time is risky. Object storage systems, however, can use the cryptographic signatures of an object to detect and automatically repair any corruption to the object. This makes object storage the perfect place to store data for long periods of time.
Sponsored by Caringo