IT considers data durable if it survives anything that might happen to it because of how the user stores it. Typical data protection systems (e.g. backup, snapshots, replication) protect data from component, media, site, and human failure – or at least they’re supposed to. Object storage, whether from a cloud provider or onsite in the data center, can provide many of those features as well. My colleague Joseph Ortiz will compare the data protection techniques of NAS vs. Object storage in a future post.
What about Bit Rot?
What typical data protection systems do not protect against is bit rot. Bit rot is a layman term applied to the concept of magnetic degradation over time. A secret (that hasn’t been kept very well) about magnetic storage is that it all degrades over time. Ones become zeros and vice versa. With any magnetic storage, it is not a matter of if the data will be corrupted – it is when.
The degree to which a medium will allow bits to be coerced over time into changing polarity is called its coercivity value, and the formula to determine coercivity is KuV/kT, where V is the volume of the magnetic bit and K is the ambient temperature of that bit in Kelvin. Unfortunately for those wishing to store data for long periods of time, disk has poor values for both. (It has very small magnetic bits and disks are hot all the time.) When those values are plugged into the coercivity formula, the values that emerge suggest that data left on disk will begin to significantly degrade after five years. This means that if you are using disk as your long term storage medium, you must monitor for and repair magnetic degradation. This must be done at a level higher than the device level, because the underlying hardware layer will not detect this magnetic degradation.
Object storage software is perfect for monitoring and repairing the effects of magnetic degradation, because such products use a unique identifier for each object that is the result of some type of algorithm (e.g. SHA-256) being run against the contents of the object. By re-running the algorithm against an object and comparing it against its unique identifier, the software makes sure that no bit rot corrupts the file. Caringo’s Swarm does this using proactive period health checks, which allows them to identify and correct any bit rot even before an object is requested. Other vendors check for and repair corruption when they read the file.
Another way the software ensures durability is to make sure the object can survive simultaneous node or site failures. This is done by replicating the object to multiple locations. An object that does all of these things would be considered durable even over long periods of time.
A storage product that is truly durable will be able to overcome any kind of damage that could happen to a file, which goes beyond a component or site failure. It includes protection against bit rot over time – which requires some higher level of intelligence which is provided by object storage vendors. It can also provide protection against accidental or malicious damage to the data as well. When evaluating storage vendors, make sure to look into both sides of the equation.
Sponsored by Caringo
Caringo was founded in 2005 to change the economics of storage by designing software from the ground up to solve the issues associated with data protection, management, organization and search at massive scale. Caringo’s flagship product, Swarm, eliminates the need to migrate data into disparate solutions for long-term preservation, delivery and analysis—radically reducing total cost of ownership. Today, Caringo software is the foundation for simple, bulletproof, limitless storage solutions for the Department of Defense, the Brazilian Federal Court System, City of Austin, Telefónica, British Telecom, Ask.com, Johns Hopkins University and hundreds more worldwide. Visit www.caringo.com to learn more.