As Storage Switzerland discussed in two articles recently flash based appliances are particularly vulnerable to power failures. This vulnerability is related to the way these appliances use DRAM to buffer writes and to store meta data tables. DRAM is volatile, meaning that if it’s not protected or if protection fails during a power loss data can also be lost or appliance downtime can result. In this article Storage Switzerland will discuss how appliance vendors attempt to protect their systems from power faults and present a new alternative that those vendors should consider.
Why use DRAM?
DRAM is used in flash appliances for typically two purposes, as a buffer area to stage inbound writes and to store mapping tables. Flash based devices typically write data in 32KB chunks made up of 16KB pages, so if a 40KB file is written to the flash device, the flash controller needs to decide what to do with the extra 8KB of data. It can write data to the flash immediately, wasting flash capacity, or for greater efficiency it can hold this data in a small DRAM buffer until more data comes in or a pre-set period of inactivity occurs forcing a DRAM flush. In most cases this DRAM buffer has data in it, so a power failure on an unprotected buffer will almost always result in data loss, which is not acceptable in high-availability environments.
DRAM is also used to store mapping tables. These are similar to inode or file allocation tables (FAT) and are used to point read requests to the data’s actual location on the flash appliance. The time it takes to update the table when new data is written and the time it takes to read the table to get the actual location directly impacts performance. Slow update and read times are especially important on zero latency devices like flash appliances, which is why DRAM is the default choice for most flash appliance vendors.
The problem with a mapping table is that, unlike the buffer described above, it requires a fairly large storage area. A large flash appliance can consume as much as 40GB of DRAM for its mapping table. If the DRAM fails this map needs to be rebuilt before data can be accessed again, a process that’s roughly the equivalent of running FSCK or CHDSK on a file system which can take hours. While no data is lost, the rebuild process can also take hours to complete even on flash. While it’s running the applications that were using the flash appliance are down. Considering that most flash appliances are deployed for mission critical applications mapping table failure in flash appliances is a major problem.
It is obviously critical that the DRAM be protected. In the past batteries have been used to protect DRAM, but they come with their own set of problems. Now super-capacitor technology enables a new type of protection. The problem is that many SSDs don’t have the logic built into them to automatically check and report on capacitor health. The host or storage system needs to be optimized to monitor that capacitor. Until that occurs an SSD that experiences power failure could loose 32KB or more of data, which can lead to data corruption. And, the problem is worse for mapping tables.
These tables can be quite large, too large for a capacitor to keep a single block of memory of this size charged long enough for the mapping table to be copied into flash prior. This leaves the flash appliance vendors with two options: 1) require the use of a UPS (Uninterruptible Power Supply) for every installation or 2) deploy new and expensive non-volatile technologies.
Since most enterprises deploying a flash appliance will likely have a UPS already in place, “requiring” one should not be a big challenge. The problem is that UPS batteries, just like any other battery, can and do fail. Because this failure could lead to hours of downtime, the appliance vendors have to do more. A few are leaning toward NVRAM. But that could add considerable cost to the appliance design and, more importantly, increase the complexity of that design since NVRAM typically has to be integrated into a system differently than a DRAM DIMM.
The Non-Volatile DIMM (NVDIMM) Alternative
Alternatives now exist, such as Non-Volatile DIMMs (NVDIMM) made by companies like Viking Technology. These are uniquely designed module subsystems that populate traditional DRAM sockets. In addition to DRAM capacity, each NVDIMM has data protection capability that should make them very interesting for flash appliance vendors.
They have integrated flash that is used to store a copy of the DRAM contents in case a power failure occurs and a supercapacitor to power the NVDIMM until the copy process is complete. While the cost of an NVDIMM is obviously higher than traditional DRAM, they are an order of magnitude less expensive than the alternate non-volatile memories (NVRAM chips) currently on their way to the market.
The NVDIMM architecture brings several advantages to vendors designing and building flash appliances. In NVDIMM enabled platforms, similar to what Viking has done with SuperMicro, are plug-compatible with existing DRAM sockets, meaning vendors incur no additional costs of design or redesign to accommodate the technology.
The NVDIMMs also divide up the protection workload to reduce vulnerability by decreasing the time required to copy data into flash. As described above it can take a few minutes to write a single 80GB block of data from one storage location to another, even if that transfer is from DRAM to flash. With NVDIMM each 64GB may be spread across 4 to 8 DIMMs. Each NVDIMM writes its memory contents separately, but simultaneously, to a storage device integrated on the module, minimizing time required for the copy process from seconds to perhaps a minute or two.
The final advantage is that the NVDIMM can be used for more than just DRAM data protection. With a cost effective and reliable DRAM solution, the use of DRAM in flash systems could be expanded. For example, write buffers could be enlarged and used as more than just a buffer for write traffic. They could coalesce writes so that less data is actually written to the flash appliance, a process that would improve flash life expectancy and performance as well. NVDIMM could also be used to store deduplication meta data in addition to mapping tables to allow for effective implementation of deduplication without impacting performance. This could further increase flash endurance and improving flash price per GB.
One of the reasons flash appliance vendors have been slow to address the issues related to power failures is the difficulties of integrating certain non-volatile technologies into their physical hardware. Since many vendors leverage off-the-shelf servers as their SSD appliances the engineering work and costs associated with any customization has not been worth the effort. As a result the default solution has been to require a UPS.
But UPSs are not perfect and the impact of a power failure can be substantial, so an alternative is needed. Companies like Viking Technology are offering NVDIMM solutions that allow system builders to create flash appliances that can maintain data in the event of a power failure using off the shelf components by simply installing these persistent DRAM modules in the existing DIMM sockets.
Viking is a client of Storage Switzerland