Flash storage continues to be the “go to” option for IT professionals looking to solve performance problems, but these infrastructure designers are struggling with how to best implement flash. Automated tiering and caching are becoming common answers to that question. Instead of IT administrators having to measure the flash worthiness of each data set, the storage system or storage software can now make data placement decisions automatically. While the two terms are often used interchangeably the technologies are not the same and administrators need to be aware of those differences.
What is Caching?
As we detail in our article “What is Storage Caching”, caching comes in three basic forms. Write-around caching, also known as “read-only” caching, only operates on data that has been written to the slower tier of storage, presumably disk. Data is only copied into cache after a certain threshold of read activity occurs to those files.
The second type, write-through caching, writes data to both the flash area and the hard disk area at the same time. This technique essentially pre-promotes data working under the assumption that the most recently written data will also most likely be the next read. In both write-around and write-through cache methods write operations happen at hard disk speeds since the application has to wait for hard drive acknowledgement of a successful write.
What is Tiering?
Instead of making a copy of the most active data, tiering typically writes data in its entirety, first to one storage area (or tier), and then moves that data to different areas based on application or user performance requirements. Most automated tiering technology starts by making sure all new or modified data is written to the hard disk area first. The technology then migrates that data up to the flash tier based on access patterns.
Moves vs. Copies
A key difference between tiering and caching is that tiering moves data to the flash storage area instead of copying it, both from slower storage to faster storage and vice versa. Caching is essentially a one-way transaction. When the cache is done with the data it was accelerating it simply nullifies it instead of copying it back to the hard disk area. Write-back caching could be considered an exception but typically the write-back technology has made its copy to the hard disk area long before the cache is done with the actual data.
In theory, this should mean that caching has less performance impact than automated tiering. That would be correct if the vendor has chosen a hard-drive-first auto-tiering design strategy, since almost certainly some data will need to be copied “up” from the hard drive tier. If the vendor chose a flash-first strategy then the majority of the time data will be pushed “down” the storage tiers and extra copies will be unnecessary.
The important difference between moves and copies is that a cache does not need to have the redundancy that tiering does. Tiering stores the only copy of data for potentially a considerable period of time so it needs to have full data redundancy like RAID or mirroring. Because of the premium cost of flash storage that redundancy may be more than the IT budget can support.
Dealing with Write Performance
To address write performance concerns we have seen a few storage systems with caching technologies introduced that write to the flash area first, instead of the primary storage area. In caching terminology this is called “write-back caching” and it’s different from the other forms of caching in that writes are acknowledged to the user or application as soon as they are written to the flash area. While the application does not have to wait for hard-drive-speed acknowledgement there is the potential for data loss and corruption if the flash fails prior to flushing all its data to the hard disk area.
In the same way we have seen automated tiering technology only a few storage systems write to the flash tier first. As with write-back caching, the application benefits from flash performance, but like write-back caching, flash-first tiering requires extra redundancy to ensure that data can survive a flash module or storage system failure. Although it makes performance acceleration more expensive, redundancy is something that can be designed into both caching and tiering solutions.
There is another concern with flash-first tiering and caching technologies that’s been well documented, flash has a finite number of times it can be written to. SLC (Single-Level Cell) is the most durable form of flash but also the most expensive and, as a result, most vendors lead with MLC (Multi-Level Cell) flash to keep prices down. But this brings an increased risk of flash failure, a concern with both tiering and caching technologies if they write to flash first or in parallel to the hard disk.
In addition, the write-through, write-back and flash-first tiering use cases are all susceptible to early flash burn out since all net new writes go to flash first or in parallel to hard disk.
These issues of data risk and flash endurance can be overcome by leveraging a small SLC storage area to act as a ‘shock absorber’ for writes. Dell, as we pointed out in our lab report “Mixed All-Flash Array Delivers Safer High Performance”, was the first vendor to take the extra step of writing to a small SLC storage area first, then leveraging automated tiering to move data to an MLC tier as it ages. This technique strikes a better balance of durability, performance and risk aversion. We expect many hybrid storage arrays to implement a similar approach in 2014.
Multiple Tiers
The single biggest difference in tiering vs. caching technology is that tiering can involve more than two types of storage areas. Caching, at least so far, has essentially been two tiered, flash and hard disk. While there are some caching technologies that will leverage RAM as a third tier those are rare. Automated tiering can be much more robust; we’ve seen vendors support as many as four tiers of storage. This allows for tiering to work for both enhancing performance enhancer and saving cost. Automated tiering could for example migrate from SLC Flash to MLC Flash, then to 15K RPM hard disk and finally to 7.5K RPM high capacity disks. As the data becomes less active the media that the data is stored on becomes less expensive.
Storage System Specific
Automated tiering is typically storage system-specific, meaning that the tiering software is designed to work with a particular storage system and all the storage hardware needs to be purchased from a single vendor. Even so, it should be considered a “must-have” feature for any organization that is ready for a storage system refresh or upgrade. An exception to this rule is the emerging software defined storage (SDS) market. A few of these solutions will allow for tiering to occur across different vendors’ hardware but it still must be controlled by a single storage management solution.
Caching on the other hand tends to be more universal and can be applied to almost any solution. There are caching appliances that run in-band on the storage network as well as on the physical servers themselves. In fact our research shows that the most common form of flash implementation is when flash is installed in a specific server having a performance problem. The most obvious compliment to that server flash hardware is server-side caching software.
Conclusion
Tiering and caching are two heavily used terms in the flash market. Both attempt to accomplish a similar goal, the automated use of a flash investment, but take different approaches. Caching is temporary in nature and typically can better utilize more scarce flash resources. Tiering is more permanent but requires a higher capacity investment in flash to be effectively utilized. In my next column I will discuss how to decide which of the two technologies is best for your data center.
Click Here To Sign Up For Our Newsletter
George Crump is the Chief Marketing Officer at VergeIO, the leader in Ultraconverged Infrastructure. Prior to VergeIO he was Chief Product Strategist at StorONE. Before assuming roles with innovative technology vendors, George spent almost 14 years as the founder and lead analyst at Storage Switzerland. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Before founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators, where he was in charge of technology testing, integration, and product selection.
StorageSwiss.com - The Home of Storage Switzerland 
Nice post George. Here are some thoughts:
Well, In tiering, total storage capacity is a sum of all individual tier capacities whereas in caching, the cache size does not ‘add’ to the overall slower storage capacity. This is one of the key differences I think. Also, caching can be multi-level similar to tiering (RAM-SSD-HDD), multi-levels are not solely a ‘tiering’ thing.
Caching is better suited to workloads that are dynamically changing say on a daily/weekly basis. Tiering works better for workloads where frequency of change or need to move data across tiers involves longer time periods – say weekends / months / quarter closing.
Cache warmup timeframe is usually minutes/hours whereas tiering ‘warmup’ is usually in days.
I’d like to know your thoughts on how in-line dedup fares with Caching/Tiering. Does it complement one or both?
Either technology has a flaw that most of my customers hate, very unpredictable. Performance data on flash the rest on disk. May not me as flexible or automated but it will maximize your dollar and the reason for flash is for predictable performance not capacity. Cache algorithms Are good but not perfect. Tieing rarely is in time.
Hi David, It is always good to know real customer pain points, thanks for sharing!
I’d really like to understand some specifics here. What kind of workloads did you / your customers tried with say SSD caching or tiering technologies where you found that the performance was unpredictable or the algorithms were not perfect? Also, if appropriate, pls share the products that you tried or at least their form factor. Say – were these hypervisor based or controller based caching / tiering products or SW only, SW+HW combo products.
Two main scenarios that I have designed solutions for are SQL and VDI. In SQL the customers database was large and required many write IOPS. We designed a solution using his existing spinning disk for the bulk of the data but determined that a few table spaces required performance. We placed those tables on flash (TEMPDB etc). With VDI we placed the GOLD boot images on Flash and the rest of the data on spinning. We were able to solve the issues with less that 3TB of flash for the entire shared environment.
The most common solutions that I replace are Equallogic and Nimble.
I work for Astute Networks. We have a shared all flash appliance that has a great balance of performance and cost. We have IP that takes fabric latency out of the picture.
Thanks, this is useful. I work for CacheBox and we offer a lean and smart SSD caching software and so I was trying to understand what kind of workloads exhibit unpredictable performance with caching or tiering and may not be well suited for either technologies.
Here is what I understood: In your case above, flash was used to store pre-identified datasets (static). If the app did not require any runtime or dynamic determination of whether a dataset needs to be on flash or spinning disk OR if there was large capacity flash available then this is a good choice. Also I presume the VDI was non-persistent desktop with predominantly read workload.
What happens if and when:
1. Performance demanding files or DBs (such as tempdb or others, index files etc.) are larger in size than available flash? Pls ignore if this is never going to be the case.
2. There are other applications running on the same server that need performance boost . Is there a way to intelligently / dynamically share flash capacity and performance across apps and across time periods of low/high IO access? Again, if sharing of flash for optimal use is not a consideration then of course this doesn’t hold good.
3. Write-heavy workload, rewrites – does flash have any reliability issues? How much is the write performance difference as opposed to HDD?
4. Performance demanding Datasets are huge – beyond 2-3 TB? There may be one time Data migration costs to all-flash, but if dataset are huge – flash costs can be huge too.
Thanks again, appreciate your thoughts.
Well one good characteristic of a good tech advisor is to know when a solution is not a good fit for the customer. With that said our solutions as many other all flash solutions can scale to 45TB in a single 2U appliance and scale to multiple appliances. We assign flash at a block or NFS level to a server. The server application or administrator can use it as needed. Flash has a main weakness. writes to a cell. we are seeing media that can survive over 30,000 writes per cell with advanced wear leveling. industry tests have proved 5-7 years of write intensive database utilization.
I understand your solution and am totally open to that your solution may be a better fit for a solution that we are not. What my customers like is the ability to distribute flash space to the datacenter in any increment and modify the allocation as business needs change. We also stay out of the compute layer. we don’t require drivers that could place compatibility, interoperability, and risks to the entire solution.
David,
Thanks for reading and the response to my article. The first point I want to clarify is that there is no “your solution” we were talking about the general category. The article merely weighed the pros and cons of two forms of flash automation. We also know the astute product quite well and believe it has a roll to play, but at,least as far as I know you need to work with some other software application or leverage something like storage vmotion to get data to and from the appliance. Not that there is anything wrong with that approach, I like the Astute product as long as the customer knows how to fully take advantage of it.
In fairness things may have changed over there at Astute but we have been trying to get a briefing with you guys for over 4 months with no success. Again no ones fault, just busy schedules. I think we have something on the calendar for next month so hopefully we will be able to provide our readers with an update at that point.
George
Sorry I may have not been clear. I meant to state that we are compatible with storage vmotion. I can ping my VP of marketing if you wish but don’t want to butt in if things are planned.
Great questions, a simple response via the comments section won’t do it justice. Let me work on a column that will address these point in more detail.