Cloud storage offers data centers almost limitless capacity and they can purchase that capacity as they need it. But to this point, most use cases for the cloud have been as a secondary storage target, a place to store archives and backups. But what if, keeping compute in the data center, the cloud could be used as primary storage? Assuming IT can overcome the obvious latency, the benefits of not having to pay for, manage and maintain storage could be significant.
Before going into the benefits of a cloud primary storage the key concern over latency has to be resolved. If it can’t be solved then cloud storage is a non-starter, even if cloud storage brought a thousand benefits and was free to use.
Dealing with the Elephant in the Room – Latency
Cloud storage, at least from a performance perspective, is more than adequate for most workloads. But the time it takes to send a bit of data from the primary data center to a cloud service provider, and receive confirmation that the write has occurred, a.k.a latency, is a legitimate concern, especially for applications that are far more sensitive to response times than files.
The Problem with the Appliance Workaround
The solution thus far has been to use appliances in the primary data centers that essentially act as caches. The most active data is stored on these appliances and as long as the data being requested by the application is on them. The problem is if there is a cache miss the gap between on-premises response and cloud response too great. A cache miss of this magnitude will certainly be noticed by users and more than likely will cause applications to crash.
The typical cache best practice is to size the cache to 5% of total data. The theory being that most of the capacity in the data center, roughly 80%, has not been assessed in 90 days, so a cache that is 5% would have the required data more often than not.
In reality, most caches sized to this practice see about a 60% hit ratio, which if the cache is flash in-front of a hard-disk array is barely acceptable by today’s standards. If the tier below the cache is the public cloud instead of a hard disk array, then a cache miss will lead to complaints about performance and potentially application crashes.
A workaround is to make the cache large enough so a cache miss would be extremely rare. Size the cache to 25% or even 50% of capacity. Taking this approach will certainly reduce the chances of a cache miss but it does not eliminate them. A cache miss, no matter how rare, could still have a very negative impact. The other problem is that a 50% of capacity cache sizing also eliminates much of the value of storing data in the cloud, the size on the on-premises storage still remains a point of concern.
As a result of these challenges the cache direct to cloud model has primarily been limited to unstructured file data use cases. This data type is more predictable from an access timeframe, it usually is very active and then goes very dormant. Also file systems are much less sensitive to response times and won’t time out waiting for a few seconds for data to be retrieved. Applications and virtual servers are the exact opposite, the way they access data is far less predictable. In some cases server files and application files never go dormant for very long. Applications and virtual machines are also far more sensitive to response times after a give access. A few seconds is all the allow.
The All Cloud Solution
Since the big challenge with using the cloud as primary storage is latency caused by the distance between the data center and the public cloud facility, the obvious solution is to move the two closer together. One solution is to move all of the organization’s applications to the cloud. While there are certainly application types that work well in the cloud, not all do. More importantly, most organizations simply don’t want to give cloud providers everything.
The Hybrid VMware Solution
An alternative, and one we explore in our on-demand webinar “How to Leverage Cloud Storage for Hybrid VMware”, is to establish an intermediary storage location. Essentially continue to follow the 5-10% sizing best practice in the data center, but have another tier of storage much closer provided by a regional service provider. In this architecture data is only milliseconds away, fast enough that most users won’t notice and applications won’t crash. The result is the organization can significantly reduce the physical footprint of on-premises storage without impacting the user experience.
In this architecture, all writes are written to the on-premises appliance and to the regional provider simultaneously. Reads are serviced directly from the on-premises appliance providing instantaneous response. All data is also replicated to the public cloud provider and is usable there for a variety of use cases.
Since its adoption by the enterprise, storage has been a thorn in the side of VMware deployments. Making sure the system delivers the right amount of performance, scaling and adding capacity and protecting the data are a constant struggle and detract from VMware’s return on investment. Leveraging a hybrid cloud architecture to host the storage resource gets VMware administrators out of the storage game. They can focus on their front line applications and storage automatically adjusts to meet the ever changing workloads it supports.
In our next column, we will detail how the organization can leverage this architecture to solve a variety of problems including backup, disaster recovery, peak load bursting as well as copy data use cases like test-dev, DevOps and analytics processing. In the meantime watch our on-demand webinar to learn how to leverage the cloud as primary storage for your VMware environment.