How to Prevent Noisy Neighbors

In response to my recent article, “What Is A Noisy Neighbor”, one of our reader’s asked if limiting the number of IOPS (via VSphere) on all but the most disk I/O intensive VMs would be a good approach for reducing the risk of noisy neighbor issues (until a better SAN could be implemented). While this approach would help ensure that the most performing demanding VMs would have priority access to storage resources, there are some caveats with this approach.

First, noisy neighbor issues are much more likely to take place from those VMs that have a high demand for storage IOPS. So in this example, limiting the IOPS of the less disk intensive VMs would do nothing for you if any one of the unrestricted, performance demanding VMs suddenly started to monopolize all the IOPS on the storage controller.

The other challenge with this approach is that it doesn’t scale very well. It may be effective if you only have to manage a handful of VMs on a single hypervisor server but if you are supporting multiple hypervisor servers and dozens (or more) of VMs that all share the same storage system, trying to determine where to place I/O limitations could prove to be burdensome.

For example, one VM could be relatively inactive today but could then become much more active tomorrow. This would require the administrator to manually reset the IOPS limitation parameters to meet this new demand. Over time, the lag time between making all these changes could negatively impact application quality of service (QoS) across a number of applications.

An upgrade to a storage system which provided controller based QoS could effectively manage this issue if the array provided separate performance tiers which fenced off resources. In other words, a hybrid array with high performance flash/SSD and conventional hard disk drives. VMs with high I/O requirements could be allocated pure flash access, for example, while other less demanding VMs could have a portion of their data sitting on flash or SSD, with the remaining data residing on conventional disk storage.

Since flash can service up I/O requests much more rapidly than conventional storage, it can help mitigate noisy neighbor issues. For those environments that can justify it, all flash storage arrays can essentially eliminate any storage I/O contention issues; with perhaps the exception being faulty software code which puts an application into an infinite I/O loop.

Another option, as discussed in the article, is to deploy server-side caching software that can promote hot data sets into storage resources directly on the host. By utilizing server DRAM, flash or SSD, caching software enables businesses to accelerate application I/O directly at the source where the application lives.

The benefit of this approach is that it allows administrators to either use existing resources to accelerate performance or to selectively upgrade certain servers with flash or SSD to improve application performance. This approach has the added benefit of freeing up the shared storage resource from having to process all the “hot” I/O requests. As a result, this can help improve or at least stabilize performance for the other less performance demanding VMs that rely on the SAN for access to data. In many cases, this approach this can actually extend the active service life of the shared storage array and enable organizations to defer costly storage array upgrades.

Lastly, having software tools which can capture baseline information on physical and virtual server performance becomes indispensable as environments start to grow. These tools can enable administrators to be pro-active about migrating application workloads to prevent contention and ensure QoS. Some tools also allow the creation of policies whereby corrective performance measures are fully automated. For example, automating the vMotion of a workload from one hypervisor to another.

In summary, there are a number of ways to help eliminate noisy neighbors from impacting your environment. As always, the right approach will depend on a number of factors. From the size of your environment, to where you are at in your technology refresh lifecycle and of course, budgetary requirements. We’ll continue to help provide guidance on all your technology needs so keep the questions coming!

Click Here To Sign Up For Our Newsletter

As a 22 year IT veteran, Colm has worked in a variety of capacities ranging from technical support of critical OLTP environments to consultative sales and marketing for system integrators and manufacturers. His focus in the enterprise storage, backup and disaster recovery solutions space extends from mainframe and distributed computing environments across a wide range of industries.

Tagged with: , , , , , ,
Posted in Article
One comment on “How to Prevent Noisy Neighbors
  1. Louw Pretorius says:

    Hi Colm, I agree that vm-based IOP-limiting does not scale very well but you have the option of using Storage IO Control (SOIC) from VMware to limit the Queue depth of the whole volume across the cluster to help protect against noisy-neighbours.

    Cheers

Comments are closed.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 22,236 other followers

Blog Stats
  • 1,553,740 views
%d bloggers like this: