When it comes to business transformation, numbers matter. For CMA, a leading healthcare application service provider for Medicaid payments systems processing and data analytics, the integration of EMC XtremIO led to a transformation of their workloads, their infrastructure, and their profit margins. This includes an eight-fold reduction in the amount of data center floor space required to host their application environment. CMA’s implementation of XtremIO has also helped drive down hardware and software costs, dramatically improved application response times, reduced administrative overhead by 50% and has enabled CMA to significantly accelerate the time it takes to assemble, configure and deliver their solution to their clients.
What Does CMA do?
As a service provider, CMA develops a product offering called MicroTerabyte. MicroTerabyte is based on an Oracle RAC cluster compute engine and is used to manage some of the largest Medicaid state entitlement systems nationally. These systems process millions of patient claims per day while servicing highly I/O-intensive analytic and database query requests for thousands of simultaneous users. Consequently, latency and IOPS performance is critical. The challenge was that the MicroTerabyte platform had been highly dependent on the deployment of multiple racks of hard disk drives (HDDs) to provide the high I/O throughput and low latency required to deliver its application performance service levels.
An Eight-Fold Hardware Reduction
Since adopting XtremIO’s scale-out flash array for their MicroTerabyte platform, CMA has seen a dramatic reduction in the amount of data center floor space required to house their solution. What formerly occupied eight fully configured cabinets in the data center now occupies a single 42U rack. This is translating into significant floor space and power savings for CMA and their clients, in addition to several other major business benefits.
Faster Time-to-Market for New Services & Clients
The reduction in the amount of physical storage infrastructure required has also reduced the time it takes for CMA’s product and engineering team to build out a MicroTerabyte system for a customer deployment. What used to take CMA’s engineering team multiple days to assemble and configure, can now be completed in a matter of hours. This is due to both the ease of use of configuring the XtremIO array and bringing it online and the physical simplicity of deploying an array with a very compact form factor. This dramatic configuration simplicity and consolidation results in a faster time to value for the client while enabling CMA’s product integration team to fulfill more orders and drive more revenue for the business.
Lower TCO, Higher Margins
CMA has also realized a significant reduction in their application development costs due to the high I/O throughput and low latency that XtremIO delivers to the MicroTerabyte Oracle RAC cluster compute engine. Thanks to this all-flash Oracle RAC design, CMA can now utilize commodity Linux clusters with fewer CPU cores over the multi-core proprietary Unix systems that they had formerly been using. This has translated not only into lower hardware costs, but it has also reduced their software costs as Oracle RAC licensing is based on the number of cores in a cluster. CMA has found that this XtremIO-based all-flash Oracle RAC solution actually drives down software costs delivering a better total cost of ownership (TCO).
These savings are being passed along to CMA’s clients as the MicroTerabyte platform can now support increased levels of performance with correspondingly fewer hardware and software resources. In addition, it is helping CMA to improve the end-user experience, respond more quickly to their clients’ demands and lower their costs while increasing profitability. In our next post we will detail CMA’s journey from the stopgap use of flash to flash as a vital part of their storage strategy.
All-Flash in Real Life – Moving Beyond Stopgap Flash
CMA is a leading healthcare application service provider for Medicaid payments systems processing and data analytics. CMA’s foray into the use of flash started approximately eight years ago when Brian Dougherty, Chief Architect, and his team began integrating flash into their traditional storage arrays. While this enabled them to achieve some performance improvements, it was merely a stopgap measure. The challenge was that these arrays could only support a relatively small compliment of flash capacity so the only way they could ensure consistent, predictable performance for their critical business applications was to maintain a heavy compliment of multiple, high speed HDDs. Also at that time, flash drives were expensive and lacked the data reduction features necessary to economize on Oracle data flash storage consumption.
HDD Cost, Complexity and Risk
To attain this performance, CMA’s storage architects had to spend up to 50% of their time designing, configuring and tuning their arrays to meet the I/O needs of MicroTerabyte’s key data analytics and payment processing applications. Provisioning storage was a 12 step process that in part consisted of building multiple RAID sets and types across dozens of HDDs, configuring flash pools and non-flash pools, short-stroking disk drives, creating metadata volumes, etc. and then deciding how to best lay data out across all these resources. This was a very complex process that left little to no margin for error. One small configuration mishap could negatively impact performance on a critical database system and result in a flood of calls to CMA’s support center and make for an unhappy customer.
Operating in Silos
Due to the complexity of the storage environment, CMA’s engineering team essentially had to silo themselves off from their application development counterparts. In other words, application and database administrators that lacked the highly specialized knowledge to configure these storage arrays could not fully collaborate in the storage configuration process. Due to this knowledge barrier, CMA storage engineers often times had to make critical storage design decisions without direct input from the people that developed the underlying application. This is an example of how infrastructure complexity was impeding CMA’s ability to fully harness the power of their most valuable asset to drive continuous business innovation – their people.
Storage Redesign Mission
Due to these challenges, Dougherty and his team embarked on a mission to fundamentally redesign the storage infrastructure back-end of the MicroTerabyte platform. Their objective was to reduce the form factor of a typical deployment down to a single 42U rack and remove all of the design and configuration complexity out of assembling the solution. By doing so, this would enable CMA to accelerate MicroTerabyte product delivery time frames while speeding up customer production deployments.
They also needed a way to eliminate the need for continuous monitoring, analyzing and re-tuning of the storage environment. In short, they were looking for a way to make application performance consistent and predictable, regardless of data growth, without requiring heavy administrator intervention. In order to effect these changes, CMA realized that they needed to supplant their heavy reliance on HDDs by adopting a flash array to front-end their critical applications.
The Flash Four Key Criteria
The CMA storage team came up with four key criteria for what they needed from a flash solution. Interestingly, performance fell to the bottom of this list in terms of overall importance. While very good performance was still an important attribute, Dougherty and his team decided that other factors deserved stronger consideration.
The reliability of the solution was ultimately going to be one of the top determining factors in their decision making process. As Dougherty put it, “When supporting a 24×7 solution, we couldn’t be in a situation where we were experiencing flash controller problems, drive related failures or interoperability issues with other layers of the infrastructure stack.” Indeed, since MicroTerabyte is marketed as a solution for enabling quality outcomes for health care patients and providers, system uptime, reliability, and flash endurance are of paramount importance.
The second criterion was simplicity of management and ease of use. Dougherty’s team was already spending half of their time designing, assembling, configuring and provisioning storage resources on the MicroTerabyte platform. Another chunk of their time was being spent monitoring, analyzing and tuning these systems post-deployment. And as stated earlier, due to the overall complexity of the storage provisioning process, there was always the risk that a minor configuration error could blow up into a major performance problem. So an easy to use, intuitive storage platform that didn’t require extensive storage expertise was a critical requirement.
Vendor Strength & Support
Vendor track record and reputation made up the third key criteria. The flash array landscape is composed of traditional storage vendors along with some industry newcomers. So one of Dougherty’s concerns was vendor reputation along with their commitment to the ongoing development and maturity of their product. In other words, anything that was going to integrate into their flagship MicroTerabyte offering, had to be supplied by an organization that was capable of providing quality support and was committed to the continuous innovation of the solution. Another key concern was if a vendor was purchased, would that disrupt or alter the development cycle.
Scale-out Consistent Performance
The final criterion was that the flash solution not only had to provide good performance but it had to provide consistent and predictable scale-out performance. As new customer workloads pile on and existing workloads expand, scale-out architecture is key to be able to add performance and capacity. Dougherty stated, “Once users become accustomed to the speed of flash, any deviation from flash speeds creates a big problem for us”. In essence, for CMA it was less about the ability to attain the highest flash I/O throughput speeds possible and more about integrating a flash architecture that could deliver a consistent end-user experience.
Surveying the Flash Landscape
In addition to testing and evaluating various server-side PCIe Flash cards, CMA looked at a whole range of flash offerings from industry start-ups to traditional storage vendors. Their analysis included hybrid arrays, all-flash arrays and as stated earlier, they had previously integrated flash into their existing storage arrays. Ultimately, CMA decided that a shared-storage all-flash array solution would best meet their needs for a reliable and simple solution to manage while providing the scalable storage performance needed for the MicroTerabyte Oracle RAC clustered architecture.
All-Flash Predictable Simplicity
Upon examining all-flash versus hybrid storage arrays, CMA concluded that an all-flash approach would best suit their needs. While some incremental savings could be realized by deploying a hybrid flash system, CMA decided that the consistent and predictable storage performance that an all-flash system would deliver was a higher business priority.
Furthermore, placing all data in flash would make for a much simpler solution since it would remove all of the guesswork out of managing data across a mixed tier of flash and HDD.
This last point was a key goal for CMA since they needed a way to streamline the MicroTerabyte storage assembly and configuration process to accelerate their time to market and increase revenue. So the benefits realized from a simpler storage deployment would more than make up for any added investment in an all-flash array. The decision to go with an all-flash system narrowed the potential field of vendors down to a handful of traditional storage suppliers and a few dedicated all-flash vendors.
Intuitive Flash Management
In addition to the simplicity of design that an all-flash system could deliver, CMA also needed a storage solution that provided an intuitive and easy to use management interface. According to Dougherty, this is where the XtremIO offering demonstrated significant product differentiation. Dougherty stated, “From a complete path link (server to switch to storage), XtremIO was much easier to implement and its management interface was much simpler than the other all-flash platforms that we evaluated”. The simplicity of configuring and provisioning storage with XtremIO has actually cut CMA’s storage administration time in half. CMA’s storage engineers were dedicating nearly half their time with storage configuration tasks but now thanks to XtremIO, Dougherty claims these tasks now only consume about 1/4th of their time.
Bringing Development Into the Fold
In fact, this simplicity is actually fostering better collaboration between the CMA storage and application development teams. Since the storage configuration and provisioning process with XtremIO doesn’t require advanced knowledge of the inner workings of the array, database administrators can now participate in the storage design process. Dougherty claims that this simplicity is helping them to build a better overall solution for their customers.
CMA began testing XtremIO in their labs prior to the EMC acquisition. Since that time, Dougherty states they have seen the product mature while continuing to provide the reliability and stability that is essential for MicroTerabyte. Moreover, with the added feature/functionality of storage services like snapshots, Dougherty claims that CMA continues to find multiple use cases for leveraging XtremIO across the data center. As an example, CMA can use XtremIO to take continuous snapshot copies of production data, without application disruption, to instantaneously refresh their test and development systems.
Providing the data that developers needed to do testing used to take CMA’s storage engineers up to a week of time. Now, through XtremIO’s in-memory snapshot capabilities, CMA’s developers can not only test against near real-time production data, but they can do so at full production workload scale. So rather than conducting application testing on “sandbox” storage systems that are much smaller representations of production workloads, CMA can perform non-disruptive testing on a full-scale production “like” environment. This combination of snapshot provisioning speed at production scale is enabling their developers to do rapid application development so that they can continuously innovate and enhance MicroTerabyte’s key data analytic and attribution applications.
With the recent addition of inline compression to the XtremIO array, CMA is now able to reduce the amount of storage space that their databases consume on XtremIO by over 50% without causing any degradation to application performance. In fact, due to other performance increases XtremIO delivered, the version of XtremIO with compression outperforms the prior version without compression. These increased storage efficiencies are helping to reduce the total cost of the solution while enabling CMA to consolidate data into a smaller form factor on the MicroTerabyte platform.
Linear Scale-Out Performance, 4x Faster Workloads
One of the other chief benefits of XtremIO’s architecture, and a key value that Dougherty pointed out, is the ability of the platform to provide efficient scale-out performance. With each XtremIO “X-Brick” deployed, Dougherty says that they are able to double the amount of storage I/O and throughput available to their applications. This linear performance scalability is critical for enabling CMA to elegantly scale MicroTerabyte’s flash storage resources as their customer environments grow. In fact, Dougherty states that their claims processing application can now run 4x faster with XtremIO under the hood. A claims processing batch job that used to take 4 hours to complete can now finish in 1 hour.
Finally, one of the key determining factors for CMA’s selection of XtremIO was that fact that it is integrated with EMC’s broader technology portfolio, including its leading business continuity and management solutions. EMC has been a long-time vendor partner of CMA’s and according to Dougherty, has a proven track record of providing exceptional solution support. Moreover, it was critically important to Dougherty and his team for the flash product they chose to have the backing of a vendor that has staying power in the market. CMA intends for the XtremIO platform to service the I/O workload requirements of their MicroTerabyte system well into the future. Having the backing of EMC gives them the confidence that the product will continue to mature and provide the increased levels of performance and feature functionality that CMA and their customers will need.
Flash is often first deployed in the data center as a tactical measure to solve a specific application point-performance related problem rather than implemented as a strategic storage asset that can service all critical application workloads. In fact, Dougherty admitted that his organization had also tactically deployed flash prior to CMA’s adoption of the XtremIO platform.
The challenge is that while the tactical use of flash can solve an immediate application performance “pain point”, it can potentially do so at the expense of sacrificing three of the other key flash criteria that Dougherty and his team identified.
Reliability, for example, should be the number one design criteria when implementing flash in a mission critical application environment. If a point-solution lacks the availability and redundancy features necessary to ensure continuous operations, any performance gains realized will quickly be offset by downtime resulting from component failures or microcode compatibility issues between the flash system and the networking/server stack. So flash reliability and maturity is essential.
Likewise, the simplicity and ease of use of a flash system should be carefully examined. Trading off one complex storage environment for another is a big step backwards, especially considering the storage performance management complexity that virtualization and cloud computing technologies have ushered into the data center. Moreover, the desired shift from operating in test/dev IT silos to more of a collaborative and streamlined Dev/Ops environment is only possible if infrastructure planners can drive complexity out of the data center.
Lastly, partnering with a technology vendor that acts as an extension of your own organization can be invaluable; particularly when that vendor’s product is going to be placed at the heartbeat of your business operations. It’s less of a question about IF things will go wrong and more about your vendor’s capabilities WHEN they go wrong. And their ongoing commitment to the maturity and innovation of the solution is equally important towards helping ensure that business application service levels will continue to be met.
For CMA as a mission-critical service provider, pulling all these requirements together through EMC XtremIO, has allowed them to transform their services development, their infrastructure, their economics, and their overall competitiveness.
Sponsored by EMC XtremIO