Modern Applications and Object Storage

New applications like Cassandra, Couchbase, MongoDB, Hadoop, Splunk and Spark are the catalyst behind data center modernization. These apps provide the scale and flexibility that increase user engagement, enable better decision making and unleash more human creativity. But these solutions also need a new infrastructure that will rapidly respond to their demands. Technologies like virtualization and now containers enable compute resources to be more flexible. Now, the focus is on the storage infrastructure to do its part. For many data centers, object storage is the key to making sure storage can keep pace.

The new generation of applications has two key attributes. First, they are designed to leverage commodity hardware to keep costs at a minimum. Second, they are designed to scale. They scale by using compute servers as nodes in a cluster. They are able to divide up the workload across these nodes, and take advantage of the addition of new nodes. And again, because these nodes are commodity, off the shelf hardware, the cost to scale is minimal.

Most of these new applications, in addition to managing structured data, also relate that structured data to files or objects (unstructured data). Other apps are primarily designed for the analysis of unstructured data. In either case, the amount of data, especially the unstructured data that they have to manage, is unprecedented. This unstructured data is more varied than in the past. It can be rich media (video/audio) or it can be data from Internet of Things (IoT), which while small, can number in the billions of files. As a result data management and performance is critical to a successful rollout.

Most of these modern application environments try to solve the storage problem themselves by integrating storage into the server nodes and then either replicating or striping data across those nodes. Some even try to manage data, shifting it between RAM, Flash and hard disk drives as appropriate. The problem with this approach is that it consumes compute to manage storage. It also creates some availability concerns.

Object Storage for Modern Applications

These applications benefit from having a dedicated storage tier just like legacy applications did over a decade ago. Those legacy applications were typically scale up in nature, one big server with as much compute and networking as the organization could justify. Modern applications also need a dedicated architecture but one that better matches their scale-out, commodity design.

Object storage is ideal for these modern applications because it’s very similar to the compute design. Object storage leverages commodity hardware and scales by adding servers as nodes to the cluster. Administrators can add or subtract capacity and performance simply by adding or removing nodes. In other words, it can scale as quickly as the compute infrastructure it is supporting scales.

Object storage systems can also support near limitless number of files as well as the variety of workload types.

The Object Storage Challenge

The problem facing most organizations as they implement an object storage system to support their new applications is the almost erector set delivery model that most object storage vendors put companies through. IT professionals have to select the object storage software, the hardware that it will run on, the media/medium that will store the data and the networking that will interconnect the nodes. If the application team is leading the project, they often don’t have the experience in all these technologies to get them quickly and successfully installed – to say nothing of operating them long term.

Even if the storage team is installing the object storage solution, they may not have familiarity with object storage and they almost certainly don’t have the time to put the solution together. In these situations organizations need to look for more turnkey solutions, like the Dell EMC Elastic Cloud Storage (ECS) product family, that enables a more rapid install and easier operations, while still providing flexibility and being cost effective. Rather than waiting for storage administrators, ECS allows developers to provision the storage they need when they need it. The result is that development teams can create and deploy new applications more quickly.

StorageSwiss Take

Object storage is an ideal complement to modern applications. It can scale to meet both the capacity, object count and performance demands of these environments. Object storage also provides a storage infrastructure that is similar in design to the compute infrastructure, but is dedicated so that storage capacity and performance is independent of the compute demands. But organizations need to be careful when selecting a system. Companies may find more turnkey systems are a better match for their needs and time availability.

Sponsored by Dell EMC

Twelve years ago George Crump founded Storage Switzerland with one simple goal; to educate IT professionals about all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought after public speaker. With over 25 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS and SAN, Virtualization, Cloud and Enterprise Flash. Prior to founding Storage Switzerland he was CTO at one of the nation's largest storage integrators where he was in charge of technology testing, integration and product selection.

Tagged with: , , , , , , ,
Posted in Blog

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 21,863 other followers

Blog Stats
  • 1,235,047 views
%d bloggers like this: