How To Scale Object Storage

Scalability has always been an important characteristic of enterprise storage systems, maybe the most important, since the rationale of consolidating storage assumes that the system can easily grow with aggregate demand. With applications like cloud storage, the ability to scale can be the defining characteristic indeed. However, legacy storage architectures are proving to be inadequate when it comes to supporting the kinds of data growth that’s all too common in today’s data center environments. Object Storage Systems are now stepping in to address this issue.

Storage Scalability

In order for a storage system to scale it must hold more raw capacity, of course, but it also needs to process more metadata. This is the information that controls where the data is stored and how it’s accessed. Metadata also supports other functions, such as data protection and storage system features. Clearly, the system’s ability to process metadata can directly impact how large it scales.

The amount of processing power a system has can govern how much metadata it can administer, similar to the way a larger storage controller can handle more data I/O. This is why scalable controller architectures can support greater overall capacity. In addition, the number of background processes the controller has to perform on that metadata for things like data protection also impact how much CPU power is left for storage growth.

But efficiency also plays a part in metadata processing, specifically the way metadata is organized and handled. This can be especially true with network attached storage (NAS) systems, some of which require an unusual amount of system resources to be reserved in order to process snapshots and other OS overhead.

Object-based storage architectures offer some significant advantages when it comes to metadata processing that enable these systems to scale much larger than traditional storage architectures. Before examining how object storage systems scale, we should look at how traditional storage systems have handled growth.

How Traditional Storage Scales

Traditional storage arrays have a central controller which handles data I/O, data protection (RAID and replication, typically) and supports storage features like snapshots. This kind of physical architecture adds disk drives behind the storage controller, which in most cases has a fixed amount of processing power, and ‘scales up’ as it grows. This can create a bottleneck in I/O processing, limiting the amount of capacity that the system can support.

Scale-out storage architectures addressed this physical limitation, to some extent, by spreading the controller function out into multiple modules, as they did with the storage capacity. This enabled them to scale processing power as disk drives were added, effectively increasing their overall capacity.

But, like scale-up storage, they use a logical data architecture that stores data in blocks on the disk drives and organizes metadata in a hierarchy, somewhat like a file system tree. As the capacity grows the amount of data blocks grows and more importantly, the amount of metadata required to handle those blocks also grows.

When these systems get very large (in capacity and number of files), as they can in cloud applications for example, the way they organize metadata can impact how well they scale. In the traditional storage systems, the inefficiency of their hierarchical structure can become a limiting factor to growth.

How Object Storage Scales

Object storage systems, on the other hand, put data into objects, discrete ‘containers’ that each have a unique identifier, called an object ID (OID). Each object is accessed by its OID which can then provide access to all the data within that object. The result is a flatter, simpler metadata structure, more like an index than a hierarchy. This reduces the processing steps required to access data and improves the efficiency of the storage system. This efficiency in turn allows object storage systems to grow much larger than traditional block based storage systems.

But there are differences between object-based data architectures as well. Most object storage systems use a centralized metadata store which runs all metadata processing through a common module or node. Like scale-up physical architectures, which can bottleneck data I/O processing, centralizing metadata can bottleneck the metadata processing, and like legacy storage controllers, create a significant limitation on scalability.

Object storage systems like Cleversafe use a decentralized metadata architecture which allows the processing function to scale along with the amount of metadata that a growing storage system would generate. They actually store metadata as separate objects, like they do regular data, and distribute them into the object storage space as well. This potentially adds a step to the process for fetching data if metadata is used to locate the OID, but for most web-scale environments it’s a small trade off to get near limitless object-count scalability. By distributing the metadata index across thousands of storage nodes they can produce extremely traversing and location of objects within the system.

Erasure Coding

Some object storage systems use a technology called “erasure coding” to provide data protection and ensure long term data integrity because it’s more efficient than traditional RAID and replication methods. As discussed in the Storage Switzerland article “A Better Answer than RAID and Replication for Cloud Storage” eliminating RAID and replication can reduce the amount of redundant data the system must carry. This can directly increase the amount of ‘new data’ the system can hold and also reduce the metadata associated with these redundant copies, freeing up those CPU cycles to support more capacity.

But many object storage systems only perform erasure coding at the disk-shelf or the storage-node level, greatly reducing its efficiency. Systems like Cleversafe perform erasure coding at the storage-system level, maximizing the ability of this technology to reduce data stored and significantly increasing system reliability. This type of erasure coding also enables geographic distribution of data for improved disaster recovery protection.

This technology also distributes the metadata processing and data storage into different modules, enabling the storage nodes to participate in the data integrity process. By directly contacting other storage nodes they can perform the ongoing data checks which ensure data integrity. And when a problem is found, they can conduct the object rebuilding process at the storage node level as well, all without creating overhead for the data ingest/access nodes.


Scalability is a key attribute of storage systems, especially in cloud environments. Aside from expanding physical capacity, scaling a storage system means handling more metadata, the critical information about where data is logically stored. Metadata grows along with the data itself and processing it can become one of the limiting factors to system scalability.

Storage controllers must conduct this metadata processing and distributed architectures, like those used in scale-out storage systems, can reduce the CPU bottlenecks which limit metadata processing. But the efficiency of the way traditional storage systems organize and handle metadata can still be a problem.

Object storage systems replace the traditional hierarchical metadata structure with a more efficient structure which uses object IDs and a simple index to access data. Some object storage systems, like Cleversafe, take this a step further by distributing metadata storage and processing similar to the way a scale-out storage system distributes the storage controller function. This can significantly increase the system’s ability to efficiently handle metadata, dramatically boosting its scalability.

Cleversafe is a client of Storage Switzerland

Eric is an Analyst with Storage Switzerland and has over 25 years experience in high-technology industries. He’s held technical, management and marketing positions in the computer storage, instrumentation, digital imaging and test equipment fields. He has spent the past 15 years in the data storage field, with storage hardware manufacturers and as a national storage integrator, designing and implementing open systems storage solutions for companies in the Western United States.  Eric earned degrees in electrical/computer engineering from the University of Colorado and marketing from California State University, Humboldt.  He and his wife live in Colorado and have twins in college.

Tagged with: , , , , ,
Posted in Article

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 22,223 other followers

Blog Stats
%d bloggers like this: