Traditionally databases are scale up, running on a single powerful but expensive server. Distributed databases like Cassandra and Couchbase are scale-out and run on commodity white box servers, each of which becomes a node in a cluster. Traditional databases are designed to store the entire data set on a storage system and then cache into memory a small working set. Distributed databases fit the ENTIRE working set of data within a RAM memory pool. Distributed databases leverage the redundancy of the cluster to make RAM as a storage tier more practical. But these new in-memory distributed databases present new questions that IT professionals need to prepare for and answer.
Distributed Database Architecture
Distributed databases are built from a cluster of small, typically white-box servers or VMs, called nodes. The memory in these nodes is aggregated, presenting a virtual RAM storage tier. RAM is the preferred storage media choice because distributed database environments typically have large ingest requirements, and no storage media beats the write performance of RAM.
The goal of a distributed database is to keep all of its data in memory at all times. Accomplishing this goal is expensive. There is the obvious expense of using RAM as storage, but there is a limit to the amount of RAM each individual server can support. As a result many organizations that have invested in a distributed database are forced to add servers for memory capacity only, not for compute.
Most organizations can’t justify a memory-only implementation. If the active working set can’t fit in the aggregated memory table across servers then the database will use a storage table that leverages traditional storage. Distributed databases like Cassandra and Couchbase will overflow data from the memory table to the storage table which is a virtual storage pool, like the memory pool but build from hard disk drives. The overflow process is relatively seamless until it comes time to find data. When a request for data comes, the database will first query the memory table. If the data is there then the response is relatively instant and all is well. If the data is not in the memory table then the storage table is searched.
In-memory databases do try to optimize the storage layer to a degree. First, they build search acceleration files so only a subset of the storage table objects need searching. Second, they have a compaction process that will remove old objects so only the most recent version of objects is searched. Despite these optimizations the list of objects can be quite extensive and the time required to search is significantly slower compared to the memory pool, especially considering again that most users use HDD-based storage for this tier.
NoSQL environments like Cassandra and Couchbase are designed to be eventually consistent. This is unlike Oracle, for example, which waits for an acknowledgement of each write. In NoSQL, while transactions are pending they are stored into a commit log, but the database continues on and does not wait for confirmation of the write. Most organizations use SSDs or NVMe flash storage for the commit log.
Detailing Cassandra and Couchbase Challenges
The first and somewhat obvious challenge is that the active data set is moved to the active pool of the cluster. Most servers are limited on the amount of DRAM installed in them. This creates a second problem: many NoSQL environments add additional nodes to increase total memory capacity – not to leverage additional compute. Buying full servers just to access additional memory further increases the cost point of a NoSQL investment and makes ROI harder to achieve.
The second challenge is that most memory is volatile. In the event of a power loss or server outage the potential for losing a few minutes of data is incredibly high. Probably less understood is the long warm up time associated with a restart. In a large cluster, nodes tend to fail, even on a daily basis. When a node fails, the memory tables need recreation from the storage table. Essentially data needs to be re-qualified and put back into the memory table. While this process is occurring, data has to be read from hard disks causing a significant loss in performance.
The final challenge is, “what happens to the NoSQL environment when it runs out of memory?” As previously discussed data spills to the storage table, which is typically hosted on a Linux POSIX file system and typically populated with hard disks. Even if the organization decides to invest in SSD for this tier, there is still the overhead associated with the Linux file system and the SCSI stack. When both IOPS and latency are factored in, flash pales in comparison to DRAM I/O performance.
NoSQL environments used to be the domain of fringe cloud companies, but now organizations of all types and sizes are deploying them. IT professionals may not be prepared for the storage challenges they create nor are they aware of the problems that the typical workarounds cause. Listen to our on-demand webinar “Overcoming the Storage Challenges Cassandra and Couchbase Create” in which we detail these challenges and provide specific solutions to addressing them.