Big Data is not just for big enterprises. Small to medium-sized enterprises (SME) can reap the same rewards. But SMEs can’t build a Big Data infrastructure like larger enterprises. The good news it they don’t have to, they can leverage the compute capabilities of the cloud. The challenge is how to get the right data to the right cloud at the right time.
Big Data Value
Big Data can come in many different forms, such as Internet of Things (IoT) devices, log data from internal systems or a correlation of data from multiple legacy applications. Organizations can use this data to help with decision making, problem solving or new product design.
Big Data has two specific requirements that have in the past excluded SMEs from participation. First, in most cases, Big Data requires a scalable storage infrastructure to accommodate all of this source material. Second, it requires a scalable compute architecture so that data sets can be processed quickly enough to make near real-time decisions.
The Cloud Conundrum
The default answer for a scalable compute and storage infrastructure is the public cloud, which can provide an almost limitless quantity of both. But the cloud may not be the most ideal location to store data. Data has gravity and while the ability to scale-up is required, it is very rare to “scale-down” storage. The result is the periodic cost of storing all of an organization’s data in the cloud over a long period of time becomes prohibitive. Additionally, on-premises object storage offers very similar ease of management and expansion while being less expensive over time.
The compute side of Big Data processing does scale-up and scale-down. Typically there is a set moment where a result or answer is needed, a range of data needs to be processed as quickly as possible so the organization can make a decision. An ideal public cloud use case, if the data is there.
The advantages of cloud compute forces many organizations to also use cloud storage so processing can occur quickly. As a result SMEs are forced to go “all-in” on the cloud even though they’d prefer not to.
The SME Cloud
For larger application datasets, the ideal SME cloud is compute only, with data loaded on an as needed basis, essentially caching data from on-premises storage to the public cloud compute. The design is the exact opposite of the typical cloud gateway where most of the data is in the cloud and active data is cached on-premises. The SME Big Data Cloud keeps data on-premises and then caches data temporarily to the cloud for processing.
The SME Big Data cloud is also unique in that the cache has to be more sophisticated than the typical first-in, last-out technology. While that approach is fine for a default action, SMEs need the ability to override the behavior and initiate compute on-demand by caching data needed for processing next to cloud compute. For example, a small, independent research organization could leverage unlimited compute resources to support grant cycles and complete necessary work without infrastructure investment.
When used correctly the cloud can be the great equalizer, enabling the SME to go toe-to-toe with larger enterprises. It is key, though, that the SME leverage just the aspects of the cloud that they actually need to optimize costs and derive the most value.
To learn more about how SMEs can better leverage the cloud for their Big Data needs watch our on demand webinar, “10-Minutes to Cloud: How to Quickly Shift Big Data Processing“.