Can Data Profiling Solve The Data Epidemic?

It comes as no surprise to any IT professional that unstructured user data is growing. What may surprise them is the rate at which that growth is occurring and the lengths of time that data needs to be retained. The good news is that, at least so far, the storage industry has been able to keep up with the capacity required to store this data increase – almost to a fault. But that “good news” is also bad news as now potentially PB’s of information is stored in data centers and there is virtually no understanding of the underlying value, importance, or risk of storing all this data.

Traditionally, an IT team may reach out to a storage vendor or storage integrator for a storage assessment engagement. The problem, other than the expense, is that storage assessments are often a one-time event the vendor uses to sell you more capacity and the data captured is rendered meaningless a few weeks after it’s created. Instead, a simple but ongoing process needs to be developed that provides better insight into the information stored across the enterprise so that the cost to store data can be reduced, the value of that data can be realized and the risk associated with that storage can be mitigated. When a storage assessment becomes an integrated process we call it “Data Profiling”.

What Is Data Profiling?

Data profiling is the process by which unstructured user data residing in a variety of sources can be efficiently scanned, indexed and classified so that the appropriate actions can be taken against that data. It is not a part of backup or archive processes, although it can certainly augment and improve those processes, since it provides organizations a way to streamline how they ultimately manage and retain this information. It has significant value over a one-time storage assessment. The results of ongoing data profiling produce a living database of information that allows data to be better managed and better mined for future value and data center costs trimmed and controlled throughout the year.

Data profiling is different than a data assessment which is typically performed by a storage vendor or integrator. The goal of most assessments is, essentially, to sell more storage to the end user. Data profiling becomes an integrated part of the data center’s storage management processes and is designed to reduce storage investment costs while increasing data intelligence.

How Data Profiling Works

Data profiling, available from vendors like Index Engines and their Catalyst Data Management platform, will typically be implemented into the data center via a dedicated appliance or run as a virtual machine within the virtual infrastructure. It then scans a variety of sources, including network shares, email stores and document collaboration databases (like Sharepoint) to categorize data by various metadata attributes like creation and modification dates, individual and group ownership as well as a detailed context index, if desired.

A data profiling process is usually implemented long after TBs of information are created and stored on a variety of storage repositories. It is important then that the data profiling solution be able to rapidly scan and ingest data stored on NAS shares, as well as be capable of scanning and indexing off-line copies of data, like those stored on legacy tape media created by backup and archive applications. If scanning, ingesting and indexing of information can be done quickly, this also allows for continual updated scans of the environment to be performed. This allows better decisions to be made on data contents since they are always based on the latest information.

Data profiling uses the latest technology that allows efficient indexing and metadata extraction. This allows networks to be scanned with extreme speed and efficiency. The resulting data index of metadata is small and compressed so as to not require significant storage capacity.

A data profiling solution should also provide defensible data movement and deletion. Many of the tools used in today’s environment will copy and migrate data and corrupt the integrity by changing owners to “admin” and last accessed date to “today”. Using reliable data profiling, an audit trail of each data removal or archive step is produced that indicates who and why data was altered in the environment. With this information recorded, organizations can explain their actions when needed even if it is years after it occurred.

The Data Profiling Pay Off

Armed with continuously updated information about the enterprise’s online and offline data, organizations can make safe and accurate decisions about the data they are storing. The most obvious goal is using data profiling to reduce the cost to store the organization’s information assets. This can be done in two ways. First, redundant copies of data can be identified and safely removed in a defensible fashion. Secondly, data that has long outlived its business value can be outright removed from the environment; again in a defensible fashion. This may be data that is old and never accessed or data that is abandoned, having been created by users that are no longer with the organization. Redundant data can account for up to 30% of utilized storage capacity with abandoned and aged data consuming an additional 40%. What organization wouldn’t want to migrate this content off of expensive primary storage systems and skip a year of capacity upgrades?

The second benefit is that the organization is now better prepared to handle litigation data requests. They can respond rapidly to eDiscovery and regulatory requests and be more capable of managing retention requirements. When a discovery request comes in or a new data compliance policy is set, a compliance manager or even IT can search across data assets simply and rapidly to identify the data impacted by that change or request. Data profiling in effect brings a “Google” like search interface to the large majority of the unstructured and semi-structured data that is stored in the environment. Finding “John Doe’s” mailbox or a four year old contract is now a quick and painless process versus a disruptive and expensive proposition.

The third benefit of data profiling is how it improves other processes in the data center. Primary storage is enhanced because now only the data routinely being accessed is maintained on that storage resource. This means that the amount of premium priced storage that needs to be purchased should be reduced significantly. The archive process should also benefit significantly as data can be moved to the archive based on value and not just the rudimentary “last modified date” parameter. Cost effective cloud storage can be leveraged and populated with data that is aged but should be kept for longer term retention. Discretion can be applied by data owner, individual or group, as well as specific content within the document.

The backup process, however, may see the most benefit from a data profiling implementation. For example, the knowledge provided by data profiling instills greater confidence in the overall archival decision making process. Not only is the right information being archived but that archive action is being fully documented. With that data removed from primary stores, far less data, both in terms of size and number of files, needs to be protected and the backup window can shrink significantly. Additionally, redundant data stored on disk and tape backup targets can be identified and either removed or consolidated.

Conclusion

The time for data profiling is now. The amount of unstructured data that organizations store is at an all time high, as is the importance to retain and re-use that data. In fact, Index Engines claims that on average 40 to 60% of data is misclassified and wasting valuable data center budget. Once properly classified data disposition strategies can be implemented and the content can be tiered, migrated or even defensibly purged in order to optimize the data center. The challenge is there may be a need for this data some time in the future. Data profiling allows data to be moved to less expensive media AND to actually be found when it is needed.

At the same time, the data requirements for litigation holds are still an important issue. The good news is data profiling software can now benefit from more powerful processors and networking speeds allowing for more rapid but more detailed content ingestion.

Since the paybacks on data profiling can be very significant, it is no longer necessary for the organization to wait for a lawsuit to justify an investment in the technology. Data profiling provides an immediate ROI with its improvements in primary storage efficiency, the enablement of the confident use of archives and the reduction in backup workloads and storage capacity. As a result, the improvements in these areas will allow the data profiling investment to pay for itself long before it is needed to respond to a lawsuit.

Index Engines is a client of Storage Switzerland

Click Here To Sign Up For Our Newsletter

About George Crump

George Crump is the Chief Marketing Officer at VergeIO, the leader in Ultraconverged Infrastructure. Prior to VergeIO he was Chief Product Strategist at StorONE. Before assuming roles with innovative technology vendors, George spent almost 14 years as the founder and lead analyst at Storage Switzerland. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Before founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators, where he was in charge of technology testing, integration, and product selection.