New Strategies To Control The Data Epidemic in State and Local Government

The fact that data is growing came as no surprise to even the newest of IT personnel, but the rate at which that data is growing is catching even the most veteran IT professionals off guard. This is especially true in the IT departments for state and local governments who, thanks to the need to store new types of digital assets for record lengths of time, are facing a data epidemic in their facilities. The challenge is that these agencies have limited budget dollars to provide their IT departments to tackle these problems. State and local government IT needs an innovative approach to cost effectively deal with the problem.

What Is Causing The Data Epidemic?

Data growth in state and local government is coming from the “Internet of Things”. The Internet of Things (IoT) is when the Internet and networks expand to places such as manufacturing floors, energy grids, healthcare facilities, and transportation arteries. “Things”, devices or sensors are being placed on and within these structures and are producing data that can be leveraged in real-time to improve efficiency.

For example most state and local government agencies now produce their own video and audio content that is used not only by the agency itself but other agencies and news organizations. In addition, these agencies have leveraged a wide variety of automation to help reduce costs agency wide including surveillance and traffic cameras, sensors that measure weather conditions, solid conditions, water quality, the list is endless. Add to that list the common requirement that these agencies keep all of their more traditional documents (office productivity, email, engineering documentation, etc.) essentially forever to adhere to freedom of information acts.

While the “Internet of Things” has helped drive down the cost to deliver government services and increase the quality of those services can provide, these “things” do place a significant strain on the storage infrastructure. This strain is causing IT to have to reconsider how it manages and stores this data and stands as a precursor to what is to come in traditional enterprise IT.

Typical Approaches To the Data Epidemic

As the agency IT department starts to deal with this data growth, they typically follow a familiar path that often starts with using external USB drives. These drives are connected to the primary servers storing all of this “things” generated content. Then as they fill, a new USB hard drive is connected and the old unit is stored on a shelf. This strategy represents classic reactionary thinking driven by an unexpected growth of data and a predictable lack of funds.

As these agency IT departments try to get this data growth under control, they follow the next logical step; buy a disk system large enough to store all this data. Typically this disk system also needs to support production applications for the agency as well. The problem is that the growth of the “things” generated content continues to grow exponentially and soon the capacity of the system is exceeded or so much capacity is added that performance suffers.

Even if a modern scale out disk storage system is bought or a capacity focused scale up storage system, the ongoing cost to continuously add disk and power and cool that system eats away at the IT budget. Essentially, these IT planners learn the hard lesson that it is not just the incremental cost of adding drives over time that make storage systems expensive, it is the ongoing expenditures of powering, cooling and maintaining those disk drives that becomes too costly.

What About Tape?

Seemingly at the end of their rope, IT Planners at these agencies are now increasingly looking at tape as a viable alternative to disk. But the initial examination brings some concern. For example, while tape is the undisputed champion of cost per GB it may also be the perennial loser when it comes to accessing data. In a tape only environment, users may have to wait several minutes for data to be restored.

Potentially more damaging than the wait time is the method of access. Unlike a hard disk, users have not in the past been able to simply access data with a point and click. Instead they had to call or email an IT person to restore data for them and depending on IT’s availability, that process could take hours if not days. This also assumes that the user would even know that the data they are looking for still exists, since most tape only archives gave no indication of what data had been removed.

The final challenge with a tape based system was that data was written in a proprietary format by the archiving application. As mentioned above, state and local agencies have a requirement to keep this data available for a very long time, potentially forever. This means keeping the archiving application running for a long period of time or doing a costly migration of media whenever a decision to change applications is made.

Innovation From An Old Source

In order for tape to be practical in state and local government it needs to be able to present data in a click to restore fashion. Doing so allows data to be presented to users seamlessly. As a result IT does not need to be involved when users want to access older data sets. The first puzzle piece of this seamlessness presentation already exists in the form of tape libraries that automate the loading of tape media based on what data is being requested.

The end puzzle piece is also available, application independent tape format compatibility. LTO tape media has had a format called LTFS available for several years now. If a solution leveraged this tape format, it would be able to provide the comfort of application independent data storage.

The remaining need is the middle puzzle piece; an application that can take advantage of the other puzzle pieces and integrate a few more to overcome tape’s access weaknesses. This will require a solution that can integrate tape libraries with traditional NAS storage to create an infinite storage area. The solution that does this integration will then need to manage the movement of data between disk and tape based on policies like data age and data type. What the users of this solution will see is NAS-like response time on their most actively accessed data sets and seamless access to older data sets that are on tape.

Solutions like this exist today from companies like Crossroads, whose StrongBox solution provides not only the seamless integration between disk and tape but also the cost effective, application independent, long term storage of data.

Net Result

The most obvious benefit of a StrongBox like solution to state and local government agencies is the creation of an infinite storage area to cost effectively solve the data epidemic that they are facing. The automated movement of data to a long term tape archive promises to reduce storage expenditures, as well as reduce data center floor space and power requirements.

There are also the hidden benefits of moving this type of storage solution into the mainstream of data center storage. If this type of solution becomes the primary NAS target for the agency, the burden of backup and disaster recovery planning is also greatly reduced. These systems can automatically be set to create redundant copies of all data written to disk, with the second copy being moved off-site. In one fell-swoop, unstructured data and data protection can be removed from the IT project list.


State and local governments are facing a data epidemic that is potentially greater than in the private sector. Not only are they dealing with at least as much data from potentially a greater variety of sources, they also have the requirement to make all that data accessible, potentially forever. Combating this epidemic requires new innovation from an old source – tape. Rather than competing against disk, tape compliments and extends the useful life of disk storage assets.

Crossroads is a client of Storage Switzerland

Click Here To Sign Up For Our Newsletter

Twelve years ago George Crump founded Storage Switzerland with one simple goal; to educate IT professionals about all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought after public speaker. With over 25 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS and SAN, Virtualization, Cloud and Enterprise Flash. Prior to founding Storage Switzerland he was CTO at one of the nation's largest storage integrators where he was in charge of technology testing, integration and product selection.

Tagged with: , , , , , , , ,
Posted in Article
2 comments on “New Strategies To Control The Data Epidemic in State and Local Government
  1. […] But now tape is moving into more traditional IT use cases as well. As we discussed in a recent article an explosion of digital content data sets and the requirement to keep those assets for extended […]

Comments are closed.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 22,263 other followers

Blog Stats
%d bloggers like this: