One of the fastest growing data types is video data and organizations of all kinds and sizes are having to deal with it. These organizations are not the traditional media and entertainment companies; now sports teams, public entities, mega-churches and even mainstream businesses are creating large stores of video information. Like traditional media and entertainment companies, making sure this data is readily accessible, while also cost effective to retain, is a major challenge.
Video Data Use Cases
The production studios are of course the traditional companies in the M&E industry. Their use case, while challenging, is well documented. The new wave of M&E companies listed above are creating video information in real-time and then storing it long-term for future use. These organizations don’t have the sequential workflow of production studios. They don’t know when certain video assets will be used.
A common assumption is that most of these organizations are storing mainly video surveillance data. While that data is also growing and is a problem, this new video data is different. Many organizations are doing production quality videos to promote their product or services; others are using video data for analysis to gain a competitive advantage. For some organizations, the video itself is the product. In almost all cases, however, this data will need to be accessed many times after its initial use.
To make matters worse, growth is being compounded. Not only do the raw capacity needs increase as camera resolutions increase, software that can mix multiple cameras is commonplace. This means that a single video, which for a mainstream organization used to be recorded with one camera at a single angle, is now recorded by multiple cameras from multiple angles at increasingly higher video resolutions. Most organizations want to keep all these “viewpoints” for as long as possible.
These organizations are trying to find storage technology that helps them serve this data to their viewers as quickly as possible while enabling them to store that data as cost effectively and as long as possible. The faster the data can be served up, the happier their viewers will be and the longer it can be retained, the greater potential for repurposing at some future point.
The Capture Fast, Keep Forever Paradox
For data that needs to be made available instantly, organizations use high-speed disk or even flash based storage systems. Because of the size of the video data and the cost of this type of storage though they can only use it as a short-term storage location. They need a way to store the longer-term asset that may be repurposed in the future.
Traditional M&E organizations have typically used tape-based archives for this long-term data. The cost and power efficiency of tape is undeniable and concerns about speed of access can be largely mitigated because of the industry’s relatively sequential workflow. In other words, they know when data is going to be needed and can start recalling data in advance. The new M&E companies do not have the luxury of recalling data sequentially, so they often attempt to go with a disk only approach.
Fast Capture is Easy, Long-Term Retention is Hard
Finding a storage system that can meet the initial capture and playback performance demands is relatively easy. High performance disk based storage systems abound and flash assisted systems are becoming more affordable. Finding a storage system that can meet the needs of long term retention while still meeting the demand of random access in an acceptable time window is more problematic. Also many of these organizations may not have the data center floor space to meet the long term capacity demands of all this data.
Production Archive – The Long-Term Retention, Rapid Recovery Balance
The key for success is to deploy a high-speed solution where the capacity required is only enough to meet video data ingest and initial viewership peaks. Then have this data offloaded to a secondary tier of storage that can then automatically copy and or move the data to tape. Ideally, these solutions should be able to go a step further and copy or move this data as needed to the cloud.
Products like Fujifilm’s Dternity integrate disk, tape and cloud to create a “production archive” that solves the problems these new M&E companies are facing. The production archive appears on the network as a single mount point that video data can be copied to. Production archive can serve less frequently used data in real-time, provide tape archival protection and even provide cold cloud archive. The disk component can be relatively high performance disk that can be expanded at a very gradual rate thanks to the tape and cloud backend. As data ages and becomes even less popular, it is automatically moved to tape. But if it is accessed, it can be automatically recovered to the appropriate disk tier.
In the video data use case, initial raw footage can be copied to the disk area and copies can be instantly made to tape for disaster recovery. Then when production of the user viewable file is created, the raw footage can be moved to the disk archive, freeing up primary storage yet again. A policy can then be set that if the raw footage is not immediately used again for further editing, it is archived to tape only storage. As the initial view demand of the production stored content subsides it can be moved to the disk part of the production archive. Since access will be from fewer users, performance stays acceptable. Then as the production archive content becomes less frequently accessed, it can automatically be moved to tape for longer term archive, and then, eventually, out to the cloud for cold storage.
Video data is now mainstream. It can serve as both a product that is sold or as a source of information that can be analyzed. The ease at which video data can be captured and processed today is largely responsible for its popularity amongst users and businesses.
Organizations of all types and sizes have to contend with the unprecedented growth in this information. Meeting initial storage demands for this content is relatively easy; creating a cost effective long term strategy that still delivers adequate on-demand performance is challenging. Production archive solutions, like Fujifilm’s Dternity, are meeting this challenge by providing a solution that integrates disk, tape and cloud for a fully automated management of video assets.
FujiFilm is a client of Storage Switzerland