Case Study – Major League Baseball’s 24/7 Cable TV Network Pushes IT to Extremes
Major League Baseball (MLB) has been around since 1869. The first televised game was in 1939 and 70 years later, MLB Network was launched on January 1, 2009. MLB Network’s goal is to provide the most comprehensive coverage of America’s national pastime all the time. Now in its seventh year on the air, it’s apparent that advancing the preservation of the game is a significant IT challenge. Storage Switzerland recently sat down with MLB Network’s Director – Media Management and Post Production, Tab Butler to see how they tackle it.
MLB Network’s Engineering & IT team must not only capture all data coming in from live games during the season, it must also try to preserve the audio and video assets that make up the game’s rich history. Since there was a 70-year gap between the first televised game and the launch of MLB Network, it also meant a lot of catching up.
In 2012, Butler was faced with tackling this challenge head on. The original system designed to support the network’s vision was showing signs of its original 2008 implementation. Its equipment rooms were fully utilized and its power and cooling were fully subscribed. Its post-production environment was ending manufacturer support and it was unable to scale to meet the demands of HD video and increased camera angles.
Butler and his team set a new vision for MLB Network’s post-production systems. The goal was to move the core systems to a new data center operated by CoreSite, as it solved both power and space limitations in the existing facility. The plan was to move record systems, storage systems, edit workstations, proxy storage systems and their DIAMOND application servers to the new facility.
The custom-built DIAMOND MAM application provides the Production Asset Management layer within the Adobe Premiere Pro CC application, enabling access to over 550,000 hours of searchable video content within the MLB archives. For example, you could query for every Derek Jeter home run, the system would queue up a series of video for you to edit into a Derek Jeter home run montage. MLB Network chose Simplivity to host the Diamond architecture, quite a compliment for the hyper-converged newcomer. MLB Network uses Simplivity because of its VM centricity and unified management.
The heavy lifting in the environment is done by Cisco UCS. MLB Network uses 50 Cisco UCS C-240 servers as HD edit workstations. They also use 26 Cisco UCS B-200 M-3 blades for its virtualization environment, and for encoding, transcoding, other video processing, as well as the Grass Valley STRATUS system. Forty Grass Valley Summit encoders, under STRATUS control are responsible for all the initial 136 streams of ingest of live video as well as incorporating other video workflows, writing to two NetApp SAN storage pools. Tegile provides the storage sub-system infrastructure that supports the USC blades and servers. Tegile was chosen because it provided very rapid shared boot performance, and for its ability to clone server images.
The HD video SANs utilize NetApp storage powered by Quantum’s StorNext with a total of 2.88PB of raw capacity, which translates into the ability to store over 90,000 hours of baseball. A third storage pool, backed by an AEL-500 tape library, captures the proxy video of all recordings. As data ages, it is moved to the tape sub-system to provide both off-site archive and on-site access to the proxy data. This allows the MLB Network IT team to keep data accessible to production, but keep costs down.
The backend of this application infrastructure is supported by tape as well. Adobe applications do not support tape so Butler and his team selected Crossroads Systems and their StrongBox product for Premiere Pro project archiving. This solution allows a tape library to be interacted with, like it were a disk system. More importantly it outputs tape in a format called LTFS that is readable by dozens of other solutions.
The team decided to keep studio playback, archive library systems, editing manpower, 50 HD edit workspaces and 250 proxy edit workstations in their main facility. The two facilities were almost 2 miles apart. To make the connection between the two facilities MLB Network counted on 2 redundant fiber paths, with 192 dark fibers per path. They ran a 320Gbps IP backbone between the facilities using a Cisco Nexus network infrastructure. Each rack supports 10Gbps Ethernet server connections, with a 40Gbps backbone between racks, and 8Gbps fibre channel. These connections allowed for plenty of performance as well as room for expansion.
Its all About the Show
All of the above infrastructure is designed to facilitate the creation of all shows that MLB Network airs. Once a show is created, it is stored as a complete project out of the Adobe suite of solutions. These projects are then sent to Crossroad’s StrongBox, which makes a copy of the project elements on both disk and multiple tapes. One set of tapes are stored off-site and over time the disk copy is removed. But StrongBox allows the rapid recall of any of these show elements as well as the entire show project, seamlessly directly from the Adobe Premiere Pro application.
Tape Comes Up Big
Tape still plays a big role at MLB Network. During the season, the network receives over 30TB of HD content per day. Storing all this data on disk long term is cost prohibitive. In 2008, MLB Network deployed an LTO-4 tape systems to solve its long term HD video storage needs. The Oracle SL8500 library, using the DIVArchive software application from Front Porch Digital (now owned by Oracle) was initially deployed. The DIVArchive software is the archive management layer for storing single video files. Each game has over seven video feeds, and many recordings of the MLB Network-created content are also recorded. The DIVArchive software provides Partial File Restore access to these very large video files. It manages data from disk cache and tape, while creating two copies of each video file, for on-site daily retrieval and off-site disaster recovery. Today, the SL8500 supports over 37,000 LTO 4 tape cartridges at 800GB each, close to 28PB of information, which grows daily.
Where things get more interesting is how MLB Network interfaces tape when they are trying to track a complex edited project, highlight package, or entire show. As described in the example above, when more than one asset is involved, an alternative tape solution is deployed. For the creation and editing of shows, features, highlights, and flashbacks, MLB Network uses Adobe Premiere Pro. Adobe Premiere Pro expects all assets to be on disk and has no understanding of off-line media like tape. Premiere Pro needs an online folder structure presented to it from the operating system to archive projects, but with over 50 edit rooms creating content, their complete projects could not be archived on disk. The amount of content is too great to allow MLB Network to keep all the content online.
The solution for MLB Network was to turn to Crossroads Systems’ Strongbox solution. As we discussed in our recent product analysis “StrongBox Archive NAS solves problem of Long-term Unstructured Data Storage”, StrongBox creates a layer of abstraction between the tape library and the application. It provides file system (NFS/CIFS) access so that a disk only application, like Adobe Premiere Pro, can actually have a tape backend. The Premiere Pro product now has what it thinks is online access to PBs of information because StrongBox presents a folder hierarchy to the application. Thanks to the ease of access, MLB Network also uses the StrongBox solution as a utility for IT.
The other big advantage of the Crossroads’ solution is that the tapes it creates are written in the LTFS format, which means that these tapes can be read by any of the dozens of applications that can read an LTFS format. For MLBN it means they are not locked into any one single solution, which is critical. As Butler says, “LTFS is ideal for Disaster Recovery.”
It Takes a Village
The mixture of server, application and storage vendors is not a coincidence. MLB Network’s Engineering & IT group is one of the best examples of an organization that has selected the best of breed for each specific use case. It allows them to meet the demands of a 7x24x365 baseball television network while keeping costs down. MLB Network chose to ignore the “tape is dead” mantra and leverage it as a key component of a very modern infrastructure, using “old” technology for a modern infrastructure, driven by this same best of breed philosophy.
George Crump is the Chief Marketing Officer at VergeIO, the leader in Ultraconverged Infrastructure. Prior to VergeIO he was Chief Product Strategist at StorONE. Before assuming roles with innovative technology vendors, George spent almost 14 years as the founder and lead analyst at Storage Switzerland. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Before founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators, where he was in charge of technology testing, integration, and product selection.
StorageSwiss.com - The Home of Storage Switzerland 