The challenges of storing video
Aug 1, 2008 12:00 PM, By Arun Taneja
An examination of video storage platforms
The largest Fortune 1000 companies have grown their storage infrastructures to hundreds of terabytes, with the largest companies having multiple petabytes. In less than a year, Web 2.0 Internet start-ups and other companies storing rich media content have consumed more storage than the larger companies have accumulated over their lifetimes.
Social networking, application hosting, auctions, photo sharing and video distribution all consume more storage than older transactional type applications. The difference is that many Web 2.0 applications tend to be participatory and collaborative applications, where the users are the ones generating much of the content. The change from provider-generated content to user-generated content makes predicting storage growth far more challenging than in the past.
In addition to consuming vast amounts of storage, these new applications often have different I/O profiles compared to transactional applications, which tend to be primarily small-block, random I/O-oriented. Web 2.0 applications tend to store larger, more variable digital content. Access patterns can also vary between read-intensive (video distribution) to almost exclusively write-intensive (remotely hosted backup) and anything in between. The storage architectures that support high-performance transactional applications are not always a good fit for new video applications.
As for scalability, it's not that traditional monolithic arrays can't scale up to petabyte-plus configurations; it's the efficiency and cost (capital and operational) required to reach these levels. Not everything scales gracefully when 10-to-20-year-old storage architectures strain to accommodate the scalability requirements of these new applications. And, management paradigms that worked fine with tens of terabytes don't always scale well when managing multiple petabytes or more.
Video storage requirements
Like storage in general, there is no one-size-fits-all device that meets the requirements of every video application. However, video applications do share five common storage requirements: cost, performance, operational simplicity, modular growth and reliability. Let's look more closely at each.
Cost
Due to the large amounts of storage required by video, the cost per terabyte needs to be far lower than price points of traditional SAN and NAS storage devices used with transactional data applications. While everyone wants cheaper storage, many video applications would not be financially feasible without inexpensive storage. For most video applications, the cost of storage must be in the range of $1000 to $2000 per terabyte, with prices declining at 30 percent to 40 percent per year.
Although within the video storage market the term inexpensive is still relative, professional applications such as news and post-production editing, VOD, and TV station on-air playout may be able to afford more expensive storage than social networking and video-surveillance applications.
Just as the purchase cost can jeopardize the market viability of a video application, so too can excessive operational management costs. For instance, the cost to manage a terabyte of storage is often overlooked in calculating cost of ownership.
The four-year total cost of ownership of a video server can often be four to five times the initial purchase cost. It doesn't matter if the storage is being managed by video engineers or a station's IT department; it still needs to be as simple as possible to manage — or costs go up. While new storage technologies can simplify and automate the configuration and ongoing operation, the cost of ongoing management should be included in cost projections.
Performance
Video production and playout applications place different demands on storage than do other tasks. Traditional data-center applications consist of many random, small-block reads and writes. Video applications typically consist of large-block, sequential streaming I/O, with unique I/O profiles. For example, many video capture applications are 99 percent writes, while video sharing applications are 99 percent reads. Just because a server can support a large bank doesn't mean that same server can handle video well.
Storage has traditionally been optimized to meet the needs of transaction processing. Caching was used to help optimize the server's performance. However, video requires large block, sequential reads and writes, which can quickly overwhelm a cache, rendering traditional storage architecture ineffective for use with video.
Video applications benefit from architectures that distribute data across multiple disks and servers to increase the degree of parallelism, i.e., bandwidth. This approach not only provides a more cost-effective implementation, but also it allows video-optimized storage to meet the performance and throughput requirements that might not be available from more traditional storage architectures.
Operational simplicity
Historically, video storage was supplied as part of a media solution. The needed storage was embedded with the applications, such as editing or master control playout. Because these applications typically live outside the domain of IT management, stations often required additional support from their vendors. Due to the sheer volume of data, companies cannot afford the operational overhead of complex management paradigms.
Also, because video engineers, who were often less familiar with IT, were in charge of maintenance, the on-site management of the storage had to be far simpler than other facility storage. This has forced some stations to begin consolidating management of storage technology within their IT departments.
Modular growth
Video consumes large amounts of storage. And, the pace of growth is not always predictable. This is especially true of applications with user-generated content. Even traditional video applications like editing and graphics can experience unexpected growth due to expansion in the number of data sources, the move to HD or a desire to store content in multiple formats to support multiple playout channels.
One approach is to purchase a large monolithic storage array up-front, which will accommodate both planned and unplanned growth. The problem with this strategy is that monolithic arrays typically cost significantly more than the identical capacity in a modular array. In addition, this approach puts the acquisition cost up-front when the ROI or viability of the business plan may be unproven.
Modular storage provides a pay-as-you-grow storage model. In addition, modular arrays, with clustered or grid architectures, offer several advantages as a video storage platform when compared with traditional dual-controller modular arrays.
With traditional dual-controller arrays, additional capacity can be added up to a point. However, when bandwidth or controller processing power reaches its limit, the only solution is a forklift upgrade, or another modular array must be added. This requires each additional array to be managed as a separate entity.
Modular storage arrays with clustered architectures allow capacity, bandwidth and processing power to be scaled in modular units. Storage virtualization software eliminates the operational complexity of having to manage multiple independent arrays by creating a single virtual array that leverages the combined resources of the cluster to provide more linear performance and scalability. The array's virtualization software takes care of automatically distributing the data and processing across new modules as they are added. Clustered storage architectures provide pay-as-you-grow scalability while providing the simplicity of managing a single entity.
| Want to use this article? Click here for options! |  |
