Broadcast archives
Nov 1, 2009 12:00 PM, By David Austerberry
Can storage keep up with an ever-growing amount of produced content?
Figure 1. The archive controller directs data flow between disk and tape, and manages the tape library.
Select image to enlarge.
With the inception of file-based productions, a broadcaster's archive has become more of an issue than ever. When programs were supplied on videotape, they were kept for the duration of the rights window and then returned to the production company or removed from a transmittable area. After transmission, commissioned material was sent to a climate-controlled warehouse to sit on a shelf for perpetuity.
The management of the warehouse ranges from a card index of tape and shelf numbers to a comprehensive solution with bar codes and a database. Tapes are checked out if a series is to be aired again, or if the content is needed as archive material in a new program.
When the case is made for the investment in an archive for files, a number of questions are raised. What is an archive for? Is it a repository of assets to be mined in the future, or is it for disaster recovery? What should be archived, and what should be trashed? What file format should be used? And finally: What storage medium should be used — data tape, spinning disks or just outsourcing the storage?
The simple answer is that you should balance the cost of buying and running the storage against the value of the assets.
The archive
An archive can serve several functions. For the newsroom, it's an essential pool of material for creating background stories to explain current events. However, news ages very quickly, and a skilled news archivist will ensure that only what is essential is archived. Add to that, most news clips are short and do not require the same amount of storage as long-form programs. It makes sense to manage a news archive as a stand-alone system, separate from the program archive.
As program production moves to tapeless formats, a file archive will replace the videotape library as the broadcaster's main repository of program assets. For a small station, it can be a backup for the disk storage. For larger broadcasters commissioning their own programming, it is a permanent repository of their assets. For any broadcaster, it can form part of their disaster recovery (DR) strategy. Data tapes can be shipped out to a remote site, or for those with deep pockets, a second tape library can be installed at the DR location. The issue here is not technological; it's a business decision.
A typical archive consists of a large RAID array for nearline storage, backed by a tape library system. The disk subsystem stores work in progress — in post or waiting for transmission. The post and transmission departments can pull files from the nearline to high-performance storage for editing and playout.
The technology
A file-based archive is just part of a larger system and could be considered a service to the media asset management (MAM). (See Figure 1.) The file archive sits at the bottom of the storage hierarchy and provides the lowest cost-per-byte at the expense of performance (time to restore an asset). The archive management application sits between the media storage and the digital asset management (DAM or MAM).
Archive manager functions
In simple terms, the archive manager can be thought of as managing the data tape library. To get the optimum performance from a tape drive, a large managed buffer is required so data can be streamed at maximum right speed to the tape. Most libraries have multiple drives, and the archive manager can prioritize read/write operation to best serve broadcast operations. For example, a late schedule change may mean a file must be restored urgently for playout.
An index maps files to tapes in a directory. The manager can group content files to suit the operations. One group of tapes could be used for spots, one for series and one for movies. Content can be grouped to single tapes, so they can be removed from the robot to store on a shelf.
Beyond the basic tasks related to store and retrieval, the archive manager can perform background checks on the integrity of drives and data, and preemptively migrate content to current drive formats and fresh media.
DAM
DAM means that most production processes can use a low-resolution proxy of the broadcast asset. The proxy is stored on a regular RAID array using generic low-cost IT storage. Today's IT networks can easily handle the demands for proxy viewing with a properly designed switched infrastructure.
A typical process flow is shown in Figure 2 on page 40. Original content is ingested and stored in the archive in the highest resolution chosen by the broadcaster. From this, versions can be made for transmission. These may be edited and segmented. Legacy material may need processing — including scratch removal and noise reduction, deinterlacing, and color correction — to clean up the picture. This processing can be applied to a copy, leaving the preservation master untouched for improved image rescue techniques in the future. The processed copy becomes the transmission master.
The key components of any archive are two processes: data movement and transcoding. The data movers receive commands from the DAM and broadcast automation to copy files from tape to nearline, or move files from nearline to tape.
No single file format fits all applications. The archive must be the highest quality, but for editing, a format like Avid, DNxHD or Apple ProRes may be more suitable. For the playout servers, a lower bit rate, long-GOP format is more appropriate. Using the right format for the job maximizes visual quality while optimizing hardware costs. To service this need for different formats, the archive management must include transcoding. This may be integral to the archive management, or it may call on a transcoding service.
Any transcode will introduce artifacts, so the workflow should be designed so that transcoding upward in bit rate and resolution is avoided. Each videotape dub dropped a generation in quality, and 10 dubs from shooting to playout were not unreasonable. The number of transcode steps in a typical file workflow can be minimal compared with videotape, so quality stands to improve. (This assumes that too much compression is not used.)
| Want to use this article? Click here for options! |  |
