Broadcast archives need new technology
Oct 1, 2010 12:00 PM, By David Austerberry
A warehouse of shelving no longer meets the needs of file-based operations.
Archiving was once a real estate issue, as with these Digital Betacam tapes at itfc’s archive in London. With files, it became a technology issue.
Until recently, the provision of a content archive was given little thought by engineers and management. Any videotape that was likely to be needed in the future was sent off in a van to a warehouse, to be stored until it was needed. The warehouse required climate control, security and maybe a small dubbing facility to run off copies of tapes when recalled.
The software to manage the archive was little different from that in a public library: bar code readers, a database of tape IDs and shelf positions, and a check-in/check-out facility.
The broadcast archive is changing, and the driver is the migration away from videotape. Now that videotape is reaching end of life as a storage technology, the archive becomes a repository for files, rather than the physical asset that a videotape represents. A file no longer has the intimate association with the recording medium as with a master videotape.
Storing files raises a whole number of new issues. It now takes serious technology planning if the archive is to last as long as tape (typically 25 to 30 years) or even longer. Data storage typically has a short life, less than five years for a disk drive.
By the time videotape starts to deteriorate, whoever sent it to the archive may well have retired; it will be someone else's problem. Files must be constantly migrated to the latest storage technology, so a technology solution to preserve the archive must be put in place today, not in 25 years time.
Other issues to consider include network bandwidth from production to the archive, storage architecture and disaster recovery planning. And there is the cost of the ongoing operation of the archive.
IT archives
I was watching a documentary about the “Domesday Book,” a record of title in England during the 11th century. One-thousand years later we can still read the books, written in ink on sheepskin parchment. There are much older written documents, such as Egyptian tablets or the Dead Sea Scrolls. But how much of our recent video material will last that long?
Many of the IT developments in data archiving are designed to maintain documents for regulatory compliance, which in most cases is less than 10 years. Broadcasters regularly air series from 40 years ago, or even older, so commodity IT systems are not necessarily going to meet the needs of long life. Broadcasters' requirements are more akin to the collections sector: museums, archives and libraries.
Programs can have value for reruns, and some have a more intangible value as historical records. The value of a news archive is the depth it can add to stories, again intangible as an asset. The balance of the value as an asset versus the cost of maintaining an archive is perhaps the biggest problem in designing an archive, in proving the ROI.
What is an archive?
An archive can be many things. For a production company, it could be somewhere to store the files from the temporary storage used for acquisition now that many camcorders use SSDs and flash storage. For a news department, the archive represents news history and will need to be maintained for decades. A sports department will have similar requirements. For a transmission department, the archive is a low-cost repository for programs for the duration of the rights window. For a program commissioner, the archive stores the program as an asset well into the future, as long as it can generate revenues.
All these archives have different requirements, both technical and cost. It is a matter of judgement as to the value of an asset, as is the long-term cost of running an archive for decades. Deciding what to keep and what to delete is not an exact science.
Which codec?
This is not a simple choice for HD and higher resolutions, although for SD, MPEG-2, I-frame at 50Mb/s is popular.
The decision rests on several factors. First, has the file been ingested from videotape, or is it an original file? Is 8-bit resolution sufficient or does it need to be stored at 10 or more bits of resolution? Should it be encoded in a standard such as MPEG (2 or 4) or JPEG 2000, or should it use something more edit-friendly such as Avid DNxHD (SMPTE VC-3) or Apple ProRes? If you need more than 8 bits of resolution, that narrows the choice, excluding MPEG-2 for example. For many codecs, there is a choice of long GOP or I-frame only.
AVC-I is emerging as a choice for HD; it supports 10-bit coding and can be edited without the need to decode GOPs.
The choice of codec is a compromise. More efficient codecs use less storage space for a given quality but may need more powerful workstations for editing. As computer performance increases, the latter becomes less of an issue. The decision made today may not be the optimum choice next year.
A major goal is to minimize the number of transcodes from acquisition to delivery, which best maintains video quality. That may dictate a proprietary format for the archive. For most broadcast applications, audio can be left uncompressed.
Another factor is the video and audio wrapper or container. MXF is the obvious choice here, but which operational pattern? OP1a makes sense for most applications, but AAF may be best if an entire project is to be stored with the intention of later extensive repurposing. Many production systems maintain metadata in a separate database. For a long-term archive, self-describing files are an advantage.
The optimum codec will depend on the application — news, recent drama, videotape transfers, and HD or SD — so most broadcasters' archives will contain a mix of codecs.
Technology
Current technology is still based on electromagnetism, whether disk or tape. The magnetic information has a long life, although the supporting systems suffer from obsolescence. The formats of tape drives and disk controllers have a short life of only four or five years, so long-term archiving needs ongoing migration of data to the latest storage formats.
Optical systems have failed to provide the storage density of tape, although they, like solid-state storage, have applications in production. However, in 10 or 20 years time, a new technology could supplant or replace magnetic storage.
There is one technology that is low-risk but expensive: 35mm film separation negatives. The images are stored as red, green and blue silver negatives. These do not fade like the dyes used in color negatives, and they can potentially last centuries in the correct storage conditions. Also, they do not rely on computer technology, which is notorious for its rapid obsolescence. Who can even start to envisage what the computer of 20 years time will be like, let alone in a century from now?
Continue on next page
| Want to use this article? Click here for options! |  |
