You are here: Home Page » /aps » /aps/storage » Don't panic: The broadcaster's guide to storage

Don't panic: The broadcaster's guide to storage

Feb 1, 2006 12:00 PM, By Craig Birkmaier

Some things are better left to the imagination. For example, I first encountered the parallel universe in which Douglas Adams existed in 1981 when I began working at the group now known as Grass Valley. While driving to an off-campus brainstorming session with my boss, we listened to the original BBC radio series “The Hitchhiker's Guide to the Galaxy,” recorded on the audio storage technology of the day — a compact cassette. I was hooked. (See “Web links” on page 18.)

The BBC tried to make a TV series from the radio scripts, but the video technology of the day failed to deliver on the incredible imagery that Adams created in our minds. Last year, the story finally made it to the big screen. Thanks to modern computer-generated imaging techniques, massive rendering farms connected to terabytes of hard disk drives and high-res digital compositing techniques, Zaphod Beeblebrox and the rest of Adam's bizarre cast of characters came to life on the big screen — but not in my head.

Adam's words stimulated my imagination in ways where computer-generated pixels failed. To be fair, I'm also a Tolkien junkie, and I thoroughly enjoyed Peter Jackson's interpretation of “Lord of the Rings.” Perhaps the difference is as simple as looking back with historical perspective versus forward into an improbable future.

According to “The Hitchhiker's Guide,” via Wikipedia, “Though often mistaken for a planet, Earth is in reality the greatest supercomputer of all time, designed by Deep Thought to discover the Great Question of Life, the Universe and Everything.” In Adam's universe, everything was part of the fabric of this supercomputer, including the most intelligent creatures on Earth — mice — and the third most intelligent — people.

The infinite improbability drive

If you had asked anyone in the world of broadcasting, circa 1984, what the world of digital television would look like a decade or two later, it is improbable that their imaginations could have come close to modern realities. The audio CD — the first digital audio format for consumers — was just reaching critical mass. The first professional digital video formats were still on the drawing board. Analog component video formats would continue to reign supreme for more than a decade. And the IBM PC XT came with 128KB of RAM and a 10MB hard drive.

A decade later, digital video compression had burst onto the scene and companies like Avid and Data Translations were making improbable claims that they would soon be able to edit online-quality video on a personal computer with off-the-shelf hard disk drives. At the Winter SMPTE Conference in San Francisco in 1995, I used my first nonlinear editing system to present a paper exploring the convergence of video and computing. The improbable had become reality as I played online-quality video on a 20ft screen from an NLE.

That system, which became the Media 100, used a Mac with off-the-shelf SCSI hard disks and a board that made it possible to input and output composite and component video and stereo audio. The built-in Ethernet network could move bits around at 10Mb/s.

Traditional broadcast equipment suppliers warned that these new computer-based tools were nothing more than toys and that the real-time world of broadcasting required dedicated hardware solutions to satisfy the performance and redundancy requirements faced in television stations and production facilities.

The upstarts responded, telling potential customers to fire the editor. Layers of middlemen and technical complexity were removed from the traditional linear world of television, as new computer-based tools set free the imaginations of a new generation of digital media content producers. These tools set the stage for the digital media workflows that are now beginning to change the way broadcasters create and manage the video and audio content that is their lifeblood.

The scalability of deep thought

For a brief period of time in the late '90s, many broadcasters imagined that the information technology revolution would hit a wall — the high-definition television wall. The sheer volume of bits would overwhelm off-the-shelf technologies.

They did not understand the fundamentals of Moore's Law, which impacts almost every aspect of information technology. Riding the Moore's Law curves for CPU performance, storage densities and network bandwidth, the improbable happened again.

The relentless pace of IT enables the HD revolution. The tools for SD video production have easily scaled up to meet the challenges of HDTV. In fact, an NLE system capable of working with uncompressed HD sources is now cheaper than the first compressed online-quality NLEs delivered in 1995.

In the early '90s, I had the opportunity to work with a team of people at Hewlett Packard who were motivated to do something new. Their instrumentation division was being mothballed, and they were given the opportunity to start a new business, if it could leverage other HP technologies. They decided to see what they could do with video.

The first product was a video capture device — a frame grabber front-end for HP printers. The second product, developed by a team led by Al Kovalick, was a video server using MPEG-2 compression. The division was eventually acquired by Pinnacle Systems, which was acquired last year by Avid. Kovalick is now the chief technology officer for Avid's West Coast operations.

Kovalick has been working on the bleeding edge of video server and media networking technologies for more than a decade. He has been involved in product development and broadcast standards development in organizations, including the SMPTE and EBU. You could say he has written the broadcaster's guide to AV and IT convergence.

Well, he kind of did, but he named it “Video systems in an IT environment.” To learn more about the issues confronted in the design of media workgroup environments that can stand up to the demands real-time broadcast operations, with full redundancy and monitoring tools, read the book. (See “Web links” on page 18.)

The workgroups of doubt

So here we are, two decades into the digital revolution, yet many broadcasters still have largely analog facilities, with a few digital islands, including standalone NLEs and play-out servers for commercials. Few have embraced the technologies that are needed to take full advantage of the new digital workflows. These workflows rely heavily on file-based media ingest and networking to allow workgroup collaboration, be it in the newsroom or the production department.

This may not be such a bad thing. Media workgroup solutions have been available since the late '90s, and the technology has been improving, while the cost has been following the Moore's Law curve. In his book, Kovalick explores the topologies for media workgroups that have been used in recent years and trends for the future.

Most workgroup solutions today are built around a centralized Fibre Channel switch to which arrays of disks are attached. Client machines typically use a Fibre Channel controller that is connected to the switch. A file manager keeps track of all of the files and manages permissions (who can read a file, who can write files, etc.). In many cases, a separate metadata server is also used to manage the metadata files that are now produced by many video acquisition and production systems. The client machines typically use an Ethernet connection to the metadata server.

This type of workgroup topology is know as a storage area network (SAN). In some larger installations, a network attached storage (NAS) controller is attached to the SAN to deliver media files to additional clients using Gigabit Ethernet. (See “Web links.”)

Avid recently introduced a new media workgroup solution called the Unity ISIS media network. (See Web links.) The system offers a glimpse into the future of intelligent storage networks, which will offer more capabilities and built-in redundancy, while reducing cost and complexity.

The heart of Avid's system is called a Data Blade. In essence, it is a NAS unit with two Gigabit Ethernet ports. Each Data Blade contains two Serial ATA (SATA) hard disks for 1TB of storage. The Data Blades are attached to Gigabit Ethernet switches, creating two redundant paths from any client to any Data Blade.

A system director keeps track of where all files are located, including metadata files. It also manages system loading and redundancy should a Data Blade or one of the Ethernet paths fail. File redundancy is accomplished using simple file mirroring techniques. Two copies of the file are stored on separate Data Blades, and the files are randomly distributed across all of the storage units. If a drive fails, the system controller can copy files that were on that drive to any available Data Blade. There is no need to rebuild the failed RAID array. This can take place even if the failed disk is not replaced and is typically faster than rebuilding the multiple-drive RAID volumes used in existing SANs.

Mostly harmless

Given the incredible pace of change over the past two decades, one might come to the conclusion that off-the-shelf IT solutions will eventually replace such products as Avid's Unity ISIS or that it is improbable that companies developing dedicated solutions for broadcasters will survive. Don't bet on it.

As the world of digital television evolves, there will be many new opportunities to exploit. For example, Avid is now working with Google and Yahoo as these Internet-spawned companies move into the video download business.

The real problem that must be addressed by broadcasters is what will their businesses look like in a decade — in two decades. It's time for those imaginations to start working overtime on the future.

Who knows, the improbable could happen again, and broadcasters could become a critical part of this supercomputer, otherwise known as the Earth. Until next month, so long and thanks for all the fish.

Craig Birkmaier is a technology consultant at Pcube Labs, and he hosts and moderates the OpenDTV forum.