You are here: Home Page » Storage & Networking » Active storage

Active storage

Jul 1, 2007 12:00 PM, BY PAUL TURNER

Figure 2. Typical grid processing management components
Click image to enlarge.

Integrated transcoding of material

It is extremely common for a facility to have media in different file formats at various stages of the workflow. The house ingest format, for example, is hardly ever used to archive material. For this reason, broadcasters are forced to transcode material as it makes its way through the production process, resulting in significant numbers of network transfers as the material is sent to the transcoding engine and the result is transferred back to the central storage.

Imagine a world where the central storage system automatically transcodes media based on business rules configured by the user and delivers it to the next stage of the process once a business metric, such as an editorial approval, has been met. The transcoding process lends itself to grid-based processing and is accelerated in direct proportion to the number of CPUs that are targeted at the transcoding.

Transoding technology also automatically generates proxies, based on user-supplied business rules. For example, high-res material is proxied to the house proxy format as soon as it is placed in the requisite location on the central grid storage system.

Content providers often need to provide a finished product in a variety of formats for the myriad of delivery and display systems. Once again, having this format conversion happen in the grid storage prior to handoff to the delivery subsystem is a great benefit.

A secondary benefit in the transcoding case comes from the inherent distributed nature of grid processing. At the moment, several transcoding manufacturers offer the ability to have multiple transcode engines available for use in a transcode farm. An individual job is given to an individual transcode engine. Therefore, if you have five engines free at any point in time, only one will be given a particular job. In the case of grid processing, however, all five could be pressed into duty to transcode the source clip, resulting in significant reduction of transcode time compared with the current method. (See Figure 3.)

Moving forward, it may also be possible to section up a transcode job so it is given to the content servers that physically contain the source slices. In this case, the individual content servers would only transcode the slices they contain, resulting in a reduction in network bandwidth, as there would be no need to transport the source slices to a specific content server for translation. These potential savings offer a real improvement in the transcoding workflow.

File verification and technical QC

Broadcasters can expend significant effort in technical QC of material after ingest and in file verification after transfer from one location to another. Such a labor-intensive activity can effectively be performed via software applications, which can check files for such things as GOP errors, macroblocking, audio levels and gamut. These applications can even detect and flag long periods of silence in the middle of an individual clip, signifying a potential error. These activities, which once required intervention by an operator, can easily be integrated into a grid storage and processing platform.

Figure 3. An example of a traditional transcode farm vs. a grid processing approach

There is no proposal that QC for content and censorship can be performed within the central storage, though one can envision that some of this can be automated. Human judgment still needs to be applied to these activities.

Having central storage actively participate in operations ultimately leads to active workflows in which material is moved from storage location to storage location throughout the entire facility based on the rules engine detailed above. Material can be recorded into an ingest server and automatically moved, while it is still being stored, to the central server. Next it can be instantly checked for technical quality. Then, via a proxy automatically generated after the full-res material passed technical QC, it is passed on to the QC operator responsible for artistic content. After this stage, the material can be automatically rewrapped for editing and packaging. And after approval, it is automatically transcoded into the needed delivery formats.

While some of the above is forward looking, there are transcoding and technical QC products that can already hand jobs off to an active storage device for processing. Certainly, grid storage systems exist with the CPU horsepower to operate as active storage devices.

Conclusion

With the advent of grid-based storage systems, the idea of storage as an active participant of the end-to-end workflow becomes possible. When storage ceases to be merely a passive agent in the workflow, several tangible improvements can be made that offer the potential to substantively improve the efficiency of operations.

Active storage and active workflows will undoubtedly become more important as further improvements in storage components and distributed software applications become available. This approach offers significant improvements in bottom line performance. When the storage performs many of the processing tasks on stored media, there are savings in time, network bandwidth and cost.

The great news is that the foundations for this change are all already available from manufacturers, and integration of these functions into central storage systems is already underway. Storage is about to get much more interesting!

Paul Turner is vice president of marketing for Omneon.