Efficient digital distribution of media
Jul 1, 2010 3:06 PM, By Brian Stevenson and Mike Nann
The basic bottleneck on public networks is the transfer protocol.
Today’s media landscape is characterized by a rapidly expanding volume of content; new opportunities and accompanying demand for wider distribution and syndication; and broadening, increasingly diverse distribution outlets. These trends bring with them the need for easier and more efficient mechanisms to deliver media between content providers, contributors, aggregators, affiliates and distribution partners.
Traditionally, the delivery of content from studios and post facilities to broadcast networks and affiliates relied on physical media and transportation, shipping videotapes or hard drives by courier. Expensive, time-consuming and labor-intensive, physical distribution is no longer efficient and has gradually been giving way to electronic distribution, although physical delivery has not yet been completely replaced. Fully digital distribution via terrestrial IP-based networks or satellite offers increased automation, greater immediacy, higher security and tremendous cost savings over tape-and-truck transport. For the benefits to be fully realized, however, there are performance impediments that must be overcome, and technical and workflow inefficiencies that can be further improved.
Figure 1. TCP data is transmitted until the receive buffer is full.
The breadth of distribution far exceeds the past scenario of broadcasters and affiliates. Distribution partners may also span multiple platforms and channels, including VOD providers, electronic sell-through, mobile providers and digital cinema — each of which may have specific formatting requirements. Many of these recipients do not have the dedicated network lines or satellite downlink access that traditional broadcast affiliates have had and may only be reachable over public networks. Overcoming inherent performance limitations in these networks is an essential aspect of increasing distribution efficiency, but just one of multiple factors.
In any individual transfer, speed is often limited by the receiver’s bandwidth, leaving much of the sender’s bandwidth unused. Optimizing unicast distribution concurrency can maximize efficient bandwidth usage, while multicasting where possible and intermediate points shared by multiple recipients can minimize the amount of data being sent across costly network links. Meanwhile, recipients’ varying requirements for file conformance, compression, container and metadata formats must be addressed in an efficient and extensible manner, ideally minimizing the delivery of multiple variants while avoiding the need for multiple tools at the receiving edge. Upon receipt, workflow processes including discretionary QC, review and approval must be addressed homogeneously and efficiently as part of the distribution system.
In this two-part article, we will take a layered approach to exploring these challenges and solutions. We will first briefly look at how low-level network transport considerations affect individual transfers. In the second part, which will be published in the August issue, we will consider optimization of overall distribution architectures to leverage concurrency in a number of scenarios. Finally, we will look at the media being moved over these architectures and conforming it to the receivers’ requirements.
The basic bottleneck — the transport protocol
No distribution system can operate at optimal efficiency without maximizing the performance of its underlying transport mechanisms. Much like even the fastest courier vehicles are limited in speed when forced to drive on dirt roads in heavy traffic, the distribution of media files over IP-based networks is limited by inherent performance impediments in the underlying communications protocols. These impediments can limit transfers to a fraction of their potential speed and reliability even on private networks, and they can all but cripple them over the public Internet, which is necessary for reaching many recipients in today’s converged workflows. The inefficiencies increase exponentially with high bandwidths and long distances — significant when delivering media across the country or between countries and continents. Multicast-based transmission will be touched on in the next article when we discuss concurrency, but to understand the fundamental challenges of Transmission Control Protocol (TCP), we will first look at unicast TCP transfers between two points.
Figure 2. An acknowledgment is sent to the transmitter when the receive buffer is full.
The root of these performance limitations is the nature of TCP, the protocol for IP networks upon which transfer mechanisms such as FTP are based. With TCP, transmitted data packets must be received in the correct order. To achieve this, data is sent sequentially up to the size of the recipient’s receive window (buffer). (See Figure 1.) When the receive window is full, the receiver sends an acknowledgement back to the sender. (See Figure 2.) This acknowledgement must arrive at the sender before more data can be transmitted.
This round-trip time, the time between sending the data and the sender receiving acknowledgement of its receipt, is referred to as latency, and it can dramatically limit transfer speeds irrespective of available bandwidth. While local area networks and intra-area links (such as those within a major city) typically have latencies of less than 10ms, latency is particularly problematic over long distances. Transmission over the public Internet between the West and East Coasts of the United States may have latency between 80ms and 100ms. Links between continents typically have latency in excess of 120ms; a tested connection between sites in Toronto and China during the writing of this paper reported more than 280ms. There are many factors influencing this latency — the number of network “hops” between the sender and receiver, the characteristics of each hop, configuration of the routers along the transfer path, the sender and receiver’s effective proximity to high-speed backbones, the sender and receiver’s particular connectivity, etc. The practical result is the same: The latency between any two transfer points is a significant factor in the overall throughput of TCP-based transfers such as FTP.
| Want to use this article? Click here for options! |  |
