Several different metrics were identified. The first group were metrics that describe the relationship between transport losses and the impact of those losses on video/audio payloads. The second group were metrics that characterized compression used. The third group helped to characterize the content.
It is important to understand why different classes were needed. In the first case, the impact of a bit error on the observed video differs greatly depending upon whether the loss affects an MPEG-compressed intra/inter predicted macroblock versus the loss of a B Frame, which might have a much less observable effect. In the second case, it is important to know the quantization and video bit rate to understand the potential impact of errors on video and audio. In the third case, we know content that is high motion is much more stressful on compression systems. Identifying this content is important when studying the impact of transmission errors on a system that is already stressed to the limit by high motion content.
One thing to remember is that the QoE metrics described in this article were designed to be used for analyzing professional video transported over private IP networks; this is a far different application from transporting consumer video over the generic Internet. The group also focused on the use of private networks to transmit video over cable systems to the home. The group determined that, except in extreme cases such as breaking news, professional video is almost always transmitted over private IP networks, so this restraint seemed reasonable. The group also looked at a particular protocol stack, basically consisting of either MPEG-2 or H.264 compressed video transported over UDP and IP.
Another important consideration that was used in the development of these metrics is that many of these metrics were designed to be measured over time — meaning that the metrics may have benefits when monitored instantaneously, but that they may also convey important trend information when they are sampled repeatedly and results are compared. The point is that trend information may be used to identify problems before they significantly impact video/audio transmission.
A sample of the metrics defined in the VSF report is shown in Table 1. Note that this is not an exhaustive list of metrics contained in the report.
A closer look
There is not enough space to cover all of these metrics in detail, but to take frame loss length as an example, the report defines loss event as the maximum number of consecutive slices/frames lost in a single loss event in a measurement interval. The report goes on to state that the measurement is important because IP losses typically happen in bursts. Together with the number of loss events, loss length is a major factor in determining the severity of a bursty loss. The longer the loss, the more it will affect the quality of decoded video.
Let’s look at another metric — motion activity. Motion activity is the average length of motion vectors in a measurement interval. Put another way, the amount of motion in a video determines its encoding complexity to a large degree. Video with a lot of motion is generally more difficult to encode than content with low motion.
Motion activity can help in estimating the base quality of the video stream as long as network losses are not present since, generally speaking, the higher the motion, the more difficult the content is to encode, and the more observable any compression artifacts are. The presence of network losses only makes a bad situation worse, and a high loss rate combined with a high motion activity metric is predictive of a lower QoE.
I should note that much of the material for this article came from the VSF report, and is the work of Pierre Costa, Stephan Winkler and other members of the Activity Group.
QoE is a topic that should be studied by broadcast engineers. While QoS is important, we should remember that perception matters, and that gets to the heart of QoE. These metrics try to quantify the impact of measurable impairments on the perceived quality of video and audio delivered to the end viewer
—Brad Gilmer is executive director of the Video Services Forum, executive director of the Advanced Media Workflow Association and president of Gilmer & Associates.