Quality control
Feb 1, 2010 12:00 PM, By Richard Duvall
Understanding PQR, DMOS and PSNR measurements
Figure 1. The video frame shown in Figure 1.1 has greater mean squared error with respect to the original reference video than the video frame in Figure 1.2. However, because the error in Figure 1.1 has a high spatial frequency and the error in Figure 1.2 has a low spatial frequency, the human eye perceives Figure 1.2 as lower quality.
Select figure to enlarge.
Consumer picture quality expectations are higher than ever, creating intense pressure on video equipment manufacturers, broadcasters, network operators and content providers to verify that their devices, systems or processes have not introduced impairments in video content that will affect perceived picture quality. This has led to the development of new automated picture quality instruments capable of evaluating picture quality across the video supply chain without the need for slow and expensive human evaluators, while also improving repeatability.
Historically, organizations have used an informal method of subjective picture quality assessment that relies on one person or a small group of people who demonstrate an ability to detect video quality impairments. These are the organization's “golden eyes.” Subjective picture quality evaluations are fraught with error and expense, and often end up only approximating viewer opinion.
These factors have led organizations to alternatives for subjective evaluation, such as the ITU-R BT.500 recommendation that describes several methods, along with requirements for selecting and configuring displays, determining reference and test video sequences, and selecting subjects for viewing audiences. Such subjective picture-quality assessments are expensive and time consuming.
Figure 2. Noise-based objective picture quality measurements compute the noise, or error, in the test video compared to a reference video.
Select figure to enlarge.
Instead, engineering, maintenance and quality assurance teams are starting to turn to a new class of instruments that use full-reference objective picture quality measurements. Using these instruments, teams can make accurate, reliable and repeatable picture quality measurements more rapidly and cost effectively than testing with actual viewers.
The tests used by instruments include Difference Mean Opinion Score (DMOS), Picture Quality Rating (PQR) and traditional Peak Signal-to-Noise Ratio (PSNR) measurements as a quick check for picture quality problems. This article explores key concepts associated with these measurements. It also provides tips and guidance for how teams can use these techniques in a variety of settings for their greatest advantage.
Subjective assessment and objective picture quality measurement
If people perceived all changes in video content equally, assessing picture quality would be much easier. A measurement instrument could simply compute the pixel-by-pixel differences between the original video content (the reference video) and the content derived from this reference video (the test video). It could then compute the Mean Squared Error (MSE) of these differences over each video frame and the entire video sequence.
Figure 3. Perceptual-based objective picture quality measurements use human vision system models to determine the perceptual contrast of reference and test videos.
Select figure to enlarge.
However, people are not mechanical measuring devices. Many factors affect viewers' ability to perceive differences between the reference and test video. Figure 1 illustrates this situation. The video frame shown in Figure 1.1 has greater MSE with respect to the original reference video than the video frame in Figure 1.2.
However, the error in Figure 1.1 has high spatial frequency, while the error in Figure 1.2 consists of blocks containing much lower spatial frequencies. The human vision system has a stronger response to the lower spatial frequencies in Figure 1.2 and less response at the higher spatial frequencies in Figure 1.1. Subjectively, Figure 1.2 is worse than Figure 1.1, even though the MSE measurement would assess Figure 1.1 as the poorer image.
Objective picture quality measurements that only measure the noise difference between the reference and test video sequences, e.g. PSNR, will not accurately and consistently match viewers' subjective ratings. To match subjective assessments, objective picture quality measurements need to account for human visual perception.
One of the two categories of full-reference objective picture quality measurements is shown in Figure 2. Noise-based measurements compute the noise, or error, in the test video compared to a reference video. This form of PSNR measurement is helpful in diagnosing defects in video processing hardware and software. Changes in PSNR values also give a general indication of changes in picture quality.
Figure 4. Shown here is a typical DMOS measurement result. Values in the 0-20 range indicate test video that viewers would rate as excellent to good relative to the reference video.
Alternative versions of the PSNR measurements adjust the base measurement result to account for perceptual factors and improve the match between the measurement results and subjective evaluations. Other noised-based picture quality measurements use different methods to determine noise and make perceptual adjustments.
A second category of full-reference objective picture quality measurements is illustrated in Figure 3. Perceptual-based measurements use human vision system models to determine the perceptual contrast of reference and test videos. Further processing accounts for several other perceptual characteristics. These include relationships between perceptual contrast and luminance and various masking behaviors in human vision. The measurement then computes the perceptual contrast difference between the reference and test videos rather than the noise difference. The perceptual contrast difference is used directly in making perceptual-based picture quality measurements. With an accurate human vision model, picture quality measurements based on perceptual contrast differences match viewers' subjective evaluations.
Picture quality rating measurements
Picture quality rating measurements convert the perceptual contrast difference between the reference and test videos to a value representing viewers' ability to notice these differences between the videos. Perceptual sensitivity experiments measure the viewer's ability to notice differences in terms of Just Noticeable Differences (JNDs).
| Want to use this article? Click here for options! |  |
