Validation test of NR metrics, J.noref

The International Telecommunications Union (ITU), which gets its charter from the United Nations (UN), is performing validation tests of no reference (NR) metrics that assess the quality of a video stream using only pixel values, resolution, and frame rate. Despite 30 years of research into NR video metrics, no such metric has been proven accurate enough for commercial deployment.

U.S. industry is very interested in this type of metric because it could enable automatic feedback loops in cameras and various video systems. The NR metric would allow the camera or video system to detect quality problems, understand why the quality is bad, and automatically fix the problem. The U.S. Department of Commerce’s NTIA/ITS is participating in this validation test of NR metrics, nicknamed J.noref. We need a set of footage to be used in an experiment that will contain 400+ short videos.

Those videos will show various responses of modern video cameras, to scenes and filming scenarios chosen to solicit different responses from the cameras. After the purchase is produced, NTIA will conduct an experiment where a panel of people will assess the quality of the videos. The NR metrics then try to estimate the quality of each video. The videos will be shared with other researchers around the world.

The videos will also be shared for research and development (R&D) purposes only (no commercial applications) on the Consumer Digital Video Library (CDVL, www.cdvl.org). This purchase is inspired by the 2015 study, Impact of Camera Pixel Count and Monitor Resolution Perceptual Image Quality (Conference Paper) - ITS, performed by NTIA/ITS, Intel, and the University of Ghent. That study compares the performance of 23 digital cameras, each used to photograph the same scene.

Figures 1, 2, and 3 show the types of scenes selected and the range of camera impairments that were produced. The full set of images can be viewed by creating a free account on CDVL, logging in, and then searching for “CCRIQ”. The goal is to compare the quality response of different cameras in responses to major variables (e.g., scene complexity, camera motion, in-view motion, lighting, and focal distance).