Skip to main content

Topic: ABX Test for Video (Read 6046 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • prawns
  • [*][*][*]
ABX Test for Video
The ABX test used for audio in this forum has proven itself very useful in comparing the quality of various audio codecs. What with all the talk of the HTML5 video element of late there has been a lot comparisons made between the quality of H.264 vs. Theora. Regardless of who in fact is king, some of the comparisons struck me as being fairly flakey. It usually consisted of "Hey look I encoded this file using x and y codec. Look how great x is against y".

So is there any software available that provides a similar ABX test that foobar2000 provides but for video?

(Full disclosure: I'm a trying to explore options for a final year project and I thought this might be a fine exercise.)

  • KikeG
  • [*][*][*][*][*]
  • Developer
ABX Test for Video
Reply #1
fileABX  may be useful for you. Using its scripting options in combination with Avisynth scripts (StackHorizontal for example) you can perform blind ABX "side-to-side" comparisons of videos.

fileABX isn't as automated as typical audio ABX comparators. It's a Windows console app that generates an ABX test consisting of randomized "blind" copies of a couple of files to compare.

  • Notat
  • [*][*][*][*][*]
ABX Test for Video
Reply #2
ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.

Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.

ABX Test for Video
Reply #3
The ABX test used for audio in this forum has proven itself very useful in comparing the quality of various audio codecs. What with all the talk of the HTML5 video element of late there has been a lot comparisons made between the quality of H.264 vs. Theora. Regardless of who in fact is king, some of the comparisons struck me as being fairly flakey. It usually consisted of "Hey look I encoded this file using x and y codec. Look how great x is against y".

So is there any software available that provides a similar ABX test that foobar2000 provides but for video?

(Full disclosure: I'm a trying to explore options for a final year project and I thought this might be a fine exercise.)



Seems like a great appliation for hardware-based ABX.

  • knutinh
  • [*][*][*][*][*]
ABX Test for Video
Reply #4
ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.

Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.

I think it is fair to say that the ultimate goal of video codecs is highest possible subjective score, i.e. reduced viewer irritation, or ultimately subjective transparancy. The means to get there may be debatable.

I agree with the thread starter that public statements about codecs in HTML5 seems amateur-like compared to e.g. audio codec listening tests. At the very least, they should use a set of test-sequences (preferreably standardized), include some metadata (like bitrate, resolution,PSNR), and of course a subjective test where bias is sought removed.

-k
  • Last Edit: 07 April, 2010, 09:10:08 AM by knutinh

  • prawns
  • [*][*][*]
ABX Test for Video
Reply #5
fileABX isn't as automated as typical audio ABX comparators. It's a Windows console app that generates an ABX test consisting of randomized "blind" copies of a couple of files to compare.
That's great. Thank you for showing me!

ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.
Of course. Ideally, an ABX test for video would have an uncompressed video source vs some codec.

Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.
Maybe if the developers have access to better tools (like perhaps an ABX test for video  ) then this situation may improve.

Seems like a great appliation for hardware-based ABX.
Sorry. I don't understand.

I agree with the thread starter that public statements about codecs in HTML5 seems amateur-like compared to e.g. audio codec listening tests. At the very least, they should use a set of test-sequences (preferreably standardized), include some metadata (like bitrate, resolution,PSNR), and of course a subjective test where bias is sought removed.
Completely agree. If a more objective approach to comparison can be achieved then we'll only improve the quality of all codecs available. A developer is more likely believe results (and therefore try and improve the algorithm) if those results are not simply opinion.

Thank you very much to all of you that replied. Sorry for my late response.

To further continue this topic,
I think it is fair to say that the ultimate goal of video codecs is highest possible subjective score, i.e. reduced viewer irritation, or ultimately subjective transparancy. The means to get there may be debatable.
Currently, I would say, many applications do not have the space and/or bandwidth available to them to justify a transparent encode so a "as-good-as-possible" encode becomes the deciding factor. Perhaps a test for comparing video codecs should run in a slightly different manner. Instead of it being a test of whether the lossy encode is indistinguishable from the original it should be a test of which lossy encode is most faithful to the original.

I propose a test where the original is always visible to the test subject. The test subject can then switch between two lossy encodings of the original, A and B. The test subject should then decide which is the better lossy encoding and proceed to repeat the test to verify the difference.



What do you think?

ABX Test for Video
Reply #6
I propose a test where the original is always visible to the test subject. The test subject can then switch between two lossy encodings of the original, A and B. The test subject should then decide which is the better lossy encoding and proceed to repeat the test to verify the difference.



What do you think?


This looks more like ABC/hr, except the reference isn't hidden.  Nothing wrong with that, only its not ABX.

One difference between listening tests and tests involving vision is that we can sensibly look at 2 things at the same time, but not so much listen to two things at the same time.

  • ExUser
  • [*][*][*][*][*]
  • Read-only
ABX Test for Video
Reply #7
Another issue with blind-testing video is that our perception of audio is bandlimited in ways that video cannot be. You can get really close to your screen and be able to make out high-frequency details more easily than you would from a far position. Transparency, then, is kind of a meaningless distinction in video, because you can discern individual sample points in video in ways that are simply not possible with audio. Furthermore, when you pause an audio track, there's no audio. When you pause a video track, there's a still-frame image which you can analyze.

I'm not sure there's a video codec out there that's visually transparent.
  • Last Edit: 28 April, 2010, 10:52:15 AM by Canar

  • Notat
  • [*][*][*][*][*]
ABX Test for Video
Reply #8
ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.
Of course. Ideally, an ABX test for video would have an uncompressed video source vs some codec.

Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.
Maybe if the developers have access to better tools (like perhaps an ABX test for video  ) then this situation may improve.


Another issue with blind-testing video is that our perception of audio is bandlimited in ways that video cannot be. You can get really close to your screen and be able to make out high-frequency details more easily than you would from a far position. Transparency, then, is kind of a meaningless distinction in video, because you can discern individual sample points in video in ways that are simply not possible with audio. Furthermore, when you pause an audio track, there's no audio. When you pause a video track, there's a still-frame image which you can analyze.

I'm not sure there's a video codec out there that's visually transparent.

I've not yet seen a demo where anyone has tried to convince me that their codec is transparent. The pitch is invariably focused on other aspects (e.g. bit rate, bandwidth, latency, cost effectiveness). Of course, they'll boast about how good the picture looks, and yes, it looks good, but anyone who knows what to look for can tell you that it is not the same as the original.

  • kyis
  • [*]
ABX Test for Video
Reply #9
There is always the easy way, I often use that in scripts to automate conservative jpg optimization but it works just as well for video, for example:

I have a Lagarith source that I want to compare to the same sequence encoded using lossless x264 (or whatever).

I load both in after effects (nuke doesn't like AVC), put them on a composition, set the top sequence to the difference blend mode, then I create another comp., put the previous comp. in there and add an invert effect followed by a level effect.

If I didn't touch the effects settings the result would be almost perfectly white (or black if you prefer to skip the invert effect) which means no visible difference.
But then I crank the levels to match the histogram borders (0 to 16 in that case) and like magic the differences appears, I say difference but it's mostly due to lousy color space conversion somewhere along the line (Lagarith defaults to RGB (but supports many color spaces, including YUV), h264 is YUV only if I'm not wrong), so I got a slight regular shift in the blue channel mostly, less in the red and even less in the green, completely invisible to the eye (I mean come on...a maximum difference of 16 in RGB value...).
No visible artifact even after going so far in looking for them, so I say h264 lossless is really lossless...just watch out for the color space thingy.

Hope this helps. ^^
  • Last Edit: 28 May, 2010, 07:38:10 PM by kyis