Skip to main content

Topic: ABX Comparator version 2.0 (Read 23511 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Peter
  • [*][*][*][*][*]
  • Administrator
ABX Comparator version 2.0
This is a major rewrite of the foobar2000 ABX component, which had been unchanged for nearly 10 years now until now.

Current version: 2.0

http://www.foobar2000.org/components/view/foo_abx

Highlights:

  • Removed functionality that should not be used during proper ABX tests.
    Trial count is configured in advance and not influenced by trial progress.
    Trial can no longer be reset while in progress.
  • Improved security for better identification of what was being ABXed.
    The logs now contain SHA1 hashes of files used.
    The logs now contain information about DSP and output settings used.
    The logs now contain a signature that can be verified online to ensure that the log has not been tampered with.
    Please keep in mind that there are no perfect security measures; in the end it's up to trusting individual users to have performed the test correctly or not. Someone with enough resources will always find a way to cheat, regardless of the amount of encryption & signature checking. We're only aiming to cover the most obvious cases.
  • New improved playback engine
    Faded transition between ABX-ed tracks. [ beta 7: crossfade is now opt-in, use responsibly ]
    Uses core output settings on foobar2000 v1.3.5 and newer. Falls back to default DirectSound when running in older foobar2000 versions.
  • Usability improvements
    Can use ReplayGain with tracks having no ReplayGain info as well as with untaggable file formats.
    Keyboard shortcuts for all essential buttons in the trial progress dialog.
    Added the ability to edit the current position timecode [beta 9].
  • Training mode
    Introduced in beta 2.
    Infinite trials (for two tracks), no log produced.
    Allows any number of tracks, but no guessing & trials for more than two.
    Trial mode now shows the current pval [beta 3 addition].


Have fun with it.
  • Last Edit: 09 January, 2015, 05:25:51 AM by Peter

  • carpman
  • [*][*][*][*][*]
  • Developer
ABX Comparator version 2.0
Reply #1
Seamless transitions and replay gain improvements much appreciated - all working well.
Thanks Peter.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

  • detmek
  • [*][*]
ABX Comparator version 2.0
Reply #2
Works here. So as online verifier. Thanks.

  • eahm
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #3
The logs now contain SHA1 hashes of files used.

Awesome, is it possible to show also the information of the file? What kind of codec and codec parameters are enough I guess (example: MP3, VBR 2).

Quote
The logs now contain information about DSP and output settings used.

Awesome!

Quote
The logs now contain a signature that can be verified online to ensure that the log has not been tampered with.

AWESOME!

Quote
Seamless transition between ABX-ed tracks.
Uses core output settings on foobar2000 v1.3.5 and newer. Falls back to default DirectSound when running in older foobar2000 versions.


Awesome, thank you!
  • Last Edit: 31 October, 2014, 02:16:43 PM by eahm

  • IgorC
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #4
Interesting.

I was trying it too.

I wonder why the number of trials is fixed at 16. I wasn't paying attention.

Quote
Trial count is configured in advance and not influenced by trial progress.

P.S. Thank You, xnor
  • Last Edit: 31 October, 2014, 02:54:16 PM by IgorC

  • xnor
  • [*][*][*][*][*]
  • Developer
ABX Comparator version 2.0
Reply #5
I wonder why the number of trials is fixed at 16.


In the dialog that opens before the ABX window opens ... there is an option to set the number of trials.
"I hear it when I see it."

  • klepetox
  • [*]
ABX Comparator version 2.0
Reply #6
Nice, but I use ABX Comparator to determine differences between various versions of mastering and I have to know what track is playing. In other words, I use ABX Comparator not for blind testing, but to hear differences between releases (i.e. differences between remastered and non remastered versions, then I really need to know what track is playing). Now it is impossible.

  • Peter
  • [*][*][*][*][*]
  • Administrator
ABX Comparator version 2.0
Reply #7
Nice, but I use ABX Comparator to determine differences between various versions of mastering and I have to know what track is playing. In other words, I use ABX Comparator not for blind testing, but to hear differences between releases (i.e. differences between remastered and non remastered versions, then I really need to know what track is playing). Now it is impossible.

What you're trying to do is not impossible but has just gotten less convenient than before.
I've been thinking about that scenario as well. I can add a separate "practice" mode where no formal ABX log is written but you can try any number of times.
  • Last Edit: 01 November, 2014, 03:27:31 AM by Peter

  • carpman
  • [*][*][*][*][*]
  • Developer
ABX Comparator version 2.0
Reply #8
I'd like that too. I sometimes use for the same purpose as klepetox.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

  • mzil
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #9
Me three. The old ABX was a fantastic sighted comparison tool and Arny and I would pitch that aspect to new people to get them into using it, as a first "baby step" towards full blown testing, so I definitely vote that some aspect akin to how it worked before, in that sighted way, should remain. [I guess by never voting you can continue to compare A to B knowing what they are from  what I discuss in post #12]
  • Last Edit: 01 November, 2014, 07:31:44 AM by mzil

  • mzil
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #10
New point. I notice when selecting a phrase repeat (loop) section the time location on the green bar graph above works differently and I was surprised that with some effort, pushing it to and fro with the mouse cursor, one can select a segment even SMALLER than .1 second, which I believe was the minimum allowed for in the earlier version. This allows isolation of a transient peak in order to detect things like time misalignment between the two files with greater precision, so I like that.
  • Last Edit: 01 November, 2014, 07:33:32 AM by mzil

  • mzil
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #11
Nice, but I use ABX Comparator to determine differences between various versions of mastering and I have to know what track is playing. In other words, I use ABX Comparator not for blind testing, but to hear differences between releases (i.e. differences between remastered and non remastered versions, then I really need to know what track is playing). Now it is impossible.


As it worked before the top in the list of your two selected songs from your foobar playlist was always then "A" during the test phase. I believe that is still the case.


  • detmek
  • [*][*]
ABX Comparator version 2.0
Reply #12
@mzil

I just tried and it is the same as before. First of selected tracks in the list is always A, second track is B.

P.S. I am using ABX comparator to test some DSP effects by converting a song with desired DSP effect, replaygain it and use ABX to compare effect by pressing just A and B, no X and Y. When the effect is too small then I use X/Y to test if I can actually hear the effect.
  • Last Edit: 01 November, 2014, 09:45:32 AM by detmek

ABX Comparator version 2.0
Reply #13
What output method does the new version use? As I see, changing it in playback settings has no influence on ABX. Strange behavior.

  • Case
  • [*][*][*][*][*]
  • Developer (Donating)
ABX Comparator version 2.0
Reply #14
It uses the output that was active when the ABX session is started. Except if you are on older foobar2000 version than 1.3.5, then it reverts to DirectSound.

ABX Comparator version 2.0
Reply #15
Nice, but I use ABX Comparator to determine differences between various versions of mastering and I have to know what track is playing. In other words, I use ABX Comparator not for blind testing, but to hear differences between releases (i.e. differences between remastered and non remastered versions, then I really need to know what track is playing). Now it is impossible.

What you're trying to do is not impossible but has just gotten less convenient than before.
I've been thinking about that scenario as well. I can add a separate "practice" mode where no formal ABX log is written but you can try any number of times.


I defend it too.

There should be training mode like in Java ABC/HR's ABX:





Case, thanks!
  • Last Edit: 02 November, 2014, 03:22:28 AM by Steve Forte Rio

ABX Comparator version 2.0
Reply #16
This is a major rewrite of the foobar2000 ABX component, which had been unchanged for nearly 10 years now until now.

Current version: 2.0 beta 1

http://www.foobar2000.org/temp/ABX_Compara....fb2k-component

Highlights:

  • Removed functionality that should not be used during proper ABX tests.
    Trial count is configured in advance and not influenced by trial progress.
    Trial can no longer be reset while in progress.
  • Improved security for better identification of what was being ABXed.
    The logs now contain SHA1 hashes of files used.
    The logs now contain information about DSP and output settings used.
    The logs now contain a signature that can be verified online to ensure that the log has not been tampered with.
    Please keep in mind that there are no perfect security measures; in the end it's up to trusting individual users to have performed the test correctly or not. Someone with enough resources will always find a way to cheat, regardless of the amount of encryption & signature checking. We're only aiming to cover the most obvious cases.
  • New improved playback engine
    Seamless transition between ABX-ed tracks.
    Uses core output settings on foobar2000 v1.3.5 and newer. Falls back to default DirectSound when running in older foobar2000 versions.
  • Usability improvements
    Can use ReplayGain with tracks having no ReplayGain info as well as with untaggable file formats.
    Keyboard shortcuts for all essential buttons in the trial progress dialog.


Thanks!

  • Peter
  • [*][*][*][*][*]
  • Administrator
ABX Comparator version 2.0
Reply #17
Beta 2 out.

Training mode has been added.

  • xnor
  • [*][*][*][*][*]
  • Developer
ABX Comparator version 2.0
Reply #18
Beta 2 out.

Training mode has been added.

Nice.

I think the only thing that is "missing" (compared to the old component) is the display of probability of guessing in the score during training mode.
  • Last Edit: 04 November, 2014, 12:00:18 PM by xnor
"I hear it when I see it."

  • xnor
  • [*][*][*][*][*]
  • Developer
ABX Comparator version 2.0
Reply #19
Also, I just did a training run and the same choice was correct 10 times in a row. I know that this may happen statistically, but did something change in the random number generator?
"I hear it when I see it."

  • mzil
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #20
Thanks Peter, It just keeps getting better!:)

A minor nit pick: I fully understand why one's cumulative, running status of "Likelihood results were due to chance alone expressed as a percent" have been removed from the formal test mode [and I agree that serves no purpose or even is arguably just flat out wrong, so it is good it was removed and I applaud that] but it still would be a very useful tool during the training mode before the test, for many of us.  If it could be reintroduced in that mode, at least as an option, that would be great.

Here's an example of how it would be useful to some of us who don't have binomial tables (or whatever those things are called, I forget) memorized:
"Interesting, I think I have discovered the difference and in this training mode I have successfully scored 6 out of 9 trials correctly. Good, but I wish I knew if that's really statistically significant or not, and by how much, or if its just a little, inconsequential bit better than chance, like was possible in the original ABX v.1".
  • Last Edit: 04 November, 2014, 03:23:05 PM by mzil

  • Peter
  • [*][*][*][*][*]
  • Administrator
ABX Comparator version 2.0
Reply #21
The random number generator has not changed. Scoring 10/10 without even listening is rare but not unheard of.

The probability value display for training mode will be brought back, maybe I can come up with a better label for it as the value is not valid unless the conditions of proper trial are satisfied (predetermined trial count, intermediate results hidden).
  • Last Edit: 04 November, 2014, 05:14:08 PM by Peter

  • mzil
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #22
I found the intermediate "likelihood due to chance, based on the number of trials you have taken so far, were we to end the practice test right here and now [not that you are allowed to, nor should be allowed to in a REAL test]"  results quite useful though, even if they are improper to display during a REAL test.

When one is in trial mode and using ABX as a fantastic learning tool, searching for possible subtle differences to focus on, it gives one immediate feedback as to if the current thing they are attempting to key off of is real, or rather just a figment of their imagination. It acts as sort of a robotic training coach who is constantly cheering you on when you have found a true audible difference, i.e. "hit pay dirt with statistical significance",  or kindly explains that what you are pursuing isn't actually real, so you are wasting your time and should move on to pursue something else.
  • Last Edit: 04 November, 2014, 06:11:41 PM by mzil

  • Peter
  • [*][*][*][*][*]
  • Administrator
ABX Comparator version 2.0
Reply #23
Beta 3 up, added pval display for training mode.

  • mzil
  • [*][*][*][*][*]
ABX Comparator version 2.0
Reply #24
Excellent! Thanks.

New issue though. I'm not sure if it existed before but I am seeing it just now. Hard to explain but here goes:

In normal test mode, selecting the tick box "Keep playback position when changing track" and listening straight through without invoking a segment to repeat, there is an accidental visual "tell", a way to determine A from B visually, because when one transitions from Play A to Play X there is a slight stutter, visually, on the progressing blue bar graph below, when asking for such a change truly alternates the sources, whereas the same action which doesn't actually do any switching (since A already equals X) the motion is smooth, continuous, and uninterrupted. Looking for this stutter (or perhaps "tiny pause/backtrack"?) gives away if any actual switching occurred.
  • Last Edit: 05 November, 2014, 06:02:54 PM by mzil