Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: ABX Comparator version 2.0 (Read 51337 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

ABX Comparator version 2.0

Reply #25 –
I forgot to mention the two test songs I randomly selected and used weren't the same length of time, if it matters.

ABX Comparator version 2.0

Reply #26 –
Quote from: mzil on
In normal test mode, selecting the tick box "Keep playback position when changing track" and listening straight through without invoking a segment to repeat, there is an accidental visual "tell", a way to determine A from B visually, because when one transitions from Play A to Play X there is a slight stutter, visually, on the progressing blue bar graph below, when asking for such a change truly alternates the sources, whereas the same action which doesn't actually do any switching (since A already equals X) the motion is smooth, continuous, and uninterrupted. Looking for this stutter (or perhaps "tiny pause/backtrack"?) gives away if any actual switching occurred.

Sorry, cannot confirm, not even with two tracks having completely different lengths.
Can anyone else confirm?
Microsoft Windows: We can't script here, this is bat country.
 

ABX Comparator version 2.0

Reply #27 –
I don't see a stuttering.

But this brings me to a related problem.
Generate a 1000 Hz and 1000.1 Hz tone, or two 1000 Hz tones with different phase offsets. Now ABX the two files.

Even if the files sound identical, every time you switch from A to X and back you can either
- hear a small glitch (X=B)
- sound continues smoothly (X=A)

This is problematic, because you can easily ABX identically sounding files using this 'trick'. I can post verifiable logs if you don't believe me.
"I hear it when I see it."

ABX Comparator version 2.0

Reply #28 –
ABXing files with different sample rates will indeed produce a glitch on song switch, bug acknowledged.
Microsoft Windows: We can't script here, this is bat country.

ABX Comparator version 2.0

Reply #29 –
Oh yeah, same file sampled at different rates causes an even greater glitch on switch.

On switch: I guess you'd need to quickly fade out. Stop playback. Start playback with quick fade in. Regardless if files are switched or not.

I'd rather have consistent switching than seamless (in some cases) switching.
"I hear it when I see it."

ABX Comparator version 2.0

Reply #30 –
Sorry, my two songs might have had differing sampling rates too. I will test some more and get back later.

ABX Comparator version 2.0

Reply #31 –
Yes, my songs had differing sampling rates, 44/16 vs 96/24 I believe. Sorry to have not seen or mentioned that. I do not see the stutter using the same sampling rate.

An attempt to fix the stutter problem (with differing sampling rates) might be to just accept that it just can't really BE fixed and introduce the exact SAME stutter to the transition of both.
So in comparing 44 vz 96, say as an example, have the program undergo an unnecessary, extra rerouting of the signal to a new sampling rate EVERY time, even when it doesn't need to:

A is 44
B is 96

A to B transitioning is done as 44 to 88 to 96 [two total steps]
A to A transitioning is done as 44 to 88 to 44 [two total steps]
B to A transitioning is done as 96 to 88 to 44 [two total steps]
B to B transitioning is done as 96 to 88 to 96 [two total steps]

Sorry, I'm just a layman and don't know much about this stuff, but as I see it the glitch, "bar graph stutter", and delay should be the same for all four possible scenarios, doing it this way.




P.S. The bar graph stutter may be more easily seen with short songs? Mine was 17 seconds. [since small time increments are more easily seen this way due to the expanded linear scale across the horizon for such a short song.]

ABX Comparator version 2.0

Reply #32 –
Great to see this updated. Can I make a request?

I often use ABX to compare different masterings of the same CD album (sometimes digitised vinyl too). They are often indexed differently, so it would be great to have the option to delay/advance one track so they can be aligned.

I appreciate this is not a useful feature for comparing lossless and compressed versions of the same audio, which may be the primary focus of this tool.

ABX Comparator version 2.0

Reply #33 –
Quote from: mzil on
An attempt to fix the stutter problem (with differing sampling rates) might be to just accept that it just can't really BE fixed and introduce the exact SAME stutter to the transition of both.
So in comparing 44 vz 96, say as an example, have the program undergo an unnecessary, extra rerouting of the signal to a new sampling rate EVERY time, even when it doesn't need to:


I can reproduce the visual stutter with different sampling rates, short files. I guess it has the same cause as the audible glitch.

I wouldn't resample the files in the ABX component... The test files shouldn't be modified in the ABX process.
"I hear it when I see it."

ABX Comparator version 2.0

Reply #34 –
Beta 4 out.

Now behaves consistently when the files being tested have mismatching sample rate or channel layout.

Seamless mode active only when testing files with identical sample rates & channel layouts.
Microsoft Windows: We can't script here, this is bat country.

ABX Comparator version 2.0

Reply #35 –
Everything working well here. Thanks again.

A minor quibble: If you were to stop 20 people on the street and ask them, "If you took a True or False test and were told your final score had a "pval=1%", can you define for me what that terminology means exactly?" I estimate that less than 1 in 20 average Joes would get it right. Whereas if you were to ask the same question but replace "pval =1%" with a phrase such as was used before (and still is in the test mode), akin to "likelihood your score was achieved simply due to chance = 1%" about 19, if not all 20 people, would be able to understand and describe what was meant.

In the training mode the use of the term "pval" is understood by all of us, sure, but WE'RE experienced in these matters;  to make the test more friendly to first time users and/or those with no background in statistics, I think using the spelled out description, which does still occur in the formal test section, would help make the training section more accessible to the masses. Just a thought.

ABX Comparator version 2.0

Reply #36 –
Quote from: Peter on
Seamless mode active only when testing files with identical sample rates & channel layouts.


The problem I explained above still exists. Two 1 kHz tones with different phase offset => on an actual seamless switch (e.g. press A, X or A, Y a couple of times) you can hear a tiny glitch.

@mzil: How about "chance of guessing: 1%"?
"I hear it when I see it."

ABX Comparator version 2.0

Reply #37 –
Is a tiny switch glitch actually a problem? The problem can't be that I'm informed as to when a switch is happening, since I already do that myself.

Quote
@mzil: How about "chance of guessing: 1%"?


The current component already does that with "probability that you were guessing: X%" so it's probably good to just keep that phrase in.

ABX Comparator version 2.0

Reply #38 –
It is a problem because it allows cheating. The glitch is only heard when A changes to B so it reveals X.

ABX Comparator version 2.0

Reply #39 –
Let me quote this again:
Quote from: xnor on
Even if the files sound identical, every time you switch from A to X and back you can either
- hear a small glitch (X=B)
- sound continues smoothly (X=A)


The glitch gets smaller and harder to detect with music. It's easiest to detect with simple tones when they are out of phase (the files wouldn't null at all).

Try to copy any file, then add some samples of delay at the beginning to one, or filter one with an all-pass filter ...
In the ABX window you can then spam A, X or A, Y keys and you should hear that either the switch to X or to Y is not as smooth. If A, Y is smooth for example, then Y=A and X=B.

I guess the problem is due to some sort of crossfading. As the waveforms don't line up, they produce a small glitch at the crossing. Why not completely fade out one file, then start fade in the other one?
"I hear it when I see it."

ABX Comparator version 2.0

Reply #40 –
Quote from: Case on
It is a problem because it allows cheating. The glitch is only heard when A changes to B so it reveals X.
FYI this thread from 2010 is about switching artifacts and its problems.
Quote from: Peter on
To get 100% artifact free track switching, the application would need to employ some kind of low-latency playback scheme and switch streams being sent (with a short crossfade) by itself without interrupting the output.
This isn't very compatible with the way foobar2000 works and fixing it would make the ABX component much more sophisticated than it is now and probably also require a new foobar2000 core version.
Peter, I just learned about the new ABX beta. Did you manage to allow artifact free track switching ?

ABX Comparator version 2.0

Reply #41 –
To summarise what has been said in other threads:

Basic signal theory dictates that for this application, transitions, be they from one signal to another, or between one signal and silence (not playing anything), must be appropriately windowed.  Use a Hann window (half-cosine envelope); 50ms should suffice.

For example:

Rectangular window; ultrasonic signal becomes audible at transitions:



With 50ms Hann window, ultrasonic signal stays inaudible at transitions:




I've only tested the current Beta once so far, but it appears not to be windowing correctly when transitioning from signal to silence.

ABX Comparator version 2.0

Reply #42 –
Quote
The glitch is only heard when A changes to B so it reveals X.


Ahh, ok.

ABX Comparator version 2.0

Reply #43 –
Quote from: xnor on
I don't see a stuttering.

But this brings me to a related problem.
Generate a 1000 Hz and 1000.1 Hz tone, or two 1000 Hz tones with different phase offsets. Now ABX the two files.

Even if the files sound identical, every time you switch from A to X and back you can either
- hear a small glitch (X=B)
- sound continues smoothly (X=A)

This is problematic, because you can easily ABX identically sounding files using this 'trick'. I can post verifiable logs if you don't believe me.



This sort of thing becomes less problematical if the switch over is a cross fade or a non-overlapped fade out followed by a fade in,  which JJ has strongly recommended all along.

ABX Comparator version 2.0

Reply #44 –
Quote from: Peter on
This is a major rewrite of the foobar2000 ABX component, which had been unchanged for nearly 10 years now until now.

Current version: 2.0 beta 4

http://www.foobar2000.org/temp/ABX_Compara....fb2k-component

Highlights:

  • Removed functionality that should not be used during proper ABX tests.
    Trial count is configured in advance and not influenced by trial progress.
    Trial can no longer be reset while in progress.
  • Improved security for better identification of what was being ABXed.
    The logs now contain SHA1 hashes of files used.
    The logs now contain information about DSP and output settings used.
    The logs now contain a signature that can be verified online to ensure that the log has not been tampered with.
    Please keep in mind that there are no perfect security measures; in the end it's up to trusting individual users to have performed the test correctly or not. Someone with enough resources will always find a way to cheat, regardless of the amount of encryption & signature checking. We're only aiming to cover the most obvious cases.
  • New improved playback engine
    Seamless transition between ABX-ed tracks.
    Uses core output settings on foobar2000 v1.3.5 and newer. Falls back to default DirectSound when running in older foobar2000 versions.
  • Usability improvements
    Can use ReplayGain with tracks having no ReplayGain info as well as with untaggable file formats.
    Keyboard shortcuts for all essential buttons in the trial progress dialog.
  • Training mode
    Introduced in beta 2.
    Infinite trials (for two tracks), no log produced.
    Allows any number of tracks, but no guessing & trials for more than two.
    Trial mode now shows the current pval [beta 3 addition].


Have fun with it.


What I don't see that has been a real world issue:

For non-training tests the timing of the start and stop of the actual sample used to ABX is locked for the duration of the test and identified in the log file..

ABX Comparator version 2.0

Reply #45 –
Quote from: Arnold B. Krueger on
This sort of thing becomes less problematical if the switch over is a cross fade or a non-overlapped fade out followed by a fade in,  which JJ has strongly recommended all along.
  Using ANY overlap at all is just asking for trouble. I'm with jj. Think of how different the same file compared against itself would sound at the transition point simply because one file had an inverted polarity to the other one, for example.

  Regarding the wording, I don't really care what the exact words for "likelihood your score was simply due to chance alone = .5%" is, I just think people in training mode should see the exact same thing there (that they do after a real test), instead of the less layman friendly "pval = .5%".

Arny, all within the confines of one trial, in real test mode, one may listen to a 5 second segment looped from early in the song, but then a minute later switch to a 10 second segment from the end of the song, etc., so Foobar ABX would have to keep track and list ALL of them, every time the phrase repeat loop feature is activated, plus it won't understand when you are trying to isolate a particular telling section precisely, and keep moving the start and stop points around slightly by knudging them up and down the time scale slightly, that each attempt isn't really the loop you focused on. A final report would look like this:

"Trial 1 segment repeats used: 10.1 seconds to 15.3 seconds, 10.0 seconds to 15.3 seconds, 9.9 seconds to 15.3 seconds, 1m08.7 seconds to 1m19.2 seconds, 1m08.7 seconds to 1m20.0seconds, 1m08.7seconds to 1m19.7 seconds, 1m08.8seconds to 1m19.7 seconds, 1m08.9seconds to 1m19.7 seconds

Trial 2 segment repeats used..."

Is that what you're after?

ABX Comparator version 2.0

Reply #46 –
If this is going to work properly for every test, I think we need to be able to switch between two modes.

Mode 1 = raised cosine 5ms cross fades between samples and between PLAY and STOP.
Mode 2 = raised cosine 5ms cross fades to/from silence between samples and between PLAY and STOP.

i.e. mode 1 allows seamless overlap, while mode 2 never lets the audio overlap and intentionally introduces a silent gap at every switch.

For some tests 1 is better, and 2 is a nuisance.
For other tests, 1 is a problem, and 2 is the best you can do.

Obviously the log file should say which was used. You need to pick one in preferences, or at the start of the test. You can't switch mid-test.

FWIW it's still not failsafe. If you have a large DC offset in one file but not the other, I think even a 5ms cross fade will be audible (or "feelable" with headphone and "visible" with some speakers) so you would still know which is which. You make it perfect!

Cheers,
David.

ABX Comparator version 2.0

Reply #47 –
Basically agreeing with Arny & David above.  It is highly desirable that test procedures should repeatable by others (to see if the results are repeatable) so every operational parameter (including start and end times, fade type, fade length, etc.) must be recorded in the log.  Even if a parameter is not variable in a particular version of this component, recording it allows users of other tools to recreate the experiment.

5ms fade seems a bit on the short-side to me; yes, my suggestion of 50ms above may seem a bit on the long-side, but better safe than sorry?  JJ suggested 40ms a while ago (linked to by Kees above), btw.

Whilst correct fading will solve a lot a problems, one which is not well-solved by it is that of temporal masking: allowing the tester to start playback at any point could allow an inaudible sound to be become separated from its temporal masking sound (either pre or post, though pre is more likely—see https://ccrma.stanford.edu/~bosse/proj/node21.html).  I don't have any suggested solution for this other than to check for it on a case by case basis (and recording all operational parameters facilitates this).

ABX Comparator version 2.0

Reply #48 –
Logging segments used in the test doesn't seem very 'solid'.
Anyone could play the whole track (which is quite short usually) but listen only for a single artifact, e.g. at the beginning or end.
"I hear it when I see it."

ABX Comparator version 2.0

Reply #49 –
Quote from: mzil on
Quote from: Arnold B. Krueger on
This sort of thing becomes less problematical if the switch over is a cross fade or a non-overlapped fade out followed by a fade in,  which JJ has strongly recommended all along.
  Using ANY overlap at all is just asking for trouble. I'm with jj. Think of how different the same file compared against itself would sound at the transition point simply because one file had an inverted polarity to the other one, for example.

Regarding the wording, I don't really care what the exact words for "likelihood your score was simply due to chance alone = .5%" is, I just think people in training mode should see the exact same thing there (that they do after a real test), instead of the less layman friendly "pval = .5%".

Arny, all within the confines of one trial, in real test mode, one may listen to a 5 second segment looped from early in the song, but then a minute later switch to a 10 second segment from the end of the song, etc., so Foobar ABX would have to keep track and list ALL of them, every time the phrase repeat loop feature is activated, plus it won't understand when you are trying to isolate a particular telling section precisely, and keep moving the start and stop points around slightly by knudging them up and down the time scale slightly, that each attempt isn't really the loop you focused on. A final report would look like this:


I had no idea that people were rewriting and remixing songs in order to ABX them.  Frankly, I'm skeptical that this is really happening often enough to worry about.

My concern is that observers can determine what people actually ABXed by looking at the log.