I easily cheated ABX in Foobar. Can this be avoided?

Topic: I easily cheated ABX in Foobar. Can this be avoided? (Read 975 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

I easily cheated ABX in Foobar. Can this be avoided?

2020-06-19 12:41:27

I've been doing many ABX tests in recent years and always praise the validity of the tool, but today I tried to see if I could cheat the ABX tool in Foobar, and I easily did (16/16) and then verified my log.
What I did was just make a copy of a file (so I had two identical files) and then change the volume on the new copy, then do the ABX test, then change the volume back on the copy, so the two files were identical. Afterwards I verified the log.

So, I obviously don't want to cheat the ABX tool, but an ABX hater could do this and claim he passed.
Could this option of cheating by changing the volume level be avoided?

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #1 – 2020-06-19 13:28:05

Quote from: board on 2020-06-19 12:41:27

then change the volume back on the copy, so the two files were identical

sha1 checksum of files was also identical?

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #2 – 2020-06-19 13:37:04

client side checks can be bypassed anyway, someone just needs to extract the secret used for "signing" the logs

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #3 – 2020-06-19 17:04:08

Quote from: Rollin on 2020-06-19 13:28:05

Quote from: board on 2020-06-19 12:41:27
then change the volume back on the copy, so the two files were identical
sha1 checksum of files was also identical?

When I use https://www.foobar2000.org/abx/signaturecheck I just get this message:
"Signature matches; the log appears to be valid."

Maybe I should rephrase my question a bit, or at least add another question: Let's say that I receive the log of a person who could have changed the volume first and then changed it back afterwards, would there be a way to make sure that he hasn't done this?

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #4 – 2020-06-19 17:04:59

Quote from: magicgoose on 2020-06-19 13:37:04

client side checks can be bypassed anyway, someone just needs to extract the secret used for "signing" the logs

I don't understand what you mean by this.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #5 – 2020-06-19 18:25:48

Quote from: board on 2020-06-19 12:41:27

What I did was just make a copy of a file (so I had two identical files) and then change the volume on the new copy, then do the ABX test, then change the volume back on the copy, so the two files were identical. Afterwards I verified the log.

I'm not sure to understand.

STEP 1:
You made an identical copy of a file. So you have file A and file B, absolutely identical and impossible to ABX in theory.

STEP 2:
You take your file B and you change it (you increase or decrease the volume—but you can also replace it with something totally different). B now becomes C.

STEP 3:
You ABXed without issue A vs C

STEP 4:
C is now replaced/reverted again to B

STEP 5:
You verify the LOG of A vs C, and it says it's correct.

If it's right, then I don't see any issue there. To prevent this, LOG file must contain checksum of both files. Then you should be able to see that someone is not testing A vs B but A vs something else. And I'm sure that checksum would be different if you change the volume.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #6 – 2020-06-19 18:43:13

Seems like anyone could "cheat" by simply modifying the log (including checksums) to suit their needs, but that's not really the value of ABX is it? It's a tool by which we can each make individual decisions on audio quality and tradeoffs. If not used honestly, then it has no real value. I'd never make a bet with someone decided only by their ability to provide me a "passed" log.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #7 – 2020-06-19 19:07:49

Quote from: Apesbrain on 2020-06-19 18:43:13

Seems like anyone could "cheat" by simply modifying the log (including checksums) …

I don't think so. There's a final signature at the end of the log to check if the log was modified by hand. It makes falsification and fraud much harder. I believe this feature very interesting in some case (imagine someone claming that CD vs 24/96 is so easy to ABX and sending log files on this board: without checksum & signature the person is OK with TOS#8; without checksum and signature on log file the claims remains dubious).

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #8 – 2020-06-19 20:33:03

Quote from: board on 2020-06-19 17:04:08

When I use https://www.foobar2000.org/abx/signaturecheck I just get this message:
"Signature matches; the log appears to be valid."

I mean sha1 of your final files (yes, files, not audio data) VS sha1 in log
Like this

Code: [Select]

File A: 06. Decay==0.flac
SHA1: 0672c87dcd975930caadb0b16f4d9912e66cd8ba
File B: 06. Decay----.opus
SHA1: 332df42a040e04fe809d7022c536b60981aca972

Because, if result of ABX is suspicious, we can ask person to share files too.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #9 – 2020-06-19 20:58:27

Quote from: Rollin on 2020-06-19 20:33:03

Quote from: board on 2020-06-19 17:04:08
When I use https://www.foobar2000.org/abx/signaturecheck I just get this message:
"Signature matches; the log appears to be valid."
I mean sha1 of your final files (yes, files, not audio data) VS sha1 in log
Like this
Code: [Select]
File A: 06. Decay==0.flac
SHA1: 0672c87dcd975930caadb0b16f4d9912e66cd8ba
File B: 06. Decay----.opus
SHA1: 332df42a040e04fe809d7022c536b60981aca972
Because, if result of ABX is suspicious, we can ask person to share files too.

Below is my log. I named the two files differently when I lowered the volume, and one is called "volume lowered" in Danish (I'm from Denmark).
So, I first lowered the volume, then did the ABX test and saved the log, and afterwards I then raised the volume again.
As for sharing files I've thought about that too, but unless the SHA1 code shows the file location, time of last edit, etc., asking the tester to send the files is essentially worthless, since the person can just copy the original, unedited, file to a different folder and then send you that one, saying he used that one for the ABX test.
I actually don't know what the SHA1 code means, so if there's something I'm missing or misunderstanding, feel free to enlighten me, as this was after all why I started the topic :-).

foo_abx 2.0.4 report
foobar2000 v1.3.17
2020-06-19 10:51:20

File A: 04. A Lady Of A Certain Age - EQ (kurve gemt som richness (only bass), pleasant treble 5) - volumen sænket.wav
SHA1: 0bc0df0846ebf5b04b46b611e3bda145eedfd720
File B: 04. A Lady Of A Certain Age - EQ (kurve gemt som richness (only bass), pleasant treble 5).wav
SHA1: c66845df41d75e6731985eb98c07103d45ea70bd

Output:
DS : Højttalere (ODAC-revB USB DAC)
Crossfading: YES

10:51:20 : Test started.
10:52:14 : 01/01
10:52:19 : 02/02
10:52:24 : 03/03
10:52:28 : 04/04
10:52:38 : 05/05
10:52:44 : 06/06
10:52:49 : 07/07
10:52:55 : 08/08
10:53:03 : 09/09
10:53:11 : 10/10
10:53:17 : 11/11
10:53:28 : 12/12
10:53:42 : 13/13
10:53:49 : 14/14
10:54:01 : 15/15
10:54:08 : 16/16
10:54:08 : Test finished.

----------
Total: 16/16
Probability that you were guessing: 0.0%

-- signature --
60b2a25c4d741b19b9e0024846a72d2c0716d65e

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #10 – 2020-06-19 21:18:51

Original file and edited file will have different sha1 checksum. And we can calculate this checksum ouside of fb2k.
As guruboolez said already.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #11 – 2020-06-20 08:23:03

Quote from: Rollin on 2020-06-19 21:18:51

Original file and edited file will have different sha1 checksum. And we can calculate this checksum ouside of fb2k.
As guruboolez said already.

I don't understand what this means. Can you explain please?
How do we calculate it?
Will the log I provided be able to show that I edited the file (volume or anything else) prior to ABX'ing it? Will it show the location and last editing date?
So, let's use the example of hi-res vs. CD quality:
If I send a hi-res file and the same song downsampled to CD quality and then upsampled again to hi-res, will we able to detect if the receiver has edited the file, or has simply placed the file in a different folder and then edited it, thereby cheating?
I now ABX'ed two completely different songs, but where I used replay-gain on one of the songs, and it wasn't mentioned in the log:

foo_abx 2.0.4 report
foobar2000 v1.3.17
2020-06-20 09:29:18

File A: 05 - Girl I Want You.flac
SHA1: 203c7894f2d62eebd0112caf5aa9bbac7be67412
File B: 06 - Pushin' Too Hard.flac
SHA1: 712fbc18757520f120cbf63d39be79a2ecd470e6

Output:
DS : Højttalere (ODAC-revB USB DAC)
Crossfading: YES

09:29:18 : Test started.
09:29:23 : 01/01
09:29:26 : 02/02
09:29:28 : 03/03
09:29:32 : 04/04
09:29:35 : 05/05
09:29:38 : 06/06
09:29:42 : 07/07
09:29:45 : 08/08
09:29:49 : 09/09
09:29:53 : 10/10
09:29:57 : 11/11
09:30:00 : 12/12
09:30:03 : 13/13
09:30:06 : 14/14
09:30:10 : 15/15
09:30:13 : 16/16
09:30:13 : Test finished.

----------
Total: 16/16
Probability that you were guessing: 0.0%

-- signature --
e0d3219f43df3e0e6fb89f5546f0da8b8a13c771

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #12 – 2020-06-20 09:01:08

Quote from: board on 2020-06-20 08:23:03

Quote from: Rollin on 2020-06-19 21:18:51
Original file and edited file will have different sha1 checksum. And we can calculate this checksum ouside of fb2k.
As guruboolez said already.
I don't understand what this means. Can you explain please?
How do we calculate it?

It's a cryptographic checksum computed from the data in the file. When a file is altered the checksum changes.
There are several programs available to calculate the checkum. You can use built-in feature in PowerShell to test it. Open PowerShell and run command "Get-FileHash -Algorithm sha1 <path to file>" and it will show the calculated checksum.

Quote from: board on 2020-06-20 08:23:03

Will the log I provided be able to show that I edited the file (volume or anything else) prior to ABX'ing it? Will it show the location and last editing date?

It shows the checksum is different so yes, it did show the files are different. Location and last editing date are irrelevant. If you share your fake files it will be immediately obvious that their checksums don't match what is in the logs.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #13 – 2020-06-20 09:14:29

Quote from: board on 2020-06-19 17:04:59

Quote from: magicgoose on 2020-06-19 13:37:04
client side checks can be bypassed anyway, someone just needs to extract the secret used for "signing" the logs
I don't understand what you mean by this.

step 1: write the log as you please
step 2: calculate the valid "signature" and append it - this gives you the log with any content you'd like to show, passing the validation

in order to do step 2, you need to reverse engineer the code of the ABX component to reveal the secret it uses to calculate the valid "signature".

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #14 – 2020-06-20 13:47:48

Quote from: magicgoose on 2020-06-20 09:14:29

in order to do step 2, you need to reverse engineer the code of the ABX component to reveal the secret it uses to calculate the valid "signature".

Ah, this is something I did not understand. The log "signature" is not simply an SHA-1 of the log file, it is a hash uniquely calculated by the ABX component. Thanks.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #15 – 2020-06-21 14:09:26

Quote from: Case on 2020-06-20 09:01:08

Quote from: board on 2020-06-20 08:23:03
Quote from: Rollin on 2020-06-19 21:18:51
Original file and edited file will have different sha1 checksum. And we can calculate this checksum ouside of fb2k.
As guruboolez said already.
I don't understand what this means. Can you explain please?
How do we calculate it?
It's a cryptographic checksum computed from the data in the file. When a file is altered the checksum changes.
There are several programs available to calculate the checkum. You can use built-in feature in PowerShell to test it. Open PowerShell and run command "Get-FileHash -Algorithm sha1 <path to file>" and it will show the calculated checksum.

Quote from: board on 2020-06-20 08:23:03
Will the log I provided be able to show that I edited the file (volume or anything else) prior to ABX'ing it? Will it show the location and last editing date?
It shows the checksum is different so yes, it did show the files are different. Location and last editing date are irrelevant. If you share your fake files it will be immediately obvious that their checksums don't match what is in the logs.

Thanks for your help. I'm not a computer wizard, so I still don't completely understand all this. Sorry.
So, let's stick to the example from earlier:
Let's say that I send someone two files that he should ABX: One real hi-res file and the same file downsampled to 16/44.1 and then upsampled again to hi-res. Will those two files have different or identical checksums?
Let's say that the receiver of these two files saves them in C:\1, but then he copies one of the files to C:\2. He then changes the volume level on the file in C:\2, then ABX'es this edited file against one of the files from C:\1 and passes it, then sends me the log of the passed ABX test, as well as the two unedited files from C:\1.
Would I then be able to see that he did this?

Also, what if he just saves the two files in one location, changes the volume on one file, does the ABX test, then changes the volume back. Would the checksum show this? I think that's what you said, but I just wanted to make sure :-).
The checksum on the two files should be different, right? The only reason why two checksums would be identical would be if you had ABX'ed two identical files, right? I tried that now, and the checksums were indeed identical.

I experimented a bit here:
I first took a true 24/96 file, downsampled it to 16/44.1, then upsampled it again to 24/96, did an ABX test (just guessing, as the result wasn't important).
These are the checksums:
File A: BMS-A.wav
SHA1: 6dc5bd69ebabefd7dd141530c9d85e3b38602f81
File B: BMS-A - 96-24 (upsampled).wav
SHA1: 68b60cce369b5b28cd8e76b930434f54d8c41841

So, the checksums are different, since there was a difference in the files (no content above 22 kHz and below -96 dB in file B).

Then I lowered the volume on the file I upsampled, took an ABX text (this time passing with 16 out of 16 correct). I got these checksums:
File A: BMS-A.wav
SHA1: 6dc5bd69ebabefd7dd141530c9d85e3b38602f81
File B: BMS-A - 96-24 (upsampled) - Kopi.wav
SHA1: 1cd2e3c6ac9cef7268d1de1bb793ced642cd273a

So, file A's checksum is identical to the one above, whereas file B's checksum is different.

After that ABX text, I then raised the volume back to what it was before and did another ABX test. Here are the checksums:
File A: BMS-A.wav
SHA1: 6dc5bd69ebabefd7dd141530c9d85e3b38602f81
File B: BMS-A - 96-24 (upsampled) - Kopi.wav
SHA1: 2b7dc8c843999a2eb3d36daeeeb8b6901dd1079d

So, the checksum of file B is different compared to the one mentioned before, although the file content is now the same yet again.

I then took the file where I had changed the volume, then made a copy, then lowered the volume on the copy, saved it, and immediately raised the volume back to what it was before and saved it again. Then I did another ABX test. These are the checksums:

File A: BMS-A - 96-24 (upsampled) - Kopi.wav
SHA1: 2b7dc8c843999a2eb3d36daeeeb8b6901dd1079d
File B: BMS-A - 96-24 (upsampled) - Kopi - 2.wav
SHA1: e2b3c80682658f1b2d650c808c8d63999d56416d

This is also a different checksum compared to the one mentioned just before.

So, I think maybe I'm answering my own questions here:
It seems that if I edit the file in ANY way, the checksums will show it. Is that correct?

So, I have two very important questions:
1: If I send two files to someone to ABX with whatever means I have (e-mail, cloud, Wetransfer, etc.), and he does an ABX test without altering the files in any way, will his checksums in his ABX log then be the same as in an ABX log for the same files on my computer?
2: Is there ANY way you can cheat the ABX test other than what I have suggested here?

Again, thanks for all your help, and sorry for the length of this message :-).

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #16 – 2020-06-21 15:48:30

Quote from: board on 2020-06-21 14:09:26

Let's say that I send someone two files that he should ABX: One real hi-res file and the same file downsampled to 16/44.1 and then upsampled again to hi-res. Will those two files have different or identical checksums?

Different

Quote from: board on 2020-06-21 14:09:26

Let's say that the receiver of these two files saves them in C:\1, but then he copies one of the files to C:\2. He then changes the volume level on the file in C:\2, then ABX'es this edited file against one of the files from C:\1 and passes it, then sends me the log of the passed ABX test, as well as the two unedited files from C:\1.
Would I then be able to see that he did this?

Yes. You can use any method to check the SHA1 checksum of the files to verify that they match what's in the logs. For example do your own ABX run and let foobar2000 show the checksums.

Quote from: board on 2020-06-21 14:09:26

Also, what if he just saves the two files in one location, changes the volume on one file, does the ABX test, then changes the volume back. Would the checksum show this? I think that's what you said, but I just wanted to make sure :-).
The checksum on the two files should be different, right? The only reason why two checksums would be identical would be if you had ABX'ed two identical files, right? I tried that now, and the checksums were indeed identical.

All correct.

Quote from: board on 2020-06-21 14:09:26

So, the checksum of file B is different compared to the one mentioned before, although the file content is now the same yet again.

File content is not the same after volume adjustments. You lose the content of the lowest bits and the adjustment introduces rounding errors.

Quote from: board on 2020-06-21 14:09:26

So, I think maybe I'm answering my own questions here:
It seems that if I edit the file in ANY way, the checksums will show it. Is that correct?

Yes. If the file content changes in any way the checksum shows it.

Quote from: board on 2020-06-21 14:09:26

So, I have two very important questions:
1: If I send two files to someone to ABX with whatever means I have (e-mail, cloud, Wetransfer, etc.), and he does an ABX test without altering the files in any way, will his checksums in his ABX log then be the same as in an ABX log for the same files on my computer?

Yes.

Quote from: board on 2020-06-21 14:09:26

2: Is there ANY way you can cheat the ABX test other than what I have suggested here?

Yes. Cheating is always possible. You have to consider how reliable the reporter is.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #17 – 2020-06-22 07:17:44

Thank you very much for your responses, Case :-).
So now I only have one question:
How would the person be able to cheat? Would he have to be a computer wiz like magicgoose wrote?

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #18 – 2020-06-22 10:15:32

One doesn't need to be a hacker to be able to cheat. Since you can't control their playback settings the tester can for example use EQ to take away all audible frequencies and the playback equipment will then alias ultrasounds to audible range. Or they can use external spectrum analyzer to see what is playing.

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #19 – 2020-06-22 14:00:15

Which is why ABX testing needs to be reproducible, it's not enough to post only the results, the samples used need to be included. Could a package approach be developed, both test results and samples included? Is there any way to include Foobar's DSP state?

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #20 – 2020-06-22 16:36:32

Samples are normally already included. This won't do anything against other possible cheats, already mentioned in this thread
1) system-wide DSP (like Equalizer APO) or speakers modified to exhibit extremely unconventional frequency response or distortion (also one can just record the output and analyze it)
2) extracting the log signing secret from the ABX component

Re: I easily cheated ABX in Foobar. Can this be avoided?

Reply #21 – 2020-06-23 15:22:24

Pardon my ignorance, but I'm not sure I get the context here, what's a one off test result indicate anyways? If someone wants to cheat the test, is that any different than a statistical anomaly? I'm not concerned with a person's, or a system's, ability, I'm interested in the results of public testing, aren't I? A single test is indicative of something, I agree, but shouldn't I only be interested in the compilation of many test results that indicate a general threshold that would be applicable to anyone implementing a CODEC? So I guess what I'm really asking is, does a single test result of corrupt data pose any real concern or is this just an insignificant issue and Foobar's implementation of ABX testing is adequate, rigorous and reliable?

Thanks in advance.