Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New web-based ABX testing tool (Read 9921 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New web-based ABX testing tool

Hi all,

It's been a long time since I've been on here but I figured that this might interest some people on here.

I've just finished the first big round of coding on an online ABX tool I've been working on (i.e. knocked up over the last 5 days). It grew out of an online ABX I created after the 'Tidal test' was released. I was annoyed at their test because I thought giving a number correct result was misleading to many people (because 50% correct is change performance), and also that their samples were poorly synced.

Anyway, I initially went and built an ABXish version of their test (http://abx.digitalfeed.net/). Once I'd finished, I figured that the code could probably be modified to be generic, so anyone could run a web-based ABX test. Anyone who is willing to go through the process of encoding files, modifying JSON, etc. anyway.

Up front: this is purely for *presenting* the test. All the hard work of creating files and so on is still manual. Actually, it's more work because you have to split the files into individual channels (stupid issue with the FLAC decoder). The goal of this is for someone to be able to set up a test and send a bunch of other people to do it without them needing to download the samples and Foobar. It also has a way of sending you people's results, which is described in the set-up documentation.

How it works:
You (the test creator) download the 'Fun Pack' (http://abx.digitalfeed.net/abx.digitalfeed.net.funpack.zip), which contains examples and instructions for how to set up a test.
You need to create the samples, images to go with the samples (if that's your thing; otherwise just some images which will 'slide' to show that it is loading), and create a configuration file (an example/template is included in the 'Fun Pack').
You put up the files on an appropriately configured hosting server (Note: Dropbox Public folders work)
You send a link to your prospective participants, which will load your configuration and run your test. Users can either report back results themselves, or (if you have somewhere you can host a web form and collect data) the tool will automatically HTTP POST the results to a URL you specify. You can also give users individual participant ids which will be included in the results POSTed.

I have seriously just knocked this up over the course of a couple of days, so it is very likely that it will have some glitches. It *does not work* in Internet Explorer. IE just doesn't have the APIs needed to do it.

Anyway, if you are interested you can download the Fun Pack and have a look at the README.txt file. It describes pretty much everything that is involved. If you just want to check out how the ABX interface appears, you can go to the list of tests I've created and pick an existing test to do. The only results I'm collecting from those other tests is what you see in the URL - the number of trials you choose and final score (and even then, only when it finally appears in Google Analytics).

In the README.txt file, it has '***REDACTED_PENDING_COMPLETION***'. The filename there is 'dyot' (do your own test), so the URL would be http://abx.digitalfeed.net/dyot.html?test=<your test instructions file URL>&pid=<optional participant ID>

Note: I haven't verified that the audio output from the browser is bit perfect. I'm on a Mac and don't have tools to do so, but if someone else finds it is or isn't it would be great to know.

Note 2: This is absolutely beta. It will very likely have weird bugs, and there are things about the interface I want to change. Feedback is welcome.

Note 3: The structure of the instructions JSON file may change in the initial stages of me fixing things, but I will try to keep backward compatibility.
happiness comes in brown paper bags.

New web-based ABX testing tool

Reply #1
You put up the files on an appropriately configured hosting server (Note: Dropbox Public folders work)
You send a link to your prospective participants, which will load your configuration and run your test. .


  Thanks for your efforts. Is there any mechanism in place where an uploaded file name which already has been used in the system is locked out from anyone attempting to upload another file with the exact same name at a later date? That would be desirable, because it may help prevent one form of fraud.

In theory scammers can falsify current ABX test logs by using a file of the same name as one being used in a public forum and when we look at their ABX test logs it will deceive us that they have listened to the same two files as everyone else taking the test, however in truth one of the files they used was their secondary file which used the exact same name as the true, legit file, but was actually their completely different song masquerading as the legit file, making it easy to ID.

New web-based ABX testing tool

Reply #2
You put up the files on an appropriately configured hosting server (Note: Dropbox Public folders work)
You send a link to your prospective participants, which will load your configuration and run your test. .


  Thanks for your efforts. Is there any mechanism in place where an uploaded file name which already has been used in the system is locked out from anyone attempting to upload another file with the exact same name at a later date? That would be desirable, because it may help prevent one form of fraud.

In theory scammers can falsify current ABX test logs by using a file of the same name as one being used in a public forum and when we look at their ABX test logs it will deceive us that they have listened to the same two files as everyone else taking the test, however in truth one of the files they used was their secondary file which used the exact same name as the true, legit file, but was actually their completely different song masquerading as the legit file, making it easy to ID.


Because this tool is 100% client side Javascript (what is loaded from the server is just HTML, CSS and Javascript files) there is no way for the server to store URLs that have already been used.

However, the URL that you send people is a complete URL. For example, the URL for my 'sample' test in Dropbox (which is the exact same one included in the 'fun pack' is:
abx.digitalfeed.net/dyot.html?test=https://dl.dropboxusercontent.com/u/8333531/abxtest/example_test.json
The full URLs of all the files to be used for the test are contained in that JSON file. Below is an excerpt from the JSON file:
    {
        "artist": "Felix Mendelssohn",
        "album": "Symphony 3 in A Minor",
        "track": "Allegro vivacissimo",
        "thumbnail": "https://dl.dropboxusercontent.com/u/8333531/abxtest/coverart_1.png",
        "background": "https://dl.dropboxusercontent.com/u/8333531/abxtest/background_1.png",
        "A": "https://dl.dropboxusercontent.com/u/8333531/abxtest/mendelssohn.30.lossless",
        "B": "https://dl.dropboxusercontent.com/u/8333531/abxtest/mendelssohn.30.opus.64"
    },

There you can see the full URLs for the various images and FLAC files (kind of; those are kind of URL stubs. There are details about that in the README.txt). So long as you don't lose control of the place you're hosting the test no-one can make that URL point to another test.


Strictly speaking, because the app is completely client-side a devious user could open the Javascript debugger, set some breakpoints, and modify the results directly in memory. The risk of that can be reduced somewhat by me obfuscating the code (which I've not done, because I wanted people who wanted to to be able to read it over). However, if someone is really clever and dedicated it will be impossible to stop them fabricating the results they provide to you.

Unless you actually have somewhere to collect the results (when they are transmitted via a HTTP POST) you have to essentially rely on users to provide the results anyway. (note: that does make me realise I need an easy way for users to get a chunk of text or CSV to send to someone running a test).

Probably the best possible protection is for me to minify the scripts so they are hard to read and therefore modify with a debugger. That will not stop someone modifying the outgoing HTTP POST if they are dedicated. Unfortunately the only way to build an ABX tool really hardened against malicious users trying to defraud the system is to write it in something like Flash, and even then it is only risk reduction.
happiness comes in brown paper bags.

New web-based ABX testing tool

Reply #3
You put up the files on an appropriately configured hosting server (Note: Dropbox Public folders work)
You send a link to your prospective participants, which will load your configuration and run your test. .


  Thanks for your efforts. Is there any mechanism in place where an uploaded file name which already has been used in the system is locked out from anyone attempting to upload another file with the exact same name at a later date? That would be desirable, because it may help prevent one form of fraud.

In theory scammers can falsify current ABX test logs by using a file of the same name as one being used in a public forum and when we look at their ABX test logs it will deceive us that they have listened to the same two files as everyone else taking the test, however in truth one of the files they used was their secondary file which used the exact same name as the true, legit file, but was actually their completely different song masquerading as the legit file, making it easy to ID.



The above is IMO is a very valid concern. Taking a good checksum of the files seems like a working solution.

If you are trying to understand what someone actually did, knowing what they listened to, any processing of the file, and the precise timing of the areas of the file they listened to are the big priorities.

We have to approach this problem from the standpoint of both scammers and inexperienced listeners.

New web-based ABX testing tool

Reply #4
from the standpoint of both scammers and inexperienced listeners.


A warning that one actually uploaded AA instead of AB could be useful for quite a few of us ...

Against the inexperienced golden ear who makes no more sophisticated scam than retaking the test until getting a sufficiently impressive p-value, it would be better to have different checksums which could later be identified as coming from the same audio ;-)

New web-based ABX testing tool

Reply #5
from the standpoint of both scammers and inexperienced listeners.


A warning that one actually uploaded AA instead of AB could be useful for quite a few of us ...

Against the inexperienced golden ear who makes no more sophisticated scam than retaking the test until getting a sufficiently impressive p-value, it would be better to have different checksums which could later be identified as coming from the same audio ;-)


Another approach would be to maintain a master log that counts the number of times or the amount of time that a file with a given checksum has been used.

New web-based ABX testing tool

Reply #6
from the standpoint of both scammers and inexperienced listeners.


A warning that one actually uploaded AA instead of AB could be useful for quite a few of us ...

Against the inexperienced golden ear who makes no more sophisticated scam than retaking the test until getting a sufficiently impressive p-value, it would be better to have different checksums which could later be identified as coming from the same audio ;-)


Another approach would be to maintain a master log that counts the number of times or the amount of time that a file with a given checksum has been used.



These are all really valid concerns, and I've been pondering them overnight.

Before anything else, I should point out: someone sufficiently dedicated will always be able to overcome protections simply because this is a completely client-side application. That means someone who is willing to use a Javascript debugger will *always* be able to hack it.

Having said that, there are several things that can be done to make sure some of the common problems are addressed. I'll list the concerns as I understand them, and ideas for what might solve them. I should point out that I'm not sure how long it will/would take me to implement these. This tool basically came out as a partial byproduct of a few-day coding frenzy so I'll probably be taking a couple of quiet programming-free days soon =)


Anyway:
Issue 1: people replacing audio files
How it could be done: A user could download the test 'instructions' JSON file, modify the filenames so that they point to another file on another server, and do the test using that.
Possible mitigations:
1. Send the full URL of the test 'instructions' file through in the final data (the HTTP POST sent at the end to a URL you specify). That way it would be apparent if it had changed.
2. Construct a SHA/MD5 of the tracks: this is possible, but FLAC decoding in Javascript is already pretty processor intensive so I'm a bit loathe to actually do it. My preference would be to make sure that modifying the URLs would be as hard as faking the SHA/MD5 would be (thus making this solution kind of inefficient and redundant).
3. Send the full URL of the test 'instructions' file through in the final data *and* only allow relative URLs for the files from the location of the settings file. Thus, if you pointed the settings file to 'http://your-server.invalid/my-settings-file' then all of your FLAC files would also need to be on 'http://your-server.invalid'. In reality, this probably provides only a marginal security benefit over #1 if any, because if the user is intercepting the loading of the settings file to patch the filenames, they can probably trace the code and modify the URLs in memory anyway. *I would implement it anyway, but my reservation is that it means the settings file and FLAC files MUST be hosted together*. I'd love feedback on this one.


Issue 2: Repeated test-taking
How it could be done:
A user could start the test, do it, and then if they were not happy with the result then just close the browser before the data is sent. They might also just remove the 'participant ID' from the URL so they are not identifiable.
Possible mitigations:
1. In the settings file, put in a flag to say whether participant IDs are mandatory - i.e. no participant ID, no test. AND
2. When a test is started, either:
  (a) Have the page do a HTTP POST to the 'post-to' URL including the participant ID, so that the person running the test can check if a participant is doing the test twice. The problem here is that someone can just make up fake participant IDs because there is currently no list (and you can't put the list in the settings file because someone can just look at it and pick one)
  (b) Have the page do a HTTP POST to the 'post-to' URL including the participant ID, and *YOUR REMOTE SERVER, WHICH YOU CURRENTLY NEED TO RUN TO HANDLE THE POSTS* responds with an 'OK' or 'NO-GO' message. Basically, *your* server would log which participant IDs are OK and which have been used, and then respond on that basis. It means you'd know who started the test.

The challenge with each of these is that they are increasing the complexity of the code that needs to be hosted by the person running the test. Maybe that is an acceptable trade-off for the security gains, but it might just be that I need to build this stuff into the server too. Hrm.
In any case, I would probably need to provide sample code for capturing the POSTS, handling the IDs and so on.


3. People faking the results sent to the POST-TO server
A user could intercept the final data sent sent to the POST-TO url and modify it, changing it to make them perform well
Possible mitigations:
1. Incorporate digital signing into the POSTing process. This would involve me incorporating actual encryption code into the page, and the settings file including a 'public key'. The public key would then be used to digitally sign the results before sending so that modifications would be detected. The challenges with this are:
  (a) it means incorporating encryption code
  (b) it means working out how to let people create the encryption keypair
  © it means providing a way for people to verify the validity of the sent results using the encryption keypair
These are, as you might imagine, pretty significant. Also, they still don't stop someone who is a reasonable coder using a Javascript debugger to modify the results in memory because then those modified results would be included, and signed as correct, in the result.





In terms of knowing details like timing, etc, I'm not sure exactly what kind of information you mean. The current results will include a 'start time' and 'end time', but that will include any time viewing intermediate results pages and so on. Of course, if I include the ability to disable intermediate results pages in the test configuration then that might help. I could also potentially record (and thus send) a whole bucketload of details: start time for each trial, time at which each button was pressed, etc. That is more complicated though so would take me longer.



Like I said, I'm not sure when I'll be able to get to some of these. I'm also responding to a few comments/requests about the original tidal test remake which this is based on (e.g. adding share buttons, ugh; showing % likelihood that the result could be by chance rather than just p<.05, p<.01, etc.).


Thanks for the feedback though. I honestly haven't done anything ABX-related for probably 9 years so it is really helpful to have input from others.

Addendum regarding digital signatures, music files, and so on:
In terms of providing security in testing, the ideal would almost certainly be for the ABX testing tool (i.e. the foobar one) to be able to read a binary 'configuration' file which includes:
1. File hashes
2. Public key for digital signing
Then when each track is played the hashes could be checked, and the results could be encrypted signed using the public key to ensure validity. By digitally signing the file hash list with the matching private key, the ABX comparator tool could verify that the list is valid *if* the public key is valid. The main challenge is stopping people extracting the public key, faking the results, and encrypting them themselves. Stopping that is virtually impossible, but you can make it relatively difficult (i.e. crack out the binary disassembler difficult)
Just a though.
happiness comes in brown paper bags.

 

Re: New web-based ABX testing tool

Reply #7
Sorry to bring up an old topic = but is gnoshi still the maintainer of this test site? I did not do as well on the test as I thought I might, but wondered if the samples on the site were well recorded to begin with?