I just finished putting this (https://github.com/oxtoacart/pyabx) together. Haven't had a chance to run any real ABX tests with it yet, but looking forward to playing with it.
Don't forget to soundproof the relay enclosure, or configure the computer controller to always toggle all channels, even when not switching between one or the other source, so you can't just detect a match by lack of relay clicking.
E: I just realized, if these outputs or inputs aren't filtered, the mere act of relay switching will already produce an audible effect on the output. So forced switching will be needed either way.
Good point about the switching noise. I had been switching the unused relays to make sure there's always a clicking noise, but that doesn't address the noise on the output itself ...
What I might try is wiring both sources to two pairs of relays, to the normally open position on one and the normally closed on the other. When switching, I'll switch all four relays, so that both sources always include a closed->open transition and vice versa.
Okay, my program now just throws in 50 random switches with a 1ms delay and that produces enough noise on each transition that I can't hear a difference. Yeah it's not a smooth transition but it's not too bothersome.