Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Quality Bitrate Test (Read 22069 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Quality Bitrate Test

Reply #25
Johnston: Very useful post. Thanks for your contribution.



About the way we tested, and I'm not saying this to rectify or justify our way of testing because I have already admitted it hasn't been very scientific.

I would play the reference, and the listener was informed the reference was playing. I would then play 2 or 3 other tracks, which could also include the reference track. The tracks I picked were based on on the feedback I got from the listener. On request, I would switch back to the reference.

The next time I will stick to the rules for a double blind test.

I also think I used too many compressed tracks and the song I picked was limited in terms of instrument variation. I think its therefore better to use less compressed tracks in the test but use 2 or 3 different songs.

Quality Bitrate Test

Reply #26
I would play the reference, and the listener was informed the reference was playing. I would then play 2 or 3 other tracks, which could also include the reference track. The tracks I picked were based on on the feedback I got from the listener. On request, I would switch back to the reference.

Sounds like ABC-HR to me.
err... i'm not using windows any more ;)

Quality Bitrate Test

Reply #27
Hi Ingemar,

Thank you for your interesting report. It sounds like an interesting test, and you have reported it well. I'd be interested to read the full Word document, but if you're happy to share it and you own the copyright, why not upload it to HA for everyone to see?

I don't discount your results. People who consistently hear things when they don't know what they're listening to are usually really hearing something.

However, there are three issues with your test which I would fix in the future if I were you. They've already been mentioned, but just to make sure you realise which ones are most important...

Firstly, I'm guessing you recognise the problem of a faulty CD player which is "fixed" without the use of test equipment, so I won't labour the point, but as you know, it's absolutely critical to the test. You might want to check that any CD player is capable of bit-perfect reading of CD-Rs, and that distortion, frequency response, jitter etc are all measurably identical (or at least acceptable!) on CD-Rs and pressed CDs. If you want to know how to do at least some of this without expensive equipment (a good sound card is sufficient for some testing), then just ask or search. This is a common starting point: http://audio.rightmark.org/products/rmaa.shtml

Secondly, there may be lies, damn lies, and statistics, but it's absolutely critical to know how the results compare with those that could be achieved by chance. For this, it's important to do an appropriate number of blind comparisons in an appropriate manner. You are not a good enough random number generator; use a real one! Changing what is played next based on previous responses is not acceptable, though you can tell people whether each response is right or wrong after they have committed to it. Report all the results: no picking or choosing of results is allowed. The number of tests which "seems reasonable" when you are testing (typically 3 or 4, if you're like me) isn't actually enough. 8 is usually set as an absolute minimum, with no leading, guiding, or bias in the samples - this means you have one known reference, one hidden reference, and one hidden compressed track, and must correctly identify the hidden tracks. Having one hidden reference, and two or three hidden compressed tracks hugely biases the test, makes calculating the statistics much harder, and probably increases the number of tests that must be performed to reach statistical significance.

Thirdly, I think you need to use the "best" versions of any encoder you choose to test. There are plenty of threads about this

Hope this helps.

Cheers,
David.

Quality Bitrate Test

Reply #28
@David: I'll be happy to send you the Word file, but it doesn't contain more info than I posted here. It's just a little nicer format using tables etc. Send me a pm with your email address and I'll send it over.



If your question remains, then I don't know 

Pff ... I don't even know what aoTuV is so I'm guessing regular ogg?


Some info on aoTuV:
http://wiki.xiph.org/index.php/VorbisEncoders

if you want to know which encoder has been used on your files, download ogginfo here: http://rarewares.org/ogg.html and do:
ogginfo file1.ogg


OggInfo output:

Code: [Select]
F:\media\test>ogginfo.exe UncompressedWave.ogg
Processing file "UncompressedWave.ogg"...

New logical stream (#1, serial: 00002491): type vorbis
Vorbis headers parsed for stream 1, information follows...
Version: 0
Vendor: BS; Lancer(SSE3MT) [20060915] (based on aoTuV r1 [20051117])
Channels: 2
Rate: 44100

Nominal bitrate: 499,821000 kb/s
Upper bitrate not set
Lower bitrate not set
Vorbis stream 1:
        Total data length: 14167217 bytes
        Playback length: 4m:13.506s
        Average bitrate: 447,079903 kb/s
Logical stream 1 ended


Guess I was using aoTuV after all. 

Quality Bitrate Test

Reply #29
Johnston: Very useful post. Thanks for your contribution.

Fighting sarcasm with sarcasm, eh?

Anyhow, tagging your thread with, "Three audiophiles subjected to blind tests," is quite misleading since they weren't actually blind tests. Trying to justify that mistake with excuses isn't helping your case too much either. I'm glad you've decided to go through with a retest following the rules. I would lay low until you have those results ... this is Hydrogen Audio after all; results are what speak.

I also think I used too many compressed tracks and the song I picked was limited in terms of instrument variation. I think its therefore better to use less compressed tracks in the test but use 2 or 3 different songs.

I am not personally familiar with the song you picked, but you would want to keep complexity and characteristics of individual instruments in mind. When picking your next songs, I would pick three. One a slow song/ballad, the second a moderate rock song with some good rhythm distortion guitar sounds, and the third a real complex electronica or electronic rock song with a lot of busy parts, panning, glitch samples, etc.

Hi-hats are also something to pay attention to, especially in rock pieces. Take any rock tune that has a strong, driving hi-hat during the chorus and put it through LAME at 128 CBR and you'll see what I mean. I would use such a song as number two in my three-song example from the previous paragraph.

I look forward to the results of your retest.

Quality Bitrate Test

Reply #30
You might want to try KikeG's nifty fileabx program. It will generate all the WAV's you need for a proper ABX test, just feed it the reference and the encoded files, and it will create a directory with files A.wav, B.wav, and X01.wav to X08.wav (8 trials by default). Just burn the wavfiles to CD and try to identify all 8 "X files"  (which is which can be found in the file "results.txt", just be sure to make the test before reading it)


Cheers, Joey.

Quality Bitrate Test

Reply #31
I apologize for being blunt; the observations are interesting, if subjective. The problem I have with them is that they agree with double blind tests to some degree, but to some degree they do not. I fear that some may be mislead by the truth that they convey into trusting in some erroneous observation.

Quality Bitrate Test

Reply #32
@Johnston: I didnt' really think about it, but its good to see you didnt take offense and have come up with a more constructive post  I will take your tips into consideration. You might want to reread my explanation on how we did the test. It was definitely a blind test but taken in such a way that the results are dubious. I have acknowledged that already and will try to do the test again in a way that 1) will give me peace of mind (since I have become sceptical too TBH) and 2) is acceptable to the community.

@Canar: Thx and accepted. I will edit my initial post to make readers aware.

@Joey: Thx, excellent tip, makes life so much easier