Total Members Voted: 161
That's great and all, but what about the codecs that are actually going to be tested?
greynolI invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272
An advanced ABXer knows that a codec that his bad at 128Kbps isn't suddenly good at 96Kbps ...
But a codec that is bad at 96kbps may suddenly be better than its peers at 128kbps.
If the test turns out to be a multiformat test...
Quote from: IgorC on 16 January, 2010, 06:03:41 PMgreynolI invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272What makes you think I haven't done this already?
You can have all encoders put to shame at 128Kbps or all encoders looks shining at 96Kbps, all this just by changing the set of samples you use.The choice of samples will be harder at 128Kbps than at 96Kbps ... at 96kbps you cannot really do a mistake in the choice of samples ... at 128Kbps lots of samples will be hard to ABX for beginners.Not all killer samples are equal, some old killer samples have become less critical with recent encoders, but some people still use them for listening tests ... if you chose one of those at 128Kbps it will likely be a sample tested for nothing.
Another 128kb/s test has no value to me. It just doesn't give conclusive results about what encoder is doing universally better.
A 128-kb as I proposed it above is not trying to answer that question. It will answer the question "which AAC encoder does best given very critical audio material". The thing is, universally, all popular modern AAC codecs do very well at this bitrate.
But how do pick "critical" material in an unbiased manner? If you just pick problem samples for AAC from this forum, ...
Would you pick problem samples identified for other formats then?
So does that mean I can put you down as one person who can routinely distinguish 128kbit AAC from lossless as part of your regular music listening experience?
Quote from: Mike Giacomelli on 16 January, 2010, 09:49:45 PMBut how do pick "critical" material in an unbiased manner? If you just pick problem samples for AAC from this forum, ...I don't rely on posts in this forum exclusively. I'm in the process of finding my own additional samples, as well as going through about 100 MP3-critical samples posted or linked to in this forum.QuoteWould you pick problem samples identified for other formats then?No. This is supposed to be an AAC test, so I only search for AAC-specific ones. However, AAC critical items should also be MP3-critical, since the underlying codec technologies are similar. If, however, I'll find a sample which is AAC- but not MP3-critical, I'll report.Chris
If you pick AAC specific problem samples, then you bias the test results against whichever codec the samples were identified from, and your results become less useful. For instance, if you pick a lot of samples identified from iTunes encoded files, then your conclusion becomes "For AAC files that iTunes has trouble with, iTunes does poorly . . ."
Your saying that you don't feel that you would be able to contribute simply reinforces my point. Thank you.
Perhaps I'm wrong in that people aren't interested in results that are routine and regular for them, but I don't think so. I may also be wrong in expecting that people are going to make unfounded suggestions that codec X is better than codec Y because of the results of this test, like what was done with the ~128kbit mp3 test, assuming that the results are not particularly interesting except for the killer samples. To this day, there are still people who can't seem to understand that the result of that test was a statistical tie.
Which is why I propose not to randomly select items from a set of "maybe-killer" samples for the 128-kb test, but to do careful pre-screening of listening material and to select only those items for which at least one of the codecs under test is clearly not transparent.
Regarding the "will be hard to ABX for beginners": IMHO, this should not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.
Remove all listeners from analysis who1. graded the reference lower than 4.5,2. graded the low anchor higher than all competitors.3. didn't grade the low anchor.4. didn't grade any of competitors.
We are lucky to have C.R. Helmrich here, and as FhG AAC doesn't participate in the test (unfortunately) I suggest to let him with his AAC dev experience build up a proposal set of samples. He is thinking already about such a set.
My opinion is that despite being very democratic, this pool was biased from the start, there is more than 100 voters but there is not more than 100 HA users doing regulary ABXing & posting their findings.
Remove all listeners from analysis who 4. didn't grade any of competitors.
1. graded the reference lower than 5.0
... But I think then we also have to modify rule 1 to:Code: [Select]1. graded the reference lower than 5.0As I noted earlier, this will greatly reduce the number of listeners left for analysis, but given that we only want experienced ones, so be it. ...
...I'm planning not to publish an average grade for this test, but only to show the results for each item separately, if on average, all codecs will be tied. That will prevent some people of falsely assigning the word "winner" to a codec. ...