Poll
Question:
96 or 128 kbps public test?
Option 1: ~96 kbps test on "average music" samples with little amount of difficult samples
votes: 55
Option 2: ~128 kbps test with a higher amount of difficult samples
votes: 85
Poll is dedicated for upcoming public test.
I understand there is a plethora of 128 tests out there and it's getting difficult to ABX, but I don't see why a 96 is necessary. I would hope that codec programmers are focusing on higher bitrates nowadays.
Vote: 128
A little on the fence about this. I haven't used "128kbps" as a target bitrate in about 6 years and it appears that storage capacities are constantly increasing while their prices keep falling. I don't think the smaller bitrates (ie 96kbps and below) have the same place in portable audio as they did years ago especially with "affordable" 32GB+ players and 64GB solid state players on the verge (except the iPod touch, that is already out but it is really expensive). So I voted for 128kbps. I realize that this will make testing much harder. Ideally, a target bitrate of 192kbps would have been nice but that would have exponentially increased the difficulty of the tests.
Let's find out if 96kbps could be viable for acceptable music listening. I do agree with storage capacities getting so large and inexpensive that 96k might seem irrelevant, however it would still be interesting for mobile applications.
I voted 128 kbps because I am interested to see which encoder would be the best for internet radios. Most of the ones I am listening to are streaming at 128 kbps and I am wondering which encoder would be the best -- if it makes a difference at all. Storage space is not a criterion for me.
I voted 96 because I assume 128 has already had lots of killer samples thrown at it by developers, & because 96 isn't a rate so many people consider for music.
I'm voting for 128 because most players have oodles of space to store music, and there isn't a real need for super-low bitrates anymore. Granted it will be tough, but isn't that a good thing (meaning most codecs are near transparency at such a low rate)?
I am more interested in the results of a 128kbps test. Hope there will be enough "difficult" samples to make the test easy to participate in.
Voted for 96 kbps even though it would've been more interesting to test even 80 kbps IMHO.
Voted for 96 kbps. Also this would be good to know the actual size of bit reservoirs (though I would rather be interested in CBR only test).
We believe that bitrates for test should be in a linear scale: 32, 64, 128, 256...
In this case, 128. BTW 64 test would be great too.
I don't see any particular reason why bitrates should be in a linear scale while all competitors admits bitrate shifting.
Even more. In case of 96 kbps test all bitrates settings could be shifted to 102 kbps or even 102.77.... kbps
We believe that bitrates for test should be in a linear scale: 32, 64, 128, 256...
In this case, 128. BTW 64 test would be great too.
Linear? 32,64,128,256 .. is exponential. +32 +64 +128. I don't see how it comes any linear
Linear would be 32,64,96,128.... +32 each step. That's linear.
The MPEG bitrate scale is both linear & exponential because it is linear within a range but ranges are exponential:
Low Quality Range: +4Kbps by Step
036Kbps
040Kbps
044Kbps
048Kbps
---------
(050Kbps The middle of the Low Quality Range)
---------
052Kbps
056Kbps
060Kbps
064Kbps
Medium Quality Range: +8Kbps by Step
---------
072Kbps
080Kbps
088Kbps
096Kbps
---------
(100Kbps The middle of the Medium Quality Range)
---------
104Kbps
112Kbps
120Kbps
128Kbps
High Quality Range: +16Kbps by Step
---------
144Kbps
160Kbps
176Kbps
192Kbps
---------
(200Kbps: The middle of the High Quality Range)
---------
208Kbps
224Kbps
240Kbps
256Kbps
Note: 320Kbps is an orpheline bitrate of the +32Kbps by step range.
This bitrate scale alone is rather useless to judge quality, but it explains why there is such a quality difference between 128Kbps & 192Kbps when you do ABXing. There is no secret, it's the big bitrate boost. Also, it learns you why using 320Kbps is overkill when 192Kbps is already almost always transparent.
Back on the topic, voted 96Kbps because on non-killer samples with non-trained users, 128Kbps will be near transparent.
Edit:
I would have voted:
96 kbps test
+ with a higher amount of difficult samplesif I would have had the choice.
The choice of samples is much more important than the choice of bitrate IMHO. (As long as all AAC competitors are LC)
IMHO 96Kbps or 128Kbps, it shouldn't change the overall results (I mean ranks) between competitors, but using 96Kbps will make the test easier.
128. Seems like the default for alot of ripping/media players.
Definitely 96k. 128k has been tested abundantly and as the most recent 128k tests have shown, public tests simply don't produce results that are statistically viable enough. 96k on the other hand has never been tested on a large - public - scale.
96kbs for sure... we've done 128 to death and all the discussed encoder options do exceptionally well with it. let's see who's pushing the envelope.
Of those who chose 128, I'd like to know how many of you can routinely ABX this from lossless. How many of you either include killer samples or listen to music that can be characterized by killer samples as part of your regular listening experience?
@greynol
I don't know about killer samples, but music with, say, a lot of hi-hat, or harpsichord, is generally regarded as being difficult. I'm sure nobody listens to all sorts of killer samples regularly, but some people would listen regularly to difficult music.
The point behind the 128 preference, with a heavy weighting of difficult samples, would be to discover, I guess, which specific codecs handled which particular difficulties best.
The general assumption, prior to testing, would be that good all codecs are going to be performing very well (quite possibly indistinguishably) on most samples; but there might still be information to be retrieved.
That's not to say that 128 is inevitably a better choice than 96, but to suggest that there is a rationale, and it's not some kind of perversity to choose the higher bit-rate. I guess it's a question of what purpose tests are serving, and if that could be articulated very clearly, maybe the choices would become clearer.
So does that mean I can put you down as one person who can routinely distinguish 128kbit AAC from lossless as part of your regular music listening experience?
If you wouldn't say routinely but regularly, you could add +1 for me. At 128k LAME still has audible problems with sharp attacks (e.g. castanets). Hihats have become better, but aren't perfect, yet. This isn't restricted to just a dozen "killer samples" but isn't too seldom for modern Jazz and classical recordings. That's the reason I always use higher bitrates. One can certainly live with that, it still sounds great. But you can identify that it isn't lossless. Precondition: quiet environment, sufficient volume. And that's overall volume, not exaggerated at problematic positions.
Of those who chose 128, I'd like to know how many of you can routinely ABX this from lossless. How many of you either include killer samples or listen to music that can be characterized by killer samples as part of your regular listening experience?
If you want to say that most people are ecpected not to encounter problems within their regular listening experience when using AAC @ 128 kbps: well, that would be a fine result. But we haven't done the test yet. We just don't know, and we don't know whether specific encoders have specific issues in certain problem areas (and maybe not only in problem areas).
The reason for testing @ 128 kbps is simply that more or less everybody can afford such a bitrate.
If the main target was to find a clear qualitative ordering among the contenders a lower bitrate would be more appropriate. But what would it be good for if nearly nobody uses it?
Of course it can happen that all the contenders will score more or less near-perfect (but it's not necessarily so) similar to the last public 128 kbps mp3 test. But that would be a very fine result IMO telling us that all the encoders tested do an excellent job.
At 128k LAME still has audible problems with sharp attacks (e.g. castanets). Hihats have become better, but aren't perfect, yet.
That's great and all, but what about the codecs that are actually going to be tested?
Of course it can happen that all the contenders will score more or less near-perfect (but it's not necessarily so) similar to the last public 128 kbps mp3 test. But that would be a very fine result IMO telling us that all the encoders tested do an excellent job.
I agree, though I wonder if those who voted 128 have considered this as a possibility.
greynol
I invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272 (http://www.hydrogenaudio.org/forums/index.php?showtopic=77272)
If the main target was to find a clear qualitative ordering among the contenders a lower bitrate would be more appropriate. But what would it be good for if nearly nobody uses it?
Of course it can happen that all the contenders will score more or less near-perfect (but it's not necessarily so) similar to the last public 128 kbps mp3 test. But that would be a very fine result IMO telling us that all the encoders tested do an excellent job.
Yes but its would also be an entirely expected, and thus very boring result. Its only an interesting result if the AAC encoders do significantly worse then MP3 encoders, which seems extremely unlikely. IMO since we can do so few tests, we should try to pick ones that are likely to give good results. Negative controls are great and all, but if you only are likely to do one test, a control is probably not the best first choice
96k will be more interesting given that it is expected that some or even all codecs may not perform quite so well.
That's great and all, but what about the codecs that are actually going to be tested?
WTF? I realized just now that LAME is not planned to be included. A second bell rings, duh, the whole things seems to be called "AAC test" for a while now and not "multiformat", anymore.
From my subjective perspective, we have accumulated a lot more unfinished AAC vs. MP3 business recently than which AAC encoder was the best. No chance to slip LAME in there with the special role as venerable touchstone?
greynol
I invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272 (http://www.hydrogenaudio.org/forums/index.php?showtopic=77272)
What makes you think I haven't done this already?
My opinion is that despite being very democratic, this pool was biased from the start, there is more than 100 voters but there is not more than 100 HA users doing regulary ABXing & posting their findings.
What this means is that many testers will be the average Joe testing their ABXing skill. But IMHO on a daily basis the average Joe is likely to use a bitrate higher than 128Kbps because for his daily music he likes transparency.
Now what this means for this pool is that the average Joe, is very likely to be willing to test a setting that is the closest possible to the setting that he uses on a daily basis, that's why the average Joe is very likely to vote for 128Kbps.
Some people said that testing 96Kbps would not be a great idea because nobody would use 96Kbps to encode his music ... this is exactly the behavior that I was expecting from an average Joe.
An advanced ABXer knows that a codec that his bad at 128Kbps isn't suddenly good at 96Kbps ... so it doesn't matter ... what matters is the choice of samples.
You can have all encoders put to shame at 128Kbps or all encoders looks shining at 96Kbps, all this just by changing the set of samples you use.
The choice of samples will be harder at 128Kbps than at 96Kbps ... at 96kbps you cannot really do a mistake in the choice of samples ... at 128Kbps lots of samples will be hard to ABX for beginners.
Not all killer samples are equal, some old killer samples have become less critical with recent encoders, but some people still use them for listening tests ... if you chose one of those at 128Kbps it will likely be a sample tested for nothing.
I voted 96.
Based on my experience of all Sebastian's tests a 128 kbps AAC public test would be unpleasant and unrewarding for the testers.
In some of the past tests 96 kbps iTunes VBR was used as a high anchor. It was never transparent to me, but I didn't find it annoying at all. In fact, it was the only encoder in the 64 kbps multiformat test that was not annoying to me. In general, I was disappointed to the quality of 64 kbps AAC-HE. However, when I listened through a playlist of the 64 kbps HE-AAC test samples in a casual situation (using my big hifi speakers instead of headphones) I was surprised about the apparently qood quality. Perhaps this explains why so many testers had problems with hearing the HE-AAC problems in the test. You really need to have lots of experience before the minor artifacts can be detected.
I don't think the samples in the previous tests can be considered "easy". They were intended to represent various typical encoding problems and were chosen after careful consideration.
In general I don't think the so called killer samples are very deadly for 128 kbps AAC. Obviously they are far from transparent when the tester is experienced and knows what to look for, but detecting the problems might be difficult for a tester who has no previous knowledge of the typical artifacts.
The two most recent 128 kbps tests clearly showed that many testers had problems with finding any differences between the encoders.
It would be good to start a new round of HA tests with a test that is easier for the testers and for the test organizers. It would already be difficult to find a good sample set for a 96 kbps test. After a successful 96 kbps test it would be easier to organize the more demanding 128 kbps test when all the experience of the 96 kbps test is available.
And there should not be a long break between the tests. Once the new generation of participants has been summoned and trained it should be actively used.
P.S.
If the test turns out to be a multiformat test the only addition to a 96 kbps test besides AAC should be Vorbis. MP3 is really not competitive @ 96 kbps (I believe many HA users can agree that or do I need to post ABX results?).
A 128 multiformat test would be different. MP3 should be included. However, I am not sure if it should be LAME. There was no winner in the last public 128 kbps MP3 test.
An advanced ABXer knows that a codec that his bad at 128Kbps isn't suddenly good at 96Kbps ...
But a codec that is bad at 96kbps may suddenly be better than its peers at 128kbps.
PS
But I think you still have a point regarding average Joe voting behavior and its roots.
I'm an average joe, and I voted for 96.
I can only speak for myself & I already posted my logs but for me Harlem & Autechre are deadly for AAC 128Kbps, I don't use lossy but I wouldn't use AAC below 192Kbps VBR actually.
The so-called quality of AAC 128Kbps is more due to poor average listening skill & bad samples choices, than to the real efficiency of AAC 128Kbps.
AAC 128Kbps is only usually transparent, if you randomly listen to average music. I hope average music is not what will be selected for the test.
rpp3po
But a codec that is bad at 96kbps may suddenly be better than its peers at 128kbps.
This is not my experience. I think that becoming "suddenly better" is more an exception than the rule.
What I mean is that from my experience the ranking between codecs didn't change, it does happen that a codec which was bad at 96Kbps is suddenly transparent at 128Kbps when other codecs are still not transparent ... but within all the encoders at 96kbps it is very likely that it was already the best result. (There is the exception of some critical point within the encoder like suddenly no more SBR, but these are exceptions)
It never happened to me that codec X would beat codec Y at 96Kbps & that suddenly codec Y would beat codec X at 128Kbps, all this within the same sample. Different codecs have different strenght & weakness on different samples, but on a target same sample the results are usually very linear.
If the test turns out to be a multiformat test...
How is that coming? Magic?
oh, yeah I see: The choise of AAC competitor for multiformat will be based on.... on... on....
on what?
greynol
I invite you to read the whole discussion's topic http://www.hydrogenaudio.org/forums/index....showtopic=77272 (http://www.hydrogenaudio.org/forums/index.php?showtopic=77272)
What makes you think I haven't done this already?
Ok, discussion around higher bitrate was started from actually very reasonable rpp3po's jump into it (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77272&view=findpost&p=678442).
Actually Sebastian's test were some kind of average music + some difficult (
not so killer test). It's important.
Untill today there is no any public killer sample test. How can somebody be sure if it's usefull or not when it was never performed?
Excuse me my naivety but when Einstein has spoken for the first time about his theory everybody thought he was retarded.
Let's think a little bit differnt just for one test. If we wrong about 128 then that's ok. Good conclusion would be that 128 for average listener will be more than enough even for very killer samples.
Untill now we can't say that.
Can we?
Another 128kb/s test has no value to me. It just doesn't give conclusive results about what encoder is doing universally better. 96kb/s should do that much better.
A fully-AAC test makes even less sense. Why are we treating AAC like a more major format than it is or ever will be? If you want to do a multi-encoder AAC test, then it doesn't require all the build-up; just do a quiet side-test with the other 12 people on Earth using it as their primary lossy format.
You can have all encoders put to shame at 128Kbps or all encoders looks shining at 96Kbps, all this just by changing the set of samples you use.
The choice of samples will be harder at 128Kbps than at 96Kbps ... at 96kbps you cannot really do a mistake in the choice of samples ... at 128Kbps lots of samples will be hard to ABX for beginners.
Not all killer samples are equal, some old killer samples have become less critical with recent encoders, but some people still use them for listening tests ... if you chose one of those at 128Kbps it will likely be a sample tested for nothing.
Which is why I propose
not to randomly select items from a set of "maybe-killer" samples for the 128-kb test, but to do careful pre-screening of listening material and to select only those items for which
at least one of the codecs under test is clearly not transparent.
Regarding the "will be hard to ABX for beginners": IMHO, this should
not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.
Another 128kb/s test has no value to me. It just doesn't give conclusive results about what encoder is doing universally better.
A 128-kb as I proposed it above is not trying to answer that question. It will answer the question "which AAC encoder does best
given very critical audio material". The thing is, universally, all popular modern AAC codecs do very well at this bitrate.
Every sentence of your post shows that you are not interested in AAC. Believe it or not, but AAC is a major format, and more than 12 people are using it (not necessarily only people ripping their CD collection... think further: iTunes, YouTube, lastFm, ...). For me, a 128-kb test as above will have a lot of value, because it will show me how close to transparency you can get at that bitrate with the most critical material available.
Chris
A 128-kb as I proposed it above is not trying to answer that question. It will answer the question "which AAC encoder does best given very critical audio material". The thing is, universally, all popular modern AAC codecs do very well at this bitrate.
But how do pick "critical" material in an unbiased manner? If you just pick problem samples for AAC from this forum, you'll mostly get things that break Nero and maybe a few that break iTunes. Thats not really a fair test since theres much fewer people who would identify problem samples in other encoders. Would you pick problem samples identified for other formats then? Perhaps this is less biased but these files may be less "critical" then you'd like.
But how do pick "critical" material in an unbiased manner? If you just pick problem samples for AAC from this forum, ...
I don't rely on posts in this forum exclusively. I'm in the process of finding my own additional samples, as well as going through about 100 MP3-critical samples posted or linked to in this forum.
Would you pick problem samples identified for other formats then?
No. This is supposed to be an AAC test, so I only search for AAC-specific ones. However, AAC critical items should also be MP3-critical, since the underlying codec technologies are similar. If, however, I'll find a sample which is AAC- but not MP3-critical, I'll report.
Chris
So does that mean I can put you down as one person who can routinely distinguish 128kbit AAC from lossless as part of your regular music listening experience?
@greynol
If this was directed at me, of course not. I would not claim to be regularly able to detect 96kbit from lossless either. But, on the other hand, I never said or implied that I or any body else, come to that, could.
What I actually said was that we expect all AAC encoders to be good at 128, but that performance on really different examples might be a possibility for differentiation. Of course, it could be that 96 + difficult samples would be more informative.
I only chipped in because you seemed to be getting angry because some people were voting for 128. You still seem to be too angry to actually read what people are saying; which, in my case, was that there could be a reason for a different choice from the one you favour (though not denying that your preference might, ultimately, be the better).
I notice that some people with more knowledge and experience and better ears than me see a point in a test at 128 with difficult samples; I also note that there are problems with that. Further, there seem to be several different ideas as to what a test now is meant to achieve; some people seem to want an exciting ABX test, with good sport in the field; some people want a general rank ordering; some people want something that illuminates the really difficult areas and fine points of encoders that are, in general terms, already good enough.
The only dog I have in this fight is help in choosing settings that won't hurt other people's ears. If this poll is only for participants in the test, and not for consumers, then I would certainly respect that decision, but it ought to be made clear. I really think things would be a lot clearer, and there'd be more light and less heat, if the purpose of tests were articulated.
I'm not angry, I really don't even care about the outcome. I just think there is a serious risk that the test is not going to reveal the results people are hoping for. I think more weight should be given to those who have actually conducted these tests and/or those who have demonstrated an ability to detect artifacts than to those who will likely not participate let alone provide data that will be useful.
Your saying that you don't feel that you would be able to contribute simply reinforces my point. Thank you.
Perhaps I'm wrong in that people aren't interested in results that are routine and regular for them, but I don't think so. I may also be wrong in expecting that people are going to make unfounded suggestions that codec X is better than codec Y because of the results of this test, like what was done with the ~128kbit mp3 test, assuming that the results are not particularly interesting except for the killer samples. To this day, there are still people who can't seem to understand that the result of that test was a statistical tie.
But how do pick "critical" material in an unbiased manner? If you just pick problem samples for AAC from this forum, ...
I don't rely on posts in this forum exclusively. I'm in the process of finding my own additional samples, as well as going through about 100 MP3-critical samples posted or linked to in this forum.
Would you pick problem samples identified for other formats then?
No. This is supposed to be an AAC test, so I only search for AAC-specific ones. However, AAC critical items should also be MP3-critical, since the underlying codec technologies are similar. If, however, I'll find a sample which is AAC- but not MP3-critical, I'll report.
Chris
If you pick AAC specific problem samples, then you bias the test results against whichever codec the samples were identified from, and your results become less useful. For instance, if you pick a lot of samples identified from iTunes encoded files, then your conclusion becomes "For AAC files that iTunes has trouble with, iTunes does poorly . . ."
Didn't somebody (muaddib?) come up with a sample choosing methodology that would be fair to use? When I read about that it seemed to make a lot of sense to me. Or we could just classify each type of encoder problem (preecho, tonal, etc.) and make sure there is an even distribution of each type of problem, and within that the usual even distribution of genres, recording era, etc.
About this poll, I don't really mind either bitrate but I would like a high proportion of difficult samples for the reason others have said. Because without them most AAC encoders are so good and results might not be as strong statistically.
If you pick AAC specific problem samples, then you bias the test results against whichever codec the samples were identified from, and your results become less useful. For instance, if you pick a lot of samples identified from iTunes encoded files, then your conclusion becomes "For AAC files that iTunes has trouble with, iTunes does poorly . . ."
Not really following you here. If we're talking about an AAC shootout (which determines the one AAC codec to be used in the multiformat test), then the sources would have to be wav/flac samples, and every AAC encoder is on equal footing.
Your saying that you don't feel that you would be able to contribute simply reinforces my point. Thank you.
It is not clear that this thread is addressed only to people who can contribute useful results. It is not perverse to think it
might be asking for a general expression of opinion from people who will read the results (and who will, by the way, value the effort and skills of the people who conduct them). I accept that, in at least some people's view, this is in fact a poll of potential participants. That's fine, and if there were a way of retracting my vote for 128, I would. But it would be better if the target audience of the poll were made clear, rather than being snarky to people who guessed wrong. The internet is a very public place, and if the title of the thread has the word "public" in it, it's good to indicate that it's not open slather. I really can see the point of asking just the people who are going to test.
Perhaps I'm wrong in that people aren't interested in results that are routine and regular for them, but I don't think so. I may also be wrong in expecting that people are going to make unfounded suggestions that codec X is better than codec Y because of the results of this test, like what was done with the ~128kbit mp3 test, assuming that the results are not particularly interesting except for the killer samples. To this day, there are still people who can't seem to understand that the result of that test was a statistical tie.
Well, that test would, in some people's eyes, have been "uninteresting", because it resulted in a statistical tie. But from another point of view, that was a most interesting fact, in that it showed that some other MP3 encoders had caught up with LAME. In other words, it all depends on what the point of the test is; and the only interest I have is to try to help to clarify that issue, in the midst of everybody making their
points.
The problem with public listening tests is that they are
- very useful on one hand (because they give some important information about encoders)
- and of restricted use on the other hand (because they give limited information about encoders mainly due to sample selection).
Not many people like such a situation. Most people prefer sporting for 'winners'. That's why listening test results are misinterpreted so often.
As sauvage said sample choice has an essential influence on the test (no matter whether @ 96 or 128 kbps).
But it's something we can't get a perfect solution for, so we shouldn't worry too much about it and be content with a set of samples which not obviously gives some bias.
We are lucky to have C.R. Helmrich here, and as FhG AAC doesn't participate in the test (unfortunately) I suggest to let him with his AAC dev experience build up a proposal set of samples (in case he is willing to do so). He is thinking already about such a set.
Of course in the time of this proposal list building new proposals can be made here.
The proposal list should be rather big, and for the final choice a random selection as proposed by IgorC can be used.
Which is why I propose not to randomly select items from a set of "maybe-killer" samples for the 128-kb test, but to do careful pre-screening of listening material and to select only those items for which at least one of the codecs under test is clearly not transparent.
Sounds good to me.
Future discussion of samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=77584) will help to avoid samples which were submitted as issues only for one particular encoder.
Regarding the "will be hard to ABX for beginners": IMHO, this should not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.
It sounds optimal at least for me.
I propose to add 4th rule.
Remove all listeners from analysis who
1. graded the reference lower than 4.5,
2. graded the low anchor higher than all competitors.
3. didn't grade the low anchor.
4. didn't grade any of competitors.
There is no any value of results when all competitors were ranked at 5.0 and low anchor at 2.0.
We are lucky to have C.R. Helmrich here, and as FhG AAC doesn't participate in the test (unfortunately) I suggest to let him with his AAC dev experience build up a proposal set of samples. He is thinking already about such a set.
Yes, he is co-organizer of test.
Thank you Chris for you great contribution.
My opinion is that despite being very democratic, this pool was biased from the start, there is more than 100 voters but there is not more than 100 HA users doing regulary ABXing & posting their findings.
Honestly I should admit you're right.
The gived situation when some well know HA members prefer 96 kbps while other 128 kbps we can test at intermediate bitrate.
Apple TVBR q55 gives ~115 kbps but there isn't such bitrate for CVBR (only 96 or 128).
115 was never tested so nobody can make claims of difficulty
It's an option to drop CVBR.
Opinions are welcomed.
Thank you, IgorC and halb27, for the flowers If some expert listener wants to help me with the pre-selection of items (which would be great), please send me or Igor a message.
Some more comments:
Mike:
I don't understand what you're trying to say, either. When selecting items for the test, I will check, for a given item, the performance of
every codec under test. Since all codecs will be AAC, that sounds fair to me. If I happen to find more critical samples for codec X than for the other codecs (of course I won't tell anyone before the test), doesn't that tell us something about codec X?
IgorC:
Remove all listeners from analysis who 4. didn't grade any of competitors.
Given that at least one competitor will be clearly non-transparent (to me at least), yes, that would make a lot of sense. But I think then we also have to modify rule 1 to:
1. graded the reference lower than 5.0
As I noted earlier, this will greatly reduce the number of listeners left for analysis, but given that we only want experienced ones, so be it.
I prefer not to use an intermediate bitrate because Fraunhofer's encoder doesn't have 112-kb VBR either I'm sure you understand that, after the test, I will want to compare Fraunhofer's encoder against the per-item winner. If not on this forum, then at work.
Note about this poll: What's the problem with non-listeners voting? If someone votes but is not going to participate (or be kicked out before analysis), that's fine! It still shows us which bitrate people are interested in.
Note about the analysis: I'm planning
not to publish an average grade for this test, but only to show the results for each item separately, if on average, all codecs will be tied. That will prevent some people of falsely assigning the word "winner" to a codec. If of course, one codec will be significantly better on average, I will show that. What do you think?
Chris
... But I think then we also have to modify rule 1 to:
1. graded the reference lower than 5.0
As I noted earlier, this will greatly reduce the number of listeners left for analysis, but given that we only want experienced ones, so be it. ...
Good idea IMO.
...I'm planning not to publish an average grade for this test, but only to show the results for each item separately, if on average, all codecs will be tied. That will prevent some people of falsely assigning the word "winner" to a codec. ...
Wonderful.
If it were me I would not consider averaging the results at all. The average itself has no real meaning, it is just a formal way to transform n results into one (loosing important information), in order to satisfy the 'sporting for winners' attitude. Sure I know not many people share my way of seeing this. But practical conclusions for encoder selection can be drawn without an average, focussing on results for samples of individual practical relevance.
All this discussion about usefulness of testing killer samples is coming through important fact that today the focus of lossy encoders has changed.
Nowdays it's not enough anymore for lossy encoders to be "good enough". As HDD space and internet band are enough large that today people see lossy encoders more as replacement of lossless without bitrate trade off.
If the approach of lossy encoders has changed then testing methodology should changed also.
Today I see more sense in testing only difficult samples at least at 128kbps.
Full ACK!
It would be nice if C.R. Helmrich and the Nero developers could disclose which problematic (classes of) samples they have discovered in their testings. We could accumulate those with the ones already known in the forum and then discuss which to include in the final test to not discriminate those encoders for which developers honestly disclosed the most problematic material.
Mike:
I don't understand what you're trying to say, either. When selecting items for the test, I will check, for a given item, the performance of every codec under test. Since all codecs will be AAC, that sounds fair to me.
So you're going to listen to samples and throw them out if they're transparent on some codecs? What if they're transparent because those codecs are simply better?
If I happen to find more critical samples for codec X than for the other codecs (of course I won't tell anyone before the test), doesn't that tell us something about codec X?
Of course, just maybe not what you want it to. If they're randomly selected, it says something about how good the two codecs are. If they're not randomly selected (say you select known problem samples first), then it may just say something about how popular each codec is with people who tend to submit problem samples.
It sounds to me like you intend to do the latter. I think you need to be very careful that you pick samples that are representative of actual audio, and not simply representative of HA AAC problem samples. I suspect the class of artifacts in typical HA user's problem sample collections may not be identical to those likely to be encountered when ripping randomly selected CDs.
So you're going to listen to samples and throw them out if they're transparent on some codecs? What if they're transparent because those codecs are simply better?
The decision of including of sample will be pretty public. http://www.hydrogenaudio.org/forums/index....showtopic=77584 (http://www.hydrogenaudio.org/forums/index.php?showtopic=77584)
Even if somebody wanted to generate biased results it would be hard to do.
I remember that Sauvage78 said that Nero performs very good while Vorbis bad on his set of samples. My findings for the same samples were totally opposite . http://www.hydrogenaudio.org/forums/index....st&p=657459 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=74781&view=findpost&p=657459)
Of course, it's not impossible to cheat but it will be very obvious then and I don't think it's possible with type of discussion we have now about samples.
It sounds optimal at least for me.
I propose to add 4th rule.
Remove all listeners from analysis who
1. graded the reference lower than 4.5,
2. graded the low anchor higher than all competitors.
3. didn't grade the low anchor.
4. didn't grade any of competitors.
There is no any value of results when all competitors were ranked at 5.0 and low anchor at 2.0.
By these rules you are deciding in advance that the result that they are all tied and almost transparent is not desirable.
IMO rule 4 greatly influences the outcome. With it you are saying that you don't want to know that the encoders are transparent to most people even if it might be the case.
Considering the direction where this seems to lead, at the end you can just make a list of problematic samples and list encoders that have problems for each of them.
By these rules you are deciding in advance that the result that they are all tied and almost transparent is not desirable.
Not at all. All encoders can be tied and allmost transparent and still be grade, for example 4.8-4.9 points.
IMO rule 4 greatly influences the outcome. With it you are saying that you don't want to know that the encoders are transparent to most people even if it might be the case.
Considering the direction where this seems to lead, at the end you can just make a list of problematic samples and list encoders that have problems for each of them.
How do we suppose to distinguish the results which are really indicators of transparency or human ignorancy?
The rule 4 is a price to pay to avoid the results of people without any hearing skills or hardware.
Giving the situation:
All this discussion about usefulness of testing killer samples is coming through important fact that today the focus of lossy encoders has changed.
Nowdays it's not enough anymore for lossy encoders to be "good enough". As HDD space and internet band are enough large that today people see lossy encoders more as replacement of lossless without bitrate trade off.
If the approach of lossy encoders has changed then testing methodology should changed also.
Today I see more sense in testing only difficult samples at least at 128kbps.
and
Regarding the "will be hard to ABX for beginners": IMHO, this should not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.
It's nothing wrong with average Joe but deaf one.
Let me add:
IMO rule 4 greatly influences the outcome. With it you are saying that you don't want to know that the encoders are transparent to most people even if it might be the case.
This will certainly not be the case for the experienced listeners in the test. As stated elsewhere, we will only take samples which
at least one encoder is clearly not transparent on (to reasonably trained ears). Moreover, the test only intends to show how experienced listeners judge the quality of the encoders,
not whether or not they are transparent to "most people".
Considering the direction where this seems to lead, at the end you can just make a list of problematic samples and list encoders that have problems for each of them.
I disagree. Such a list wouldn't show how well - or bad - an encoder handles a problematic (i.e. non-transparent) sample in comparison to other codecs, which is what the listening test will show.
Chris
Did you check how many results would be accepted from previous (easier) tests would be accepted if all rules are applied?
IMO what would be more beneficial is to require everybody to do ABX for each sample that he/she wants to grade. That way mistakes by pulling wrong slider would be avoided (on successful ABX original is grayed out).
Also it would be beneficial to create tutorial with each,single,small step that proper test must consist of.
Did you check how many results would be accepted from previous (easier) tests would be accepted if all rules are applied?
Yes, we did.
Also it would be beneficial to create tutorial with each,single,small step that proper test must consist of.
Do you mean a tutorial for the listeners on "what the rules are" and how to proceed before and during the test? That sounds good. Will be done.
Chris
Also it would be beneficial to create tutorial with each,single,small step that proper test must consist of.
Do you mean a tutorial for the listeners on "what the rules are" and how to proceed before and during the test? That sounds good. Will be done.
Yes, that is what I meant. Some really detailed test written by experienced participants in listening tests, maybe also adding some hints how to search for problems in encoded samples and maybe also explaining that quiet environment is needed and such things.
Of course the rules you propose should be included to avoid someone spending hours and that his result doesn't get accepted.