Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.

Poll

96 or 128 kbps public test?

~96 kbps test on "average music" samples with little amount of difficult samples
[ 55 ] (39.3%)
~128 kbps test with a higher amount of difficult samples
[ 85 ] (60.7%)

Total Members Voted: 160

Topic: 96 or 128 kbps public AAC test (Read 40119 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

96 or 128 kbps public AAC test

Reply #50 –
Full ACK!

It would be nice if C.R. Helmrich and the Nero developers could disclose which problematic (classes of) samples they have discovered in their testings. We could accumulate those with the ones already known in the forum and then discuss which to include in the final test to not discriminate those encoders for which developers honestly disclosed the most problematic material.

96 or 128 kbps public AAC test

Reply #51 –
Quote from: C.R.Helmrich on
Mike:
I don't understand what you're trying to say, either. When selecting items for the test, I will check, for a given item, the performance of every codec under test. Since all codecs will be AAC, that sounds fair to me.


So you're going to listen to samples and throw them out if they're transparent on some codecs?  What if they're transparent because those codecs are simply better?

Quote from: C.R.Helmrich on
If I happen to find more critical samples for codec X than for the other codecs (of course I won't tell anyone before the test), doesn't that tell us something about codec X?


Of course, just maybe not what you want it to.  If they're randomly selected, it says something about how good the two codecs are.  If they're not randomly selected (say you select known problem samples first), then it may just say something about how popular each codec is with people who tend to submit problem samples. 

It sounds to me like you intend to do the latter.  I think you need to be very careful that you pick samples that are representative of actual audio, and not simply representative of HA AAC problem samples.  I suspect the class of artifacts in typical HA user's problem sample collections may not be identical to those likely to be encountered when ripping randomly selected CDs.

96 or 128 kbps public AAC test

Reply #53 –
Even if somebody wanted to generate biased results it would be hard to do.

I remember that Sauvage78 said that Nero performs very good while Vorbis bad on his set of samples. My findings for the same samples were totally opposite . http://www.hydrogenaudio.org/forums/index....st&p=657459

Of course, it's not impossible to cheat but it will be very obvious then and I don't think it's  possible with type of discussion we have now about samples.

96 or 128 kbps public AAC test

Reply #54 –
Quote from: IgorC on
It sounds optimal at least for me.
I propose to add 4th rule.
Code: [Select]
Remove all listeners from analysis who
1. graded the reference lower than 4.5,
2. graded the low anchor higher than all competitors.
3. didn't grade the low anchor.
4. didn't grade any of competitors.

There is no any value of results when all competitors were ranked at 5.0 and low anchor at 2.0.

By these rules you are deciding in advance that the result that they are all tied and almost transparent is not desirable.
IMO rule 4 greatly influences the outcome. With it you are saying that you don't want to know that the encoders are transparent to most people even if it might be the case.
Considering the direction where this seems to lead, at the end you can just make a list of problematic samples and list encoders that have problems for each of them.

96 or 128 kbps public AAC test

Reply #55 –
Quote from: muaddib on
By these rules you are deciding in advance that the result that they are all tied and almost transparent is not desirable.

Not at all. All encoders can be tied and allmost transparent and still be grade, for example 4.8-4.9 points.

Quote from: muaddib on
IMO rule 4 greatly influences the outcome. With it you are saying that you don't want to know that the encoders are transparent to most people even if it might be the case.
Considering the direction where this seems to lead, at the end you can just make a list of problematic samples and list encoders that have problems for each of them.

How do we suppose to distinguish the results which are really indicators of transparency or human ignorancy?

The rule 4 is a price to pay to avoid the results of people without any hearing skills or hardware.

Giving the situation:
Quote from: IgorC on
All this discussion about usefulness of testing killer samples is coming through important fact that today the focus of lossy encoders has changed.
Nowdays it's not enough anymore for lossy encoders to be "good enough". As HDD space and internet band are enough large  that today people see lossy encoders more as replacement of lossless without bitrate trade off.

If the approach of lossy encoders has changed then testing methodology should changed also.
Today I see more sense in testing only difficult samples at least at 128kbps.


and

Quote from: C.R.Helmrich on
Regarding the "will be hard to ABX for beginners": IMHO, this should not be a test for untrained listeners. No matter which bitrate! This is exactly the mixture which gives inconclusive results: experienced and unexperienced listeners. At such high bitrates, only the former should participate. This is HA, so I expect quite a number of members belonging to that group.


It's nothing wrong with average Joe but deaf one.

96 or 128 kbps public AAC test

Reply #56 –
Let me add:
Quote from: muaddib on
IMO rule 4 greatly influences the outcome. With it you are saying that you don't want to know that the encoders are transparent to most people even if it might be the case.

This will certainly not be the case for the experienced listeners in the test. As stated elsewhere, we will only take samples which at least one encoder is clearly not transparent on (to reasonably trained ears). Moreover, the test only intends to show how experienced listeners judge the quality of the encoders, not whether or not they are transparent to "most people".

Quote
Considering the direction where this seems to lead, at the end you can just make a list of problematic samples and list encoders that have problems for each of them.

I disagree. Such a list wouldn't show how well - or bad - an encoder handles a problematic (i.e. non-transparent) sample in comparison to other codecs, which is what the listening test will show.

Chris
If I don't reply to your reply, it means I agree with you.

96 or 128 kbps public AAC test

Reply #57 –
Did you check how many results would be accepted from previous (easier) tests would be accepted if all rules are applied?

IMO what would be more beneficial is to require everybody to do ABX for each sample that he/she wants to grade. That way mistakes by pulling wrong slider would be avoided (on successful ABX original is grayed out).
Also it would be beneficial to create tutorial with each,single,small step that proper test must consist of.

96 or 128 kbps public AAC test

Reply #58 –
Quote from: muaddib on
Did you check how many results would be accepted from previous (easier) tests would be accepted if all rules are applied?

Yes, we did.

96 or 128 kbps public AAC test

Reply #59 –
Quote from: muaddib on
Also it would be beneficial to create tutorial with each,single,small step that proper test must consist of.

Do you mean a tutorial for the listeners on "what the rules are" and how to proceed before and during the test? That sounds good. Will be done.

Chris
If I don't reply to your reply, it means I agree with you.
 

96 or 128 kbps public AAC test

Reply #60 –
Quote from: C.R.Helmrich on
Quote from: muaddib on
Also it would be beneficial to create tutorial with each,single,small step that proper test must consist of.
Do you mean a tutorial for the listeners on "what the rules are" and how to proceed before and during the test? That sounds good. Will be done.

Yes, that is what I meant. Some really detailed test written by experienced participants in listening tests, maybe also adding some hints how to search for problems in encoded samples and maybe also explaining that quiet environment is needed and such things.
Of course the rules you propose should be included to avoid someone spending hours and that his result doesn't get accepted.