Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 192 kbit/s listening test at SoundExpert (Read 111401 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

192 kbit/s listening test at SoundExpert

EDIT of Jan 9 2007:

Finally 192 kbit/s listening test has finished with the following results:
  • Ogg Vorbis, AAC+, AAC LC and WMA 9.1 showed the best audio quality
  • MP3 and MPC are second
  • AAC LC from iTunes is last due to the bug

See 192 kbit/s page for details. SoundExpert says “thank you” for participation and “sorry” for such long period of testing which was necessary for experiments with reliability parameter of ratings. Next tests will be substantially shorter. Once again - Thank you!

[/color]

EDIT of July 16:

192 kbit/s listening test is opened. Final list of contenders looks as follows:

1. aac VBR@193.3 (NeroRef 1002)
2. aac VBR@197.8 (iTunes 6.0)
3. aac+ CBR@196.8 (Winamp 5.24)
4. mp3 VBR@200.5 (Lame 3.97b2)
5. mpc VBR@193.8 (1.15v)
6. ogg VBR@192.8 (-aoTuV-b4.51)
7. wma 9.1 CBR@198.0 (WMPlayer10)

First results (not reliable though) will appear immediately after first grades is returned by participants. Each rating needs about 300 grades. It might seem that a lot of work is required but fortunately a single testing session (one test file) is short and simple. So it won’t be too hard to test 3-5 test files at a time. You are welcome … and thank you in advance!

To participate in this listening test, please, download and grade a test file.



New 192 kbit/s listening test at SoundExpert is scheduled to start July 15. Here is my proposal of codec contenders:

1.   mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR
2.   aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3.   he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR
4.   wma 9.1 std: WMEncoder [bitrate VBR, 192 kbit/s] 188.0 kbit/s FBR
4a.   wma 9.1 std: WMP10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5.   ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6.   mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR

Not sure about #4. Need your comments and other suggestions. Thank you in advance.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #1
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

Also, not trolling here, but MPC is more or less  obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO. It is just unuseable outside of the PC, and even there its horrible seeking lag makes it a chore.

HE-AAC also, not needed.

About WMA..  1 would be enough.

Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)

192 kbit/s listening test at SoundExpert

Reply #2
Quote
3. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


Is this a typo?  Does HE-AAC even work at 192k?  IIRC most encoders don't allow it above 96kbps.  Even if it technically works, it seems like a pointless test.

192 kbit/s listening test at SoundExpert

Reply #3
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

It gives 173.1 kbit/s for SE test samples. I'm not sure ...

Also, not trolling here, but MPC is more or less  obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO.

Even if it shows better results?

HE-AAC also, not needed. Keep it small

BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.

About WMA..  1 would be enough probably, and one that is supported by all hardware players, which should rule out 9.1 afaik (I dont have a WMA portable) Or was that pro?
Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)

There will be single wma (at least for now). The question is which one - cbr from WMP10 or vbr from WMEncoder?
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #4
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

It gives 173.1 kbit/s for SE test samples. I'm not sure ...

<<< Whoa what are you encoding?  173 is well in the upper range of preset medium (V4), maybe V3

Also, not trolling here, but MPC is more or less  obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO.

Even if it shows better results?

<<< Yes, even then....

HE-AAC also, not needed. Keep it small

BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.

<<< A new sucker is born every day, so they say 
No really, Ivan et al stated many times that it makes no sense at all.

192 kbit/s listening test at SoundExpert

Reply #5
Is this a typo?  Does HE-AAC even work at 192k?  IIRC most encoders don't allow it above 96kbps.  Even if it technically works, it seems like a pointless test.

No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #6
Is this a typo?  Does HE-AAC even work at 192k?  IIRC most encoders don't allow it above 96kbps.  Even if it technically works, it seems like a pointless test.


No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.


Those results are clearly wrong.  At 320k MP2 will sound awesome.  So will MP3 and AAC.  Theres nothing to test at those bitrates, so theres clearly a problem in their methodology.  Looking at their setup, I'd say the most obvious problem is that they ask people to answer which is the most "degraded" sample without telling them what the original sounds like, which sort of calls into question what exactly the results mean. 

My guess would be "nothing at all".

192 kbit/s listening test at SoundExpert

Reply #7
Format selection looks good, even MPC since the last encoder update was 2005, same year as ogg aoTuV 4.51 and Lame 3.97b2.

No test of WMA Pro or iTunes 6.04 AAC?  Might be good to include iTunes AAC as it has not had a test against the recent Nero AAC 1.0.0.2.  Which one would edge out the other this time...

Can't wait for the results on this test and the 128kbps test.

- Gow
Zune 80, Tak -p4 audio library, Lossless=Choice

192 kbit/s listening test at SoundExpert

Reply #8
Suggest you use known codec killer samples because at these bitrates transparency will be the norm.

192 kbit/s listening test at SoundExpert

Reply #9
192 test and you want to use LAME -V1?? For my music, V1 is almost always 250 kbps or more - not even close to 192. I suggest -V2 or even -V3 if you want a fair comparison.

Why not throw the iTunes AAC encoder in the mix as well? It is probably the 3rd most popular encoder out there - behind LAME MP3 and WMA (referring to the greater population, not the HA population)

192 kbit/s listening test at SoundExpert

Reply #10
Just my personal opinion... but the last 128kbps test show that is almost imposible to tell the diference using new codecs even at that bitrate... is just insane that people can tell the diference at 192kbps... only a few "bat ears" people are going to take the challenge...

Personaly for me Ogg Vorbis reach transparency at 128kbps and only if is a dificult sample I can tell the diference with real care using ABX because I probably never will going to hear the diference if I hear the original first and the encoded later... at 160kbps Ogg Vorbis is just over kill (for me), I will never can tell the diference of the original and the encoded one...

I personaly would like to see a test at 96kbps...
JorSol
aoTuVb5 -q4

192 kbit/s listening test at SoundExpert

Reply #11
  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.

192 kbit/s listening test at SoundExpert

Reply #12
Keep with the tests with the two versions of WMA is a good idea,look how is going the improvements in wma.But why not 128 this bitrate is the most used in internet and is a "high compression" when 192 is an ''average compression"
Sorry for my bad english.

192 kbit/s listening test at SoundExpert

Reply #13
  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.
Please show me the results of an ABX test which back up this statement!
If not, please read TOS #8 once more...

Back on topic:
I see no reason to perform an ABX comparison test at a bitrate above 130-150kbps. Cause the 128kbps test show that 90-something % of all test samples are already transparent. So I can't imagine how hard it would be to differ lossy and source at a bitrate 50% above this. Good luck! 


 

192 kbit/s listening test at SoundExpert

Reply #15
Sorry, this is my fault – there is no popular explanation of SoundExpert (SE) testing methodology on the site still (except two boring ones in pro zone). So I have to explain over and over again: SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases. Thus, all SE ratings above 5.00 show amount of perception margin of human auditory system for those tested devices (only codecs now). The absence of reference sample in SE test setup is not essential as the artifacts are clearly audible to more or less extent. The actual aim of SE listening session is to grade the annoyance of those artifacts.

Since 2001 a fixed set of reference (critical) sound samples (from SQAM disc mostly) is used for SE listening tests. All average bit rates are calculated on basis of these samples and called “file based bit rate” (FBR). As the sample set consist of different sound pieces including solo instruments, voices and speech the resulting bit rates (in VBR mode) are slightly lower than in case of pop/rock music (quite close to the bit rates for classical music though).

For now I’m going to add only one coder per format. For AAC I would like to test (and promote) Nero encoder because it is new, good and free. For WMA I tend to choose 192 cbr variant from WMP10 because it represents more real life scenario than WMEncoder with 192 bitrate VBR. The more so, as there is no problem to add other contenders later (say, in the autumn) including new Lame, Ogg, iTunes, Winamp and Vista ones.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #16
I'm not sure that test is very useful either.

It rates transparent encoders by how far their distortions/artifacts are below the masking curve. What matters is how reliably they reach transparency, not how far 'beyond' they get I'd say.

192 kbit/s listening test at SoundExpert

Reply #17
Quote
SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases.


What is the purpose of this? Isn't a listening test supposed to test how well a codec performs under normal conditions?

192 kbit/s listening test at SoundExpert

Reply #18
In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear. Also don’t forget that any listening test is performed only with finite number (10-20 at most) of test samples and then the results are generalized to all music. That’s why the more perception margin - the better. Most people understand this intuitively and use slightly higher bitrates than usually claimed to be transparent (128 – according to last test).
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #19
Quote
In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear.


Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons.  I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

192 kbit/s listening test at SoundExpert

Reply #20
Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons.  I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution 
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #21
  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.


Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?

192 kbit/s listening test at SoundExpert

Reply #22

  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.


Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?


Yes, I did. But I though it should better to add an interesting codec.

192 kbit/s listening test at SoundExpert

Reply #23
Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons.  I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution 


Ok, but have you shown that the results are relevent to anything?  Because saying that 320k AAC+ SBR is the best encoder REALLY makes me doubt the relevence of the results. 

In addition, unless I am missing something, you have not addressed the other objection I raised to the methodology.

192 kbit/s listening test at SoundExpert

Reply #24
Methedology aside, I would like to add my input on the formats being tested.

Since Microsoft is moving to WMP10, I suggest sticking with it and only using one WMA setting with WMP10.  I would also suggest throwing in iTunes AAC.  You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc.  I really see no need for testing HE-AAC at such high bitrates especially when there is no hardware support for HE-AAC.  Granted, the hardware support for MPC is zero to none but it is a more "establilshed" format that is on its last leg.  The only reason I could see testing HE-AAC would be to show that Nero AAC and iTunes AAC (which are LC-AAC encoders) can perform equally well and that HE-AAC encoding above 64kbps is just not needed.

As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low.  For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps.  -V 1 was just too high as it would produce average bitrates of 270kbps with metal.