HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Serge Smirnoff on 2006-07-01 22:23:04

Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-01 22:23:04
EDIT of Jan 9 2007:

Finally 192 kbit/s listening test has finished with the following results:

See 192 kbit/s page (http://www.soundexpert.info/coders192.jsp) for details. SoundExpert says “thank you” for participation and “sorry” for such long period of testing which was necessary for experiments with reliability parameter of ratings. Next tests will be substantially shorter. Once again - Thank you!

[/color]

EDIT of July 16:

192 kbit/s listening test (http://www.soundexpert.info/coders192.jsp) is opened. Final list of contenders looks as follows:

1. aac VBR@193.3 (NeroRef 1002)
2. aac VBR@197.8 (iTunes 6.0)
3. aac+ CBR@196.8 (Winamp 5.24)
4. mp3 VBR@200.5 (Lame 3.97b2)
5. mpc VBR@193.8 (1.15v)
6. ogg VBR@192.8 (-aoTuV-b4.51)
7. wma 9.1 CBR@198.0 (WMPlayer10)

First results (not reliable though) will appear immediately after first grades is returned by participants. Each rating needs about 300 grades. It might seem that a lot of work is required but fortunately a single testing session (one test file) is short and simple. So it won’t be too hard to test 3-5 test files at a time. You are welcome … and thank you in advance!

To participate in this listening test, please, download and grade a test file (http://ftp://www.soundexpert.info/).


New 192 kbit/s listening test at SoundExpert is scheduled to start July 15. Here is my proposal of codec contenders:

1.   mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR
2.   aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3.   he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR
4.   wma 9.1 std: WMEncoder [bitrate VBR, 192 kbit/s] 188.0 kbit/s FBR
4a.   wma 9.1 std: WMP10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5.   ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6.   mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR

Not sure about #4. Need your comments and other suggestions. Thank you in advance.
Title: 192 kbit/s listening test at SoundExpert
Post by: sony666 on 2006-07-01 23:26:58
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

Also, not trolling here, but MPC is more or less  obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO. It is just unuseable outside of the PC, and even there its horrible seeking lag makes it a chore.

HE-AAC also, not needed.

About WMA..  1 would be enough.

Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)
Title: 192 kbit/s listening test at SoundExpert
Post by: saratoga on 2006-07-01 23:42:53
Quote
3. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


Is this a typo?  Does HE-AAC even work at 192k?  IIRC most encoders don't allow it above 96kbps.  Even if it technically works, it seems like a pointless test.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-01 23:46:30
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

It gives 173.1 kbit/s for SE test samples. I'm not sure ...

Also, not trolling here, but MPC is more or less  obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO.

Even if it shows better results?

HE-AAC also, not needed. Keep it small

BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.

About WMA..  1 would be enough probably, and one that is supported by all hardware players, which should rule out 9.1 afaik (I dont have a WMA portable) Or was that pro?
Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)

There will be single wma (at least for now). The question is which one - cbr from WMP10 or vbr from WMEncoder?
Title: 192 kbit/s listening test at SoundExpert
Post by: sony666 on 2006-07-01 23:50:49
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

It gives 173.1 kbit/s for SE test samples. I'm not sure ...

<<< Whoa what are you encoding?  173 is well in the upper range of preset medium (V4), maybe V3

Also, not trolling here, but MPC is more or less  obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO.

Even if it shows better results?

<<< Yes, even then....

HE-AAC also, not needed. Keep it small

BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.

<<< A new sucker is born every day, so they say 
No really, Ivan et al stated many times that it makes no sense at all.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-01 23:55:37
Is this a typo?  Does HE-AAC even work at 192k?  IIRC most encoders don't allow it above 96kbps.  Even if it technically works, it seems like a pointless test.

No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s (http://www.soundexpert.info/coders320.jsp) and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.
Title: 192 kbit/s listening test at SoundExpert
Post by: saratoga on 2006-07-02 00:25:50
Is this a typo?  Does HE-AAC even work at 192k?  IIRC most encoders don't allow it above 96kbps.  Even if it technically works, it seems like a pointless test.


No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s (http://www.soundexpert.info/coders320.jsp) and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.


Those results are clearly wrong.  At 320k MP2 will sound awesome.  So will MP3 and AAC.  Theres nothing to test at those bitrates, so theres clearly a problem in their methodology.  Looking at their setup, I'd say the most obvious problem is that they ask people to answer which is the most "degraded" sample without telling them what the original sounds like, which sort of calls into question what exactly the results mean. 

My guess would be "nothing at all".
Title: 192 kbit/s listening test at SoundExpert
Post by: Gow on 2006-07-02 01:14:04
Format selection looks good, even MPC since the last encoder update was 2005, same year as ogg aoTuV 4.51 and Lame 3.97b2.

No test of WMA Pro or iTunes 6.04 AAC?  Might be good to include iTunes AAC as it has not had a test against the recent Nero AAC 1.0.0.2.  Which one would edge out the other this time...

Can't wait for the results on this test and the 128kbps test.

- Gow
Title: 192 kbit/s listening test at SoundExpert
Post by: kennedyb4 on 2006-07-02 02:51:53
Suggest you use known codec killer samples because at these bitrates transparency will be the norm.
Title: 192 kbit/s listening test at SoundExpert
Post by: ezra2323 on 2006-07-02 03:43:22
192 test and you want to use LAME -V1?? For my music, V1 is almost always 250 kbps or more - not even close to 192. I suggest -V2 or even -V3 if you want a fair comparison.

Why not throw the iTunes AAC encoder in the mix as well? It is probably the 3rd most popular encoder out there - behind LAME MP3 and WMA (referring to the greater population, not the HA population)
Title: 192 kbit/s listening test at SoundExpert
Post by: jorsol on 2006-07-02 03:49:04
Just my personal opinion... but the last 128kbps test show that is almost imposible to tell the diference using new codecs even at that bitrate... is just insane that people can tell the diference at 192kbps... only a few "bat ears" people are going to take the challenge...

Personaly for me Ogg Vorbis reach transparency at 128kbps and only if is a dificult sample I can tell the diference with real care using ABX because I probably never will going to hear the diference if I hear the original first and the encoded later... at 160kbps Ogg Vorbis is just over kill (for me), I will never can tell the diference of the original and the encoded one...

I personaly would like to see a test at 96kbps...
Title: 192 kbit/s listening test at SoundExpert
Post by: Jillian on 2006-07-02 04:34:08
  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.
Title: 192 kbit/s listening test at SoundExpert
Post by: Diow on 2006-07-02 04:45:49
Keep with the tests with the two versions of WMA is a good idea,look how is going the improvements in wma.But why not 128 this bitrate is the most used in internet and is a "high compression" when 192 is an ''average compression"
Title: 192 kbit/s listening test at SoundExpert
Post by: krmathis on 2006-07-02 05:03:30
  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.
Please show me the results of an ABX test which back up this statement!
If not, please read TOS #8 (http://www.hydrogenaudio.org/forums/index.php?showtopic=3974#entry149481) once more...

Back on topic:
I see no reason to perform an ABX comparison test at a bitrate above 130-150kbps. Cause the 128kbps test show that 90-something % of all test samples are already transparent. So I can't imagine how hard it would be to differ lossy and source at a bitrate 50% above this. Good luck! 
Title: 192 kbit/s listening test at SoundExpert
Post by: Jillian on 2006-07-02 05:27:44

  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.
Please show me the results of an ABX test which back up this statement!
If not, please read TOS #8 (http://www.hydrogenaudio.org/forums/index.php?showtopic=3974#entry149481) once more...




Oh I'm sorry about that.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-02 09:11:19
Sorry, this is my fault – there is no popular explanation of SoundExpert (SE) testing methodology on the site still (except two boring ones in pro zone (http://www.soundexpert.info/prozone.htm)). So I have to explain over and over again: SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases. Thus, all SE ratings above 5.00 show amount of perception margin of human auditory system for those tested devices (only codecs now). The absence of reference sample in SE test setup is not essential as the artifacts are clearly audible to more or less extent. The actual aim of SE listening session is to grade the annoyance of those artifacts.

Since 2001 a fixed set of reference (critical) sound samples (from SQAM disc mostly) is used for SE listening tests. All average bit rates are calculated on basis of these samples and called “file based bit rate” (FBR). As the sample set consist of different sound pieces including solo instruments, voices and speech the resulting bit rates (in VBR mode) are slightly lower than in case of pop/rock music (quite close to the bit rates for classical music though).

For now I’m going to add only one coder per format. For AAC I would like to test (and promote) Nero encoder because it is new, good and free. For WMA I tend to choose 192 cbr variant from WMP10 because it represents more real life scenario than WMEncoder with 192 bitrate VBR. The more so, as there is no problem to add other contenders later (say, in the autumn) including new Lame, Ogg, iTunes, Winamp and Vista ones.
Title: 192 kbit/s listening test at SoundExpert
Post by: gaekwad2 on 2006-07-02 09:47:46
I'm not sure that test is very useful either.

It rates transparent encoders by how far their distortions/artifacts are below the masking curve. What matters is how reliably they reach transparency, not how far 'beyond' they get I'd say.
Title: 192 kbit/s listening test at SoundExpert
Post by: LaserSokrates on 2006-07-02 10:02:41
Quote
SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases.


What is the purpose of this? Isn't a listening test supposed to test how well a codec performs under normal conditions?
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-02 10:08:38
In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear. Also don’t forget that any listening test is performed only with finite number (10-20 at most) of test samples and then the results are generalized to all music. That’s why the more perception margin - the better. Most people understand this intuitively and use slightly higher bitrates than usually claimed to be transparent (128 – according to last test).
Title: 192 kbit/s listening test at SoundExpert
Post by: Ivan Dimkovic on 2006-07-02 12:12:10
Quote
In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear.


Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons.  I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-02 12:54:38
Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons.  I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution 
Title: 192 kbit/s listening test at SoundExpert
Post by: Sebastian Mares on 2006-07-02 13:48:32
  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.


Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?
Title: 192 kbit/s listening test at SoundExpert
Post by: Jillian on 2006-07-02 14:27:18

  I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test.


Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?


Yes, I did. But I though it should better to add an interesting codec.
Title: 192 kbit/s listening test at SoundExpert
Post by: saratoga on 2006-07-03 01:23:43
Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons.  I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution 


Ok, but have you shown that the results are relevent to anything?  Because saying that 320k AAC+ SBR is the best encoder REALLY makes me doubt the relevence of the results. 

In addition, unless I am missing something, you have not addressed the other objection I raised to the methodology.
Title: 192 kbit/s listening test at SoundExpert
Post by: kornchild2002 on 2006-07-03 02:38:50
Methedology aside, I would like to add my input on the formats being tested.

Since Microsoft is moving to WMP10, I suggest sticking with it and only using one WMA setting with WMP10.  I would also suggest throwing in iTunes AAC.  You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc.  I really see no need for testing HE-AAC at such high bitrates especially when there is no hardware support for HE-AAC.  Granted, the hardware support for MPC is zero to none but it is a more "establilshed" format that is on its last leg.  The only reason I could see testing HE-AAC would be to show that Nero AAC and iTunes AAC (which are LC-AAC encoders) can perform equally well and that HE-AAC encoding above 64kbps is just not needed.

As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low.  For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps.  -V 1 was just too high as it would produce average bitrates of 270kbps with metal.
Title: 192 kbit/s listening test at SoundExpert
Post by: bubka on 2006-07-03 03:34:18
although this is probably the test i would be must interested in, i doubt you will get many different results that are outside the margin or error
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-03 09:34:21
Ok, but have you shown that the results are relevent to anything?  Because saying that 320k AAC+ SBR is the best encoder REALLY makes me doubt the relevence of the results.

In order to prove relevance of the methodology a big thoroughly organized standard listening test has to be performed. Sorry, for now I haven’t such possibility. All I can do is to test as many different codecs as possible and to watch how SE ratings correlate with results of other listening tests. That’s what I’m doing.

Concerning CT High Bitrate HE-AAC Encoder @320 see this HA discussion (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=43656&view=findpost&p=390921). I will add this codec to 192 and other SE groups to investigate the situation further.

In addition, unless I am missing something, you have not addressed the other objection I raised to the methodology.

If you mean absence of reference sample see my post #16, end of first paragraph.

As usually discussion on codec contenders turned into discussion on methodology flaws. May be it’s a good idea to open special topic, say, “SoundExpert testing methodology criticism”?

... I would also suggest throwing in iTunes AAC.  You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc.


Now I plan to add only one codec per format. Others could be added later. What AAC coder you prefer to include in this case – Free Nero or iTunes. I would prefer Nero. I’m pretty sure that Apple’s codec will have lower score just because it produces very audible artifact with one of SE test sample (glockenspiel) (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41278&view=findpost&p=365030) – a bug definitely.

As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low.  For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps.  -V 1 was just too high as it would produce average bitrates of 270kbps with metal.

But -V2 produces only 173.1 kbit/s on SE samples. Other contenders are 191-198 kbit/s.  Will it be honest competition?
Title: 192 kbit/s listening test at SoundExpert
Post by: LaserSokrates on 2006-07-03 10:23:13
Quote
But -V2 produces only 173.1 kbit/s on SE samples.

Unless these ones are killersamples, -V2 is transparent to me (didn't ABX lately, but I had problems with V6 and V5, so I just assume this). When using SARTAMP, all encoded samples might have audible artefacts.
BTW, I still don't get the pourpose of SARTAMP.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-03 11:09:02
BTW, I still don't get the pourpose of SARTAMP.

Without SARTAMP all codecs above 160 kbit/s seem to be equal. This is not so indeed and the technology helps to rank them above 160 where standard listening tests become unreliable. In some circumstances greater quality margin is useful. In any case it is useful to preserve as much sound quality as possible at given bitrate.
Title: 192 kbit/s listening test at SoundExpert
Post by: user on 2006-07-03 12:07:05
Those people, who don#t understand the way soundexpert tests work, should keep quiet and just watch at the moment, and make a criticismn topic besides this topic.

As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.

So, see the soundexpert tests as additional special tool, to collect informations about the encoders.

btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...
Title: 192 kbit/s listening test at SoundExpert
Post by: guruboolez on 2006-07-03 12:29:11
As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.

...like HE-AAC absolute superiority at 128 and 320 kbps.
http://www.soundexpert.info/coders128.jsp (http://www.soundexpert.info/coders128.jsp)
http://www.soundexpert.info/coders320.jsp (http://www.soundexpert.info/coders320.jsp)

AFAIK, it goes against all studies, listening tests and developers' experience.

Quote
btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones...

Who the hell are you to give such stupid advice to people? Modern encoders are most often very good. Collective listening tests performed by Roberto, Sebastian and Gabriel have proved it four consecutive time. I'm a trained listener; I can ABX lossy encoders at ~200 kbps but I must also admit that ~128 VBR encoders are often damn good and even transparent, even on critical listening conditions. Just count the 5.0 mark on this test (http://www.hydrogenaudio.org/forums/index.php?showtopic=38792&hl=). And several people here have confirmed how difficult ABXing has become at ~130 kbps and how satisfied they are by these encodings. I personaly find your opinion as very haughty - especially for someone who hadn't brought any ABX test at higher bitrate on this board in the four last years (or did I miss them?).
Title: 192 kbit/s listening test at SoundExpert
Post by: SebastianG on 2006-07-03 13:14:53
Those people, who don#t understand the way soundexpert tests work, should keep quiet and just watch at the moment, and make a criticismn topic besides this topic.

Sorry, but I consider criticism to be very much on-topic. You don't have to worry about me repeating "issues", though -- I've already said what I had to say. But let me link to a thread where Ivan, Garf and myself are questioning this testing methodology here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=43656&view=findpost&p=382955).

After pressing "preview" and seeing that guruboolez already commented on your last paragraph (I totally agree) I'm not going to say anything else to that one.

Sebastian
Title: 192 kbit/s listening test at SoundExpert
Post by: jimmy69 on 2006-07-03 14:20:50
I'm new to this forum.  I don't understand why he-aac and mpc are going to be in the test when the only thing they work on is a computer.  It would be good if the itunes aac encoder is there because its proven to be a compedetive codec in the past and because I use it.  Why don't you use the wma encoder from WMP11.  Finally what is the difference with LAME V1, V2, and V3.
Title: 192 kbit/s listening test at SoundExpert
Post by: kornchild2002 on 2006-07-03 14:51:53
... I would also suggest throwing in iTunes AAC.  You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc.


Now I plan to add only one codec per format. Others could be added later. What AAC coder you prefer to include in this case – Free Nero or iTunes. I would prefer Nero. I’m pretty sure that Apple’s codec will have lower score just because it produces very audible artifact with one of SE test sample (glockenspiel) (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41278&view=findpost&p=365030) – a bug definitely.

As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low.  For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps.  -V 1 was just too high as it would produce average bitrates of 270kbps with metal.

But -V2 produces only 173.1 kbit/s on SE samples. Other contenders are 191-198 kbit/s.  Will it be honest competition?



The only reason I suggest throwing in iTunes AAC is that it has yet to be compared against the new Nero AAC encoder.  Yes, I think it is pertinant to test the new Nero AAC encoder since I don't think anyone has tested it yet but I think it would be worth while to stack it up against a very popular AAC encoder, the iTunes AAC encoder.  I would much rather see Nero AAC (LC-AAC) and iTunes AAC (LC-AAC) and take out the WinAmp HE-AAC encoder especially since hardware support for HE-AAC is zero to none.  Maybe HE-AAC could be added later on and keep iTunes AAC and Nero AAC in the initial runs.

As for the -V 2 having a ~20kbps lower bitrate, this might not be fair but I think it should still be tested especially since so many people use -V 2 over -V 1.  Again, my claim on thhis might not hold any merit as I have had little experience on setting up proper ABX tests.

I think I am going to continue to push iTunes AAC though, that way it can be directly compared with Nero AAC just to see which one is better.  Besides, iTunes AAC has almost always retained a very slight lead (not statistically higher but still) over Nero AAC.  It would be nice to see the tides turn towards Nero AAC if those are the true results of the test.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-04 09:08:29
Welcome back to the topic. I think, instead of fitting codecs and settings to SE groups, better to fit the groups to widely used codecs/settings. Frankly talking, those groups are created for user convenience only. SE system allows to group codecs according to different criteria. For example: CBR/VBR/ABR or according to quality ratings or actual bitrate as for now. Present division by bitrate is quite reasonable but it is not a strict guideline for testing. While theoretical efficiency of compression algorithms is interesting mostly for developers and advanced users, practical efficiency of widely used codecs and settings is interesting for majority of average users. Taking all this into account I suggest revised list of contenders:

1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Sorry, I have to include #7 for research purposes.

Why don't you use the wma encoder from WMP11.

... because it is beta (not finished/tuned yet) and because it’s illegal to publish results of testing according to WMPlayer 11 End User License Agreement (EULA).

Finally what is the difference with LAME V1, V2, and V3.

They are quality settings of Lame encoder which affect resulting audio quality and actual size of output mp3 files.
Title: 192 kbit/s listening test at SoundExpert
Post by: sTisTi on 2006-07-04 12:30:58
1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR

Since everyone is now using --preset fast standard (aka -V2 -vbr-new) instead of the vbr-old setting, it would probably make more sense to test the fast version. As a historical comparison (and for die-hard 3.90.3 fans  ), additionally testing the legacy Lame 3.90.3 at the --alt-preset standard setting might be an option if you still have room for another contender.
Title: 192 kbit/s listening test at SoundExpert
Post by: Silversight on 2006-07-04 13:49:53
Nearly all codecs are transparent to most people at that bitrate, so artificially amplifying artifacts that are inaudible in normal listening setups only serves to determine which codec is "best" to transcode from, and transcoding from lossy sources shouldn't be done in the first place.

Lossy codecs aim at transparent perceived sound at the lowest possible bitrate. As gaekwad2 already said, this methodology rates how far a codec goes "over the top". When two codecs produce transparent results at a certain bitrate, it only shows which one might sound better at lower bitrates.

Since the test is most probably done regardless of this criticism anyway, I suggest waiting for aoTuV Vorbis Beta 5.
Title: 192 kbit/s listening test at SoundExpert
Post by: [JAZ] on 2006-07-04 14:55:04
@ Silversight and everyone else:

Stop talking about the test procedure. SoundExpert has been doing tests like this for some time now. It's not necessary to take part or accept the results themselves. If it is really neccesary, i would ask for separating this conversation to another thread.

@ Serge :

If you talk about using standard settings for LAME, shouldn't this be applicable to Nero, Vorbis and Musepack aswell? and if you use 3.97b2, it would be better to use --vbr-new, as already noted.

Edit:
suggestions:
vorbis -q6.25 or -q6
nero : -q0.625 or -q0.6
mpc : -q5.5  (?)
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-04 15:40:29
Quote
' date='Jul 4 2006, 17:55' post='408710']... and if you use 3.97b2, it would be better to use --vbr-new, as already noted.

Ok. Done.

1. mp3: Lame 3.97b2 [--preset fast standard] 173.1 kbit/s FBR

Quote
' date='Jul 4 2006, 17:55' post='408710']If you talk about using standard settings for LAME, shouldn't this be applicable to Nero, Vorbis and Musepack aswell?

suggestions:
vorbis -q6.25 or -q6
nero : -q0.625 or -q0.6
mpc : -q5.5  (?)


The only reason of choosing “unusual” quality settings was to achieve target bitrate as close as possible. Also AFAIC these codecs are equally tuned for all q-parameters. But if there are some “popular” values I’m ready to make changes.
Title: 192 kbit/s listening test at SoundExpert
Post by: Gow on 2006-07-04 17:33:12
1. mp3: Lame 3.97b2 [--preset fast standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


Perfect.  Varied selection, iTunes vs. Nero, etc.

Will be interesting to see the results.

- Gow
Title: 192 kbit/s listening test at SoundExpert
Post by: guruboolez on 2006-07-05 07:17:44
Quote
1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


min = 177.7 kbps & max = 198 kbps [iTunes, WMA]. Difference is higher than 10% (and 20 kbps).
Moreover, it seems that you decided to use very precise quality level (0.626, 6.37 & 5.49), probably to match a precise bitrate - but a different criterion (popularity) is used for LAME. This second criterion leads to considerably lower the bitrate and therfore handicap the format. It's likely that for most people there wouldn't be any perceptual difference between V1 and V2 in normal listening conditions but isn't your methodology supposed to amplify the distortions?

I'd use the same criterion for all encoder and try either to follow "popular" settings" or to match the same approximate bitrate for all. You said that:
“1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR”
Then use V1. The deviation would be much lower (min = 191 kbps and max 200 kbps => ~4...5%) and even perfectible (by increasing MPC bitrate).
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-05 09:04:40
I'd use the same criterion for all encoder and try either to follow "popular" settings" or to match the same approximate bitrate for all. You said that:
“1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR”
Then use V1. The deviation would be much lower (min = 191 kbps and max 200 kbps => ~4...5%) and even perfectible (by increasing MPC bitrate).

I rounded q-parameters for Nero, Ogg, MPC:

1.  mp3: Lame 3.97b2 [--preset fast standard or –V1] 173.1 or 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.
Title: 192 kbit/s listening test at SoundExpert
Post by: guruboolez on 2006-07-05 09:46:46
Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.

Coherency is the most important point IMO. If a majority of people would rather see HE-AAC at 64 kbps instead of LC at 192, would you include the first one among all ~200 kbps encodings?
-V2 is clearly too low bitrate according to your methodology; V1 is marginaly higher. The second one is the best choice if you want to keep the comparison fair.

Other solution: try to build a bitrate table based on various albums. Values may significantly change (and would correspond to a real/daily usage of codecs)
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-05 16:46:24
Coherency is the most important point IMO. If a majority of people would rather see HE-AAC at 64 kbps instead of LC at 192, would you include the first one among all ~200 kbps encodings?

Why not? Especially if they are comparable in quality. I know at least one ISO listening test with different codecs at different bitrates. So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.

Other solution: try to build a bitrate table based on various albums. Values may significantly change (and would correspond to a real/daily usage of codecs)

Unfortunately this approach is not ideal as well because actual figures will depend on type of albums chosen for bitrate calculations. Add more classical music albums and you’ll get lower values, more hard/metal – higher values. Such figures will be “just for reference” in any case like SE ones.
Title: 192 kbit/s listening test at SoundExpert
Post by: stephanV on 2006-07-05 18:09:43
Why not? Especially if they are comparable in quality. I know at least one ISO listening test with different codecs at different bitrates. So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.

You call it a '192 kbps' test, so you should choose the setting that is closest to that bitrate. If -V1 --vbr-new is closer then you should choose that one. You seem to have chosen the setting for every other codec closest to 192 kbps, so why not for LAME?

Quote
Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.

No it can't. IMO 173.1 is neither close to 160 (8% diff) or 192 (10% diff), both differences are too big. If you want to call your test '~ xxx kbps-test' thats fine, but at least let all contenders respect that bitrate. Otherwise the name of your test is completely irrelevant to what you are actually testing.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-05 19:04:18
Ok. You win, guys:

mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR

The next test will be heterogeneous for sure and will include most popular codecs/settings. I have to think out a proper name for it or make it completely nameless in order to simplify discussion.

This set seems to be final now:

1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR
Title: 192 kbit/s listening test at SoundExpert
Post by: saratoga on 2006-07-05 19:34:51
Quote
As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.


I seriously doubt you've looked closely at those results.  Do so now, and then try and find me a test showing 128k AAC-HE beating all other codecs.  Unless I've missed something, you cannot.  The ranking of AAC-LC and Ogg encoders also looks contrary to previous results.

Quote
So, see the soundexpert tests as additional special tool, to collect informations about the encoders.


Yes, but what information?  Given that they do not correlate well with double blind listening tests, what exactly is being measured?

No one seems to be able to answer that.

Quote
btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...


I'm guessing this was meant to be a clever way to stay inside TOS#8 without having to support your assertion, or really even provide an arguement beyon "I'm right, you're wrong".
Title: 192 kbit/s listening test at SoundExpert
Post by: Alex B on 2006-07-05 20:29:56
1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)
Title: 192 kbit/s listening test at SoundExpert
Post by: spoon on 2006-07-05 21:16:17
Why is WMA the only codec using CBR? why not use VBR on WMA as well?
Title: 192 kbit/s listening test at SoundExpert
Post by: Alex B on 2006-07-05 21:29:50
Out of curiosity, I tested LAME, Vorbis, Nero and Musepack bitrates with the same set of 25 test tracks that I have used previously [1] (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=37011&view=findpost&p=328558) [2] (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=38955&view=findpost&p=344516).

At first I encoded the files with LAME 3.97b2 @ -V2 --vbr-new. The average bitrate was 196 kbps.

Then I tweaked the other codecs to produce exactly the same average bitrate. After a few encoding rounds I settled with these final Q settings :

aoTuV 4.51b : -q 6.20  (196 kbps)
Nero (2006-05-26): -q 0.552  (196 kbps)
Musepack 1.15v: --quality 5.75  (196 kbps)

I think that Q settings like these would be more comparable and fairer.

I used Mr QuestionMan for bitrate measurements. The MrQ reports are available here: mrq.zip (http://kotisivu.mtv3.fi/alexb/ha/mrq.zip)
Title: 192 kbit/s listening test at SoundExpert
Post by: beto on 2006-07-06 02:16:29
Quote


As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.


I seriously doubt you've looked closely at those results.  Do so now, and then try and find me a test showing 128k AAC-HE beating all other codecs.  Unless I've missed something, you cannot.  The ranking of AAC-LC and Ogg encoders also looks contrary to previous results.

Quote
So, see the soundexpert tests as additional special tool, to collect informations about the encoders.


Yes, but what information?  Given that they do not correlate well with double blind listening tests, what exactly is being measured?

No one seems to be able to answer that.

Quote
btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...


I'm guessing this was meant to be a clever way to stay inside TOS#8 without having to support your assertion, or really even provide an arguement beyon "I'm right, you're wrong".


Why can't we just cut the offtopic? If you want to discuss the test methodology start a new thread as already requested by Serge in this very thread.
People insist and insist and insist in polluting this thread just because they don't agree with the methodology. What an annoying behaviour....
Title: 192 kbit/s listening test at SoundExpert
Post by: kornchild2002 on 2006-07-06 03:33:36
Mike can voice his opinions, he was just posting his comments against those of User.  Granted, I think the testing methedology needs to be discussed elsewhere but I don't agree with User's comments, especially the last quote in dealing with 128kbps VBR files.  I think anyone who makes that blatent of a statement needs to have their heads checked.  The audio world does not revolve around one pair of ears.  Yes, I perceive that 128kbps VBR iTunes AAC is transparent.  No, I don't have a "Hi-Fi" system but I am not about to go out and spend $3,000 on something that I don't need.  I would rather spend $200 and get a 1000 watt Logitech speaker system that does a pretty good job.  The closest thing I have to Hi-Fi is the iPod Hi-Fi and we all know that really isn't Hi-Fi at all.

Anyways, back to the test.  I am really glad that iTunes AAC was included.  I think the formats are now final.  Some bitrate tweaks might need to be made in order to get the bitrates to equal one another but this is a very small issue.  I do feal bad about advising -V 2 now especially when people did not like the 20kbps difference from the target bitrate.  Oh well.  With most music, I imagine -V 2 would have been fine but I guess the samples being used must be "less complex" from past samples used.

Still, methedology asside, I will be awaiting these results.  I think the results will be interesting for a test conducted in this manner.

Edit: Spelling
Title: 192 kbit/s listening test at SoundExpert
Post by: guruboolez on 2006-07-06 07:40:32
Why not? Especially if they are comparable in quality.
So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.

Fine. Then you shoud favor MPC --standard and vorbis -q6. It would lower the bitrate, as -V2 does for LAME (but probably not as much of course)

Quote
Unfortunately this approach is not ideal as well because actual figures will depend on type of albums chosen for bitrate calculations. Add more classical music albums and you’ll get lower values, more hard/metal – higher values. Such figures will be “just for reference” in any case like SE ones.

I know. Nevertheless, I guess that the average bitrate would be more representative. As example, LAME seems to give a lower than average bitrate for classical. But my own bitrate table [16 hours of music, 150 different tracks] gives me 180 kbps for -V2 [--vbr-new]; you got 173 kbps. People listening other kind of music reported ~190...200 kbps with -V2 with peaks at 220 kbps and more.
173 kbps according to your methodology seems to be significantly different from what people reported in the past several times.
I'm sure that a set of various disc would give good results for a bitrate table. Not necessary perfect, but at least worth to try

EDIT: at last, I found again the message I was looking for:

http://www.hydrogenaudio.org/forums/index....ndpost&p=367334 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=41847&view=findpost&p=367334)

Quote
- Musepack restriction was set by recommendations of musepack.net people - non-"integral" quality values are not as well-tested and are hardly ever used.


So testing something between -q5 and -q6 is not recommended by the musepack.net maintainers. Moreover, it's not representative (it's likely that people are using the old "presets").[/color]
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-06 09:37:07
Out of curiosity, I tested LAME, Vorbis, Nero and Musepack bitrates with the same set of 25 test tracks that I have used previously [1] (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=37011&view=findpost&p=328558) [2] (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=38955&view=findpost&p=344516).

At first I encoded the files with LAME 3.97b2 @ -V2 --vbr-new. The average bitrate was 196 kbps.
As example, LAME seems to give a lower than average bitrate for classical. But my own bitrate table [16 hours of music, 150 different tracks] gives me 180 kbps for -V2 [--vbr-new]; you got 173 kbps. People listening other kind of music reported ~190...200 kbps with -V2 with peaks at 220 kbps and more.
173 kbps according to your methodology seems to be significantly different from what people reported in the past several times.
I'm sure that a set of various disc would give good results for a bitrate table. Not necessary perfect, but at least worth to try

With all respect to your actual bitrate findings SE bitrate calculations have its own “pros”. And the most important one is possibility of testing codecs in precisely close conditions because SE FBR values correspond directly to the samples being used for testing. I would rather offer some kind of coefficients for users in order they could recalculate SE FBR for classical music, pop/rock, electronics, trance, ambient, new age, speech and so on. So I don’s see a serious reason to change now SE bitrate calculation method.

So testing something between -q5 and -q6 is not recommended by the musepack.net maintainers. Moreover, it's not representative (it's likely that people are using the old "presets").[/color]

I've asked for help from musepack.net people.

IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)

Not bag idea. What do you think, people.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-06 09:47:44
Why is WMA the only codec using CBR? why not use VBR on WMA as well?

The nearest WMA vbr mode (188 kbit/s) could be achieved only in WMEncoder by using bitrate based VBR 192. Anybody interested in? BTW, WMPlayer presets will be tested later.
Title: 192 kbit/s listening test at SoundExpert
Post by: Alex B on 2006-07-06 12:50:26
With all respect to your actual bitrate findings SE bitrate calculations have its own “pros”. And the most important one is possibility of testing codecs in precisely close conditions because SE FBR values correspond directly to the samples being used for testing. I would rather offer some kind of coefficients for users in order they could recalculate SE FBR for
classical music, pop/rock, electronics, trance, ambient, new age, speech and so on. So I don’s see a serious reason to change now SE bitrate calculation method.

Since your approach is to fit the test samples as close as possible to a given disk space, naturally you should continue doing that.

As we know, the other approach would be to fit a good representation of a typical complete audio library to a given disk space. This might not be as complex as one would like to think. After a certain amount of various tracks the bitrates seem to harmonize nicely and give consistent and comparable results over a wide variety of codecs and VBR settings.

Measuring just the test samples is not suitable if the test tries to be as useful as possible for general public. Even if a certain short sample is generally "difficult" or "easy" for all codecs, each individual codec presents usually a very different reaction. The resulting VBR setting may differ considerably from what is a commonly used VBR setting for a given target. (Like the LAME V1, V2 and possibly also Nero bitrates exhibit now).

The third approach would be to match the average bitrate with a given streaming bandwidth. Naturally, then the settings that produce an even average bitrate with different tracks (or samples) should be used.

Quote
Musepack restriction was set by recommendations of musepack.net people - non-"integral" quality values are not as well-tested and are hardly ever used.

Interesting. I have not heard anything negative about the "in-between" Musepack settings before.

Quote
The nearest WMA vbr mode (188 kbit/s) could be achieved only in WMEncoder by using bitrate based VBR 192. Anybody interested in? BTW, WMPlayer presets will be tested later.

This is a typical WMA VBR problem. VBR75 is too low and VBR90 produces surprisingly high bitrates. (This is most likely an unfixed VBR mode bug. The previous WM codec version produced much lower VBR90 bitrates.)

I guess that the best current WMA codec for the test target would be WMA Pro in VBR 192 kbps 2-pass mode, so if you accept the Windows Media Encoder route then you could as well use WMA Pro. (BTW, dBpowerAMP has the same options in a more usable GUI.)


Edit: typo
Title: 192 kbit/s listening test at SoundExpert
Post by: gameplaya15143 on 2006-07-06 16:40:42
1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Looks good to me  I can't wait to see the results.

For all the people questioning he-aac at 192kbps... keep in mind the high rate encoder is being used.  The LC portion is probably lowpassed around 18khz **guess** and SBR is only used above that.  Since many people can't hear much above 16khz, the SBR wouldn't be noticable to them.
Title: 192 kbit/s listening test at SoundExpert
Post by: jimmy69 on 2006-07-07 12:17:13
About using V1 or V2 for lame.  I've found V2 to be closer at 192kbs then V1.  I did this by comparing the file size of lame (3.97 b2) with itunes aac.  I found that lame and itunes were very close.  The two only being different by about 100kb at the most while a couple files were the same.  At V1 all lames files were larger.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-08 09:23:14
So testing something between -q5 and -q6 is not recommended by the musepack.net maintainers. Moreover, it's not representative (it's likely that people are using the old "presets").[/color]

This danger turned out to be exaggerated. According to clarification from musepack.net forum (http://www.musepack.net/forum/viewtopic.php?t=317): “Fractional parameters are well tuned. The only non "tuned" parameters are those other than --quality n or --preset name.”

IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)

Not bag idea. What do you think, people.

People seem to be indifferent to this idea. Also taking into account that new contenders will be added to the test later I decided NOT TO increase target bitrate from 192 to 198, though it seems natural for present set of codec contenders.

Here is final set of codecs to be tested:

1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

The listening test will start July 15.

Thank you very much for the discussion. I propose to use this thread further for talking about problems during testing and its results.
Title: 192 kbit/s listening test at SoundExpert
Post by: Garf on 2006-07-08 11:55:24
For all the people questioning he-aac at 192kbps... keep in mind the high rate encoder is being used.  The LC portion is probably lowpassed around 18khz **guess** and SBR is only used above that.  Since many people can't hear much above 16khz, the SBR wouldn't be noticable to them.


Even if this were true, it still leads the the question: Why use HE-AAC then, if you aren't going to use the SBR part? To slow down the encoding and decoding and reduce compatibility? Using SBR for >16kHz is a gigantic waste of resources and is probably less efficient than direct coding.

I maintain it's a spectacularly stupid idea.
Title: 192 kbit/s listening test at SoundExpert
Post by: Sagittaire on 2006-07-13 09:01:30

For all the people questioning he-aac at 192kbps... keep in mind the high rate encoder is being used.  The LC portion is probably lowpassed around 18khz **guess** and SBR is only used above that.  Since many people can't hear much above 16khz, the SBR wouldn't be noticable to them.


Even if this were true, it still leads the the question: Why use HE-AAC then, if you aren't going to use the SBR part? To slow down the encoding and decoding and reduce compatibility? Using SBR for >16kHz is a gigantic waste of resources and is probably less efficient than direct coding.

I maintain it's a spectacularly stupid idea.


Little question

1) Upscaling can help for Source Quality (44.1 Khz -> 88.2 or 96 Khz) ???

2) If yes why not high quality upsampling at 88.2 or 96 Khz conversion and aac+ encoding at 192 Kbps ???

But perhaps really stupid idea ... ???
Title: 192 kbit/s listening test at SoundExpert
Post by: Garf on 2006-07-13 11:41:22
1) Upscaling can help for Source Quality (44.1 Khz -> 88.2 or 96 Khz) ???


No.
Title: 192 kbit/s listening test at SoundExpert
Post by: gameplaya15143 on 2006-07-13 16:42:04
Why use HE-AAC then, if you aren't going to use the SBR part?
The SBR part is used, its just that joe average can't hear it.

Quote
Using SBR for >16kHz is a gigantic waste of resources and is probably less efficient than direct coding.
There should be a listening test to determine the validity of that assumption 
..and does your speculation still hold at ~100kbps?
Title: 192 kbit/s listening test at SoundExpert
Post by: SebastianG on 2006-07-13 17:22:45
Using SBR for only the part above 16 kHz and operating on a sampling rate of 44100 Hz both rule out each other since the LC part operates on half the sampling rate (and thus can only cover 0-11 kHz at max).
Title: 192 kbit/s listening test at SoundExpert
Post by: Ivan Dimkovic on 2006-07-13 17:43:22
No, actually with 192 kbps you would prolly use oversampled SBR, where SBR codec operates on 88.2 kHz - and LC-AAC core operates untouched at 44.1 kHz.

I cannot recall, but I think since the frequency resolution ror placing the SBR borders at 88.2 kHz is very coarse you could either start-off with SBR at around 14 kHz (too low as SBR would introduce pre-echo artifacts of its own, even with two times higher time resolution) or somewhere at 18 kHz (irrelevant, inaudible)  - so it is not too useful at 192 kbps and would prolly just waste couple of kbits/s.

Quote
..and does your speculation still hold at ~100kbps?


It is not a speculation - it is an ABX verified claim by people that actually invented SBR in their own AES paper 6199 "aacPlus, Only a Low-Bitrate Codec?"

http://www.aes.org/publications/preprints/lists/117.cfm (http://www.aes.org/publications/preprints/lists/117.cfm)

Because of the AES copyrights I am in no position to publish results of those ABX tests, but they prove that "Downsampled SBR" (operating at 88.2 kHz) brings no benefits to the overall quality compared to "plain" SBR or LC-AAC at 96 kbps.  It does, however, bring significant increase in complexity of the decoders.

I see no possible way how it can help at higher bit rates where LC-AAC has much more bits in disposal for transparent coding in its own frequency range.

But if this discussion continues in the direction asking for proofs of this I might as well ask AES to give HA permission to publish the ABX test results if possible.
Title: 192 kbit/s listening test at SoundExpert
Post by: WmAx on 2006-07-13 19:25:34
But if this discussion continues in the direction asking for proofs of this I might as well ask AES to give HA permission to publish the ABX test results if possible.


Facts are not copyrightable in the United States. You can freely report the test results and procedures. The graphs, verbatim text and other specific items are copyrightable, and thus require permission(s) from the copyright holder(s).

-Chris
Title: 192 kbit/s listening test at SoundExpert
Post by: Ivan Dimkovic on 2006-07-13 22:45:48
Problem is - there are no numbers, but just print-outs of excel tables with confidence intervals

The only possibility is that I redraw them - which is not the best way I think.
Title: 192 kbit/s listening test at SoundExpert
Post by: gameplaya15143 on 2006-07-14 15:45:11
Problem is - there are no numbers, but just print-outs of excel tables with confidence intervals

The only possibility is that I redraw them - which is not the best way I think.
  I wanted to see the pretty pictures.
Quote
or somewhere at 18 kHz
  So I guess my 'guess' might not be too far from the truth.

I did a quick test to discover what the lowpass frequency was on CT lc-aac @ 192kbps, analfreq showed that it did not use any lowpass at this rate.

Looking at the high rate encoder's frequency at 192kbps, my guess would be that the SBR starts at about 17.5-18khz (long-term average showed the frequencies above 17.5khz to be slightly louder).

Just thought I'd share.  And yea, IMHO SBR should be pretty worthless at such a high bitrate.  I wonder if the claims at soundexpert about downsampled sbr has anything to do with the 'effect' that sbr has on how something sounds (kind of like how one would use an equalizer to have a more enjoyable listening experience).
Title: 192 kbit/s listening test at SoundExpert
Post by: Steve999 on 2006-07-15 01:44:40
Serge --

Since I usually target about 192 kbps for my average rip, the next test will be very interesting for me.  I'm hoping everything is well above 5.0 at that rate, so I can feel safe in just choosing whatever codec is most convenient to use or has the best features (gapless / tagging / compatability etc.) for a particular use.  I fiind the above-5.0 results very interesting, especially since I use EQ, so that a little extra margin beyond bare transparency is a very good thing.

I would like to have seen Apple Itunes 6 VBR mp3, maybe 160 kpbs vbr mp3 highest quailty (which with itunes 6 will oten result in about 192 kbps).  Such a test would be very interesting to joe sixpack (who is more likely to use itunes than anything else), I think, but I probably should have asked earlier.

Anyway, thanks for your efforts!!!
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-15 08:01:04
I would like to have seen Apple Itunes 6 VBR mp3, maybe 160 kpbs vbr mp3 highest quailty (which with itunes 6 will oten result in about 192 kbps).  Such a test would be very interesting to joe sixpack (who is more likely to use itunes than anything else), I think, but I probably should have asked earlier.

Sorry, it’s too late now to change codec contenders as the test will start today/tomorrow. But the next one will be something like “top 5 most popular compression modes” without any particular target format or bitrate (just to see how they compare with each other). MP3 from iTunes has a good chance to be included.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-16 12:38:07
The test is opened.

First results (not reliable though) will appear on this page (http://www.soundexpert.info/coders192.jsp) immediately after first grades is returned by participants. Each rating needs about 300 grades. It might seem that a lot of work is required but fortunately a single testing session (one test file) is short and simple. So it won’t be too hard to test 3-5 test files at a time. You are welcome … and thank you in advance!

In order to participate in this test, please, download and grade a test file (http://ftp://www.soundexpert.info/).
Title: 192 kbit/s listening test at SoundExpert
Post by: jimmy69 on 2006-07-17 07:02:44
From those first results I don't understand what it means when the number is above.  Is it the higher the number the better or the number that is closer to 5?
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-17 07:17:24
From those first results I don't understand what it means when the number is above.  Is it the higher the number the better or the number that is closer to 5?

The higher the better (more perceived quality margin).
Title: 192 kbit/s listening test at SoundExpert
Post by: jimmy69 on 2006-07-17 07:20:03
In the case, I know that these results are not reliable but wow I wasn't expecting wma to be on top.  I was expecting it to be around last or second to last.
Title: 192 kbit/s listening test at SoundExpert
Post by: Garf on 2006-07-17 07:55:52
Serge --

Since I usually target about 192 kbps for my average rip, the next test will be very interesting for me.  I'm hoping everything is well above 5.0 at that rate, so I can feel safe in just choosing whatever codec is most convenient to use or has the best features (gapless / tagging / compatability etc.) for a particular use.  I fiind the above-5.0 results very interesting, especially since I use EQ, so that a little extra margin beyond bare transparency is a very good thing.

I would like to have seen Apple Itunes 6 VBR mp3, maybe 160 kpbs vbr mp3 highest quailty (which with itunes 6 will oten result in about 192 kbps).  Such a test would be very interesting to joe sixpack (who is more likely to use itunes than anything else), I think, but I probably should have asked earlier.

Anyway, thanks for your efforts!!!


You might want to read the comments me, Ivan, Woodinville and SebastianG gave regarding this testing methodology first, before reaching any "conclusions".
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-17 08:16:06
In the case, I know that these results are not reliable but wow I wasn't expecting wma to be on top.  I was expecting it to be around last or second to last.

In case of present testing activity first more or less reliable results are expected in a month or so. Now moving ratings are more like entertainment
Title: 192 kbit/s listening test at SoundExpert
Post by: SebastianG on 2006-07-17 14:16:25
Let's see how long AAC+ stays in the top sector. 
Title: 192 kbit/s listening test at SoundExpert
Post by: guruboolez on 2006-07-17 14:25:48
Let's see how long AAC+ stays in the top sector. 

According to the current results, AAC+ is ~30% better than Nero LC-AAC and Vorbis aoTuV and -more important- with a high reliability of these results. It implies that AAC+ should stay not on top (as WMA and MPC current positions are not reliable thus fixed) but at least as clearly superior to Nero LC-AAC, Vorbis aoTuV and LAME MP3. If positions will change in the future then the reliability indicator would appear as... not reliable 

Am I wrong?
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-07-17 16:03:22
According to the current results, AAC+ is ~30% better than Nero LC-AAC and Vorbis aoTuV and -more important- with a high reliability of these results. It implies that AAC+ should stay not on top (as WMA and MPC current positions are not reliable thus fixed) but at least as clearly superior to Nero LC-AAC, Vorbis aoTuV and LAME MP3. If positions will change in the future then the reliability indicator would appear as... not reliable 

Am I wrong?

You right, you right, guru, - reliability indicator is not reliable indeed. Now it shows error tube for last 10 values (new rating value is calculated each time a grade for this codec is returned). To be true, the figure “10” was spun out of thin air just to test this indicator. Now statistics show that the error tube has to be three times longer at least (and to be dependent in some way on number of testing points). New indicator is almost ready and will be implemented in a week or two (with other new features). But now it is informative only for me
Title: 192 kbit/s listening test at SoundExpert
Post by: Bodhi on 2006-08-17 20:55:35
Sorry but I don't get the point to those tests...

Have a look at guruboolez's (http://www.hydrogenaudio.org/forums/index.php?showtopic=36465) and have a look at SoundExpert's (http://www.soundexpert.info/coders192.jsp) listening tests. I'm not fluent (I'm french speaking) so maybe I missed something ? All those tests seem to be completly subjective...

Final judge: MY EARS

Title: 192 kbit/s listening test at SoundExpert
Post by: Steve999 on 2006-08-18 04:04:49
http://www.soundexpert.info/coders192.jsp;...id=CMOCFENMCBDA (http://www.soundexpert.info/coders192.jsp;jsessionid=CMOCFENMCBDA)

Looks to me like the results are solidifying and lame is really making a very nice showing.  The concept of degrees of transparency is an interesting one.  I sort of imagine it as leaving a little room for some singal processing / EQ without making otherwise masked artifacts audible.
Title: 192 kbit/s listening test at SoundExpert
Post by: HotshotGG on 2006-08-18 05:42:30
Quote
I see no reason to perform an ABX comparison test at a bitrate above 130-150kbps. Cause the 128kbps test show that 90-something % of all test samples are already transparent. So I can't imagine how hard it would be to differ lossy and source at a bitrate 50% above this. Good luck!


I am looking at this test and I am thinking the exact same thing myself. Countless tests have been performed I find those results hard to believe 
Title: 192 kbit/s listening test at SoundExpert
Post by: greynol on 2006-08-18 06:47:59
Here:
Sorry, this is my fault – there is no popular explanation of SoundExpert (SE) testing methodology on the site still (except two boring ones in pro zone (http://www.soundexpert.info/prozone.htm)). So I have to explain over and over again: SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases. Thus, all SE ratings above 5.00 show amount of perception margin of human auditory system for those tested devices (only codecs now). The absence of reference sample in SE test setup is not essential as the artifacts are clearly audible to more or less extent. The actual aim of SE listening session is to grade the annoyance of those artifacts.

Since 2001 a fixed set of reference (critical) sound samples (from SQAM disc mostly) is used for SE listening tests. All average bit rates are calculated on basis of these samples and called “file based bit rate” (FBR). As the sample set consist of different sound pieces including solo instruments, voices and speech the resulting bit rates (in VBR mode) are slightly lower than in case of pop/rock music (quite close to the bit rates for classical music though).

Edit: smilie and unnecessary comment removed.
Title: 192 kbit/s listening test at SoundExpert
Post by: Diow on 2006-08-18 23:44:08
Why use -b320 with stereo in lame 3.90.3 instead the --alt--preset insane ?
EDIT:included the version of lame.
Title: 192 kbit/s listening test at SoundExpert
Post by: Bodhi on 2006-08-20 01:17:09
Why use -b320 with stereo in lame 3.90.3 instead the --alt--preset insane ?
EDIT:included the version of lame.

Because it's just another: "What's the difference between a camel ?"

Another nonsense: Am I gonna hear the difference between transparency and transparency ?

Or: am I God ? (God, please forgive me).
Title: 192 kbit/s listening test at SoundExpert
Post by: halb27 on 2006-08-23 21:22:51
Why use -b320 with stereo in lame 3.90.3 instead the --alt--preset insane ?
EDIT:included the version of lame.

This was my suggestion. Exactly speaking I suggested to use 3.90.3 -b320 as well as api.
But this was a bit of too much Lame participants to Serge which I can understand. At that time I prefered -b320 GPSYCHO over api NSPSYTUNE, so the -b320 usage made it into the test.

You are quite right as to prefer the joint stereo using api, and I have changed my mind meanwhile and I prefer 3.90.3 joint stereo nspsytune as well.

Sorry that we don't have the api results because of this history.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-09-14 14:20:10
While current 192 kbit/s listening test is going smoothly (but slowly, to be true), there were several improvements of ratings’ visualization made:

•   In addition to actual rating a bar now displays min and max values. As each device is tested under nine different sound samples, there are nine different ratings for a device. Now min and max ones are indicated.
•   Calculation of ratings’ reliability was toughened to make it more reliable itself (specially for guruboolez (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=46126&view=findpost&p=412712)  ). This became possible after analysis of ratings' behavior during previous listening tests.
•   Short help section appeared on every ratings page under bars.
•   Tiny ruler was added to each bar to increase diagram usability.

All these improvements are fully visible only on 192 kbit/s page because only those ratings are updated now.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2006-11-07 09:47:32
Finally 192 kbit/s listening test (http://www.soundexpert.info/coders192.jsp) is coming to the end. Longer testing period was due to summer inactivity of visitors and toughening criteria of ratings reliability. Now each rating needs more returned grades to become reliable. Nevertheless it’s time to finish the test because new WMA pro codec is waiting for testing. Also HA people are going to start new test and I wouldn’t like the tests to be overlapped. So I encourage everybody to test a few files during this week in order to finish this test. After that we could start discussion on new one which I suppose will include people’s most popular codecs and settings and new competitors of course. 

Here is a link to a test file (http://ftp://www.soundexpert.info/) for your convenience. Thank you in advance for participation.

EDIT: test link added
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2007-01-09 10:22:08
Finally 192 kbit/s listening test has finished with the following results:

See 192 kbit/s page (http://www.soundexpert.info/coders192.jsp) for details. SoundExpert says “thank you” for participation and “sorry” for such long period of testing which was necessary for experiments with reliability parameter of ratings. Next tests will be substantially shorter. Once again - Thank you!
Title: 192 kbit/s listening test at SoundExpert
Post by: vlada on 2007-01-15 06:32:55
I would like to somment on the usability:

AAC-LC - I agree, there are many iPods, but how many other players can play AAC? Some mobile phones, maybe.

Ogg Vorbis - Almost every good HW player supports it (Cowon - iAudio, iRiver even iPod with Rockbox firmware). It's a sad true, that there is almost no Ogg Vorbis support in car players and cellphones. There are probably no SW players (or only very few) without Ogg Vorbis support. And don't forget it's the only free fromat from the contenders.

AAC+ - Very poor HW support.

WMA 9.1 - Almost no HW support, very poor SW support.

MPC - Supported in HW only via RockBox. SW support isn't very good neither.

MP3 - Supported everywhere.
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2007-02-07 23:37:58
I would like to somment on the usability:

AAC-LC - I agree, there are many iPods, but how many other players can play AAC? Some mobile phones, maybe.

Ogg Vorbis - Almost every good HW player supports it (Cowon - iAudio, iRiver even iPod with Rockbox firmware). It's a sad true, that there is almost no Ogg Vorbis support in car players and cellphones. There are probably no SW players (or only very few) without Ogg Vorbis support. And don't forget it's the only free fromat from the contenders.

AAC+ - Very poor HW support.

WMA 9.1 - Almost no HW support, very poor SW support.

MPC - Supported in HW only via RockBox. SW support isn't very good neither.

MP3 - Supported everywhere.

Actually the post dipped me into serious thinking. Finally I realized that SoundExpert has no resources and (what is more important) no desire to monitor coders popularity and usability. This is just not our business (at least at the moment). The project is entirely about sound quality and nothing else. So it is just irresponsible to make recommendations according to something we are not sure of.

From now SE recommendations on coders choice will concern sound quality only, revealed during SE listening tests. This policy will be clearly stated in short disclaimer. Corresponding changes have been already made on the site.
Title: 192 kbit/s listening test at SoundExpert
Post by: Sl@vik on 2007-05-28 11:40:11
I would like to somment on the usability:

AAC-LC - I agree, there are many iPods, but how many other players can play AAC? Some mobile phones, maybe.

Ogg Vorbis - Almost every good HW player supports it (Cowon - iAudio, iRiver even iPod with Rockbox firmware). It's a sad true, that there is almost no Ogg Vorbis support in car players and cellphones. There are probably no SW players (or only very few) without Ogg Vorbis support. And don't forget it's the only free fromat from the contenders.

AAC+ - Very poor HW support.

WMA 9.1 - Almost no HW support, very poor SW support.

MPC - Supported in HW only via RockBox. SW support isn't very good neither.

MP3 - Supported everywhere.

Hi Serge! What is about new Lame 3.98 beta 3???
Title: 192 kbit/s listening test at SoundExpert
Post by: Serge Smirnoff on 2007-05-28 15:13:56
Hi Serge! What is about new Lame 3.98 beta 3???

If the version 3.98 beta 3 is close to its final release (like in case of 3.97 beta 2), I'm ready to add it.