Skip to main content ↑ ↓
Topic: 192 kbit/s listening test at SoundExpert (Read 94359 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

192 kbit/s listening test at SoundExpert

Reply #25 –
although this is probably the test i would be must interested in, i doubt you will get many different results that are outside the margin or error
Chaintech AV-710

192 kbit/s listening test at SoundExpert

Reply #26 –
Quote from: Mike Giacomelli on
Ok, but have you shown that the results are relevent to anything?  Because saying that 320k AAC+ SBR is the best encoder REALLY makes me doubt the relevence of the results.

In order to prove relevance of the methodology a big thoroughly organized standard listening test has to be performed. Sorry, for now I haven’t such possibility. All I can do is to test as many different codecs as possible and to watch how SE ratings correlate with results of other listening tests. That’s what I’m doing.

Concerning CT High Bitrate HE-AAC Encoder @320 see this HA discussion. I will add this codec to 192 and other SE groups to investigate the situation further.

Quote from: Mike Giacomelli on
In addition, unless I am missing something, you have not addressed the other objection I raised to the methodology.

If you mean absence of reference sample see my post #16, end of first paragraph.

As usually discussion on codec contenders turned into discussion on methodology flaws. May be it’s a good idea to open special topic, say, “SoundExpert testing methodology criticism”?

Quote from: kornchild2002 on
... I would also suggest throwing in iTunes AAC.  You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc.


Now I plan to add only one codec per format. Others could be added later. What AAC coder you prefer to include in this case – Free Nero or iTunes. I would prefer Nero. I’m pretty sure that Apple’s codec will have lower score just because it produces very audible artifact with one of SE test sample (glockenspiel) – a bug definitely.

Quote from: kornchild2002 on
As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low.  For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps.  -V 1 was just too high as it would produce average bitrates of 270kbps with metal.

But -V2 produces only 173.1 kbit/s on SE samples. Other contenders are 191-198 kbit/s.  Will it be honest competition?
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #27 –
Quote
But -V2 produces only 173.1 kbit/s on SE samples.

Unless these ones are killersamples, -V2 is transparent to me (didn't ABX lately, but I had problems with V6 and V5, so I just assume this). When using SARTAMP, all encoded samples might have audible artefacts.
BTW, I still don't get the pourpose of SARTAMP.

192 kbit/s listening test at SoundExpert

Reply #28 –
Quote from: LaserSokrates on
BTW, I still don't get the pourpose of SARTAMP.

Without SARTAMP all codecs above 160 kbit/s seem to be equal. This is not so indeed and the technology helps to rank them above 160 where standard listening tests become unreliable. In some circumstances greater quality margin is useful. In any case it is useful to preserve as much sound quality as possible at given bitrate.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #29 –
Those people, who don#t understand the way soundexpert tests work, should keep quiet and just watch at the moment, and make a criticismn topic besides this topic.

As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.

So, see the soundexpert tests as additional special tool, to collect informations about the encoders.

btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...

192 kbit/s listening test at SoundExpert

Reply #30 –
Quote from: user on
As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.

...like HE-AAC absolute superiority at 128 and 320 kbps.
http://www.soundexpert.info/coders128.jsp
http://www.soundexpert.info/coders320.jsp

AFAIK, it goes against all studies, listening tests and developers' experience.

Quote
btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones...

Who the hell are you to give such stupid advice to people? Modern encoders are most often very good. Collective listening tests performed by Roberto, Sebastian and Gabriel have proved it four consecutive time. I'm a trained listener; I can ABX lossy encoders at ~200 kbps but I must also admit that ~128 VBR encoders are often damn good and even transparent, even on critical listening conditions. Just count the 5.0 mark on this test. And several people here have confirmed how difficult ABXing has become at ~130 kbps and how satisfied they are by these encodings. I personaly find your opinion as very haughty - especially for someone who hadn't brought any ABX test at higher bitrate on this board in the four last years (or did I miss them?).

192 kbit/s listening test at SoundExpert

Reply #31 –
Quote from: user on
Those people, who don#t understand the way soundexpert tests work, should keep quiet and just watch at the moment, and make a criticismn topic besides this topic.

Sorry, but I consider criticism to be very much on-topic. You don't have to worry about me repeating "issues", though -- I've already said what I had to say. But let me link to a thread where Ivan, Garf and myself are questioning this testing methodology here.

After pressing "preview" and seeing that guruboolez already commented on your last paragraph (I totally agree) I'm not going to say anything else to that one.

Sebastian

192 kbit/s listening test at SoundExpert

Reply #32 –
I'm new to this forum.  I don't understand why he-aac and mpc are going to be in the test when the only thing they work on is a computer.  It would be good if the itunes aac encoder is there because its proven to be a compedetive codec in the past and because I use it.  Why don't you use the wma encoder from WMP11.  Finally what is the difference with LAME V1, V2, and V3.

192 kbit/s listening test at SoundExpert

Reply #33 –
Quote from: Serge Smirnoff on
Quote from: kornchild2002 on
... I would also suggest throwing in iTunes AAC.  You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc.


Now I plan to add only one codec per format. Others could be added later. What AAC coder you prefer to include in this case – Free Nero or iTunes. I would prefer Nero. I’m pretty sure that Apple’s codec will have lower score just because it produces very audible artifact with one of SE test sample (glockenspiel) – a bug definitely.

Quote from: kornchild2002 on
As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low.  For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps.  -V 1 was just too high as it would produce average bitrates of 270kbps with metal.

But -V2 produces only 173.1 kbit/s on SE samples. Other contenders are 191-198 kbit/s.  Will it be honest competition?



The only reason I suggest throwing in iTunes AAC is that it has yet to be compared against the new Nero AAC encoder.  Yes, I think it is pertinant to test the new Nero AAC encoder since I don't think anyone has tested it yet but I think it would be worth while to stack it up against a very popular AAC encoder, the iTunes AAC encoder.  I would much rather see Nero AAC (LC-AAC) and iTunes AAC (LC-AAC) and take out the WinAmp HE-AAC encoder especially since hardware support for HE-AAC is zero to none.  Maybe HE-AAC could be added later on and keep iTunes AAC and Nero AAC in the initial runs.

As for the -V 2 having a ~20kbps lower bitrate, this might not be fair but I think it should still be tested especially since so many people use -V 2 over -V 1.  Again, my claim on thhis might not hold any merit as I have had little experience on setting up proper ABX tests.

I think I am going to continue to push iTunes AAC though, that way it can be directly compared with Nero AAC just to see which one is better.  Besides, iTunes AAC has almost always retained a very slight lead (not statistically higher but still) over Nero AAC.  It would be nice to see the tides turn towards Nero AAC if those are the true results of the test.

192 kbit/s listening test at SoundExpert

Reply #34 –
Welcome back to the topic. I think, instead of fitting codecs and settings to SE groups, better to fit the groups to widely used codecs/settings. Frankly talking, those groups are created for user convenience only. SE system allows to group codecs according to different criteria. For example: CBR/VBR/ABR or according to quality ratings or actual bitrate as for now. Present division by bitrate is quite reasonable but it is not a strict guideline for testing. While theoretical efficiency of compression algorithms is interesting mostly for developers and advanced users, practical efficiency of widely used codecs and settings is interesting for majority of average users. Taking all this into account I suggest revised list of contenders:

1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Sorry, I have to include #7 for research purposes.

Quote from: jimmy69 on
Why don't you use the wma encoder from WMP11.

... because it is beta (not finished/tuned yet) and because it’s illegal to publish results of testing according to WMPlayer 11 End User License Agreement (EULA).

Quote from: jimmy69 on
Finally what is the difference with LAME V1, V2, and V3.

They are quality settings of Lame encoder which affect resulting audio quality and actual size of output mp3 files.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #35 –
Quote from: Serge Smirnoff on
1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR

Since everyone is now using --preset fast standard (aka -V2 -vbr-new) instead of the vbr-old setting, it would probably make more sense to test the fast version. As a historical comparison (and for die-hard 3.90.3 fans  ), additionally testing the legacy Lame 3.90.3 at the --alt-preset standard setting might be an option if you still have room for another contender.
Proverb for Paranoids: "If they can get you asking the wrong questions, they don't have to worry about answers."
-T. Pynchon (Gravity's Rainbow)

192 kbit/s listening test at SoundExpert

Reply #36 –
Nearly all codecs are transparent to most people at that bitrate, so artificially amplifying artifacts that are inaudible in normal listening setups only serves to determine which codec is "best" to transcode from, and transcoding from lossy sources shouldn't be done in the first place.

Lossy codecs aim at transparent perceived sound at the lowest possible bitrate. As gaekwad2 already said, this methodology rates how far a codec goes "over the top". When two codecs produce transparent results at a certain bitrate, it only shows which one might sound better at lower bitrates.

Since the test is most probably done regardless of this criticism anyway, I suggest waiting for aoTuV Vorbis Beta 5.
Nothing is impossible if you don't need to do it yourself.

192 kbit/s listening test at SoundExpert

Reply #37 –
@ Silversight and everyone else:

Stop talking about the test procedure. SoundExpert has been doing tests like this for some time now. It's not necessary to take part or accept the results themselves. If it is really neccesary, i would ask for separating this conversation to another thread.

@ Serge :

If you talk about using standard settings for LAME, shouldn't this be applicable to Nero, Vorbis and Musepack aswell? and if you use 3.97b2, it would be better to use --vbr-new, as already noted.

Edit:
suggestions:
vorbis -q6.25 or -q6
nero : -q0.625 or -q0.6
mpc : -q5.5  (?)

192 kbit/s listening test at SoundExpert

Reply #38 –
Quote
' date='Jul 4 2006, 17:55' post='408710']... and if you use 3.97b2, it would be better to use --vbr-new, as already noted.

Ok. Done.

1. mp3: Lame 3.97b2 [--preset fast standard] 173.1 kbit/s FBR

Quote
' date='Jul 4 2006, 17:55' post='408710']If you talk about using standard settings for LAME, shouldn't this be applicable to Nero, Vorbis and Musepack aswell?

suggestions:
vorbis -q6.25 or -q6
nero : -q0.625 or -q0.6
mpc : -q5.5  (?)


The only reason of choosing “unusual” quality settings was to achieve target bitrate as close as possible. Also AFAIC these codecs are equally tuned for all q-parameters. But if there are some “popular” values I’m ready to make changes.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #39 –
Quote from: Serge Smirnoff on
1. mp3: Lame 3.97b2 [--preset fast standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


Perfect.  Varied selection, iTunes vs. Nero, etc.

Will be interesting to see the results.

- Gow
Zune 80, Tak -p4 audio library, Lossless=Choice

192 kbit/s listening test at SoundExpert

Reply #40 –
Quote
1. mp3: Lame 3.97b2 [--preset standard] 177.1 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


min = 177.7 kbps & max = 198 kbps [iTunes, WMA]. Difference is higher than 10% (and 20 kbps).
Moreover, it seems that you decided to use very precise quality level (0.626, 6.37 & 5.49), probably to match a precise bitrate - but a different criterion (popularity) is used for LAME. This second criterion leads to considerably lower the bitrate and therfore handicap the format. It's likely that for most people there wouldn't be any perceptual difference between V1 and V2 in normal listening conditions but isn't your methodology supposed to amplify the distortions?

I'd use the same criterion for all encoder and try either to follow "popular" settings" or to match the same approximate bitrate for all. You said that:
“1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR”
Then use V1. The deviation would be much lower (min = 191 kbps and max 200 kbps => ~4...5%) and even perfectible (by increasing MPC bitrate).

192 kbit/s listening test at SoundExpert

Reply #41 –
Quote from: guruboolez on
I'd use the same criterion for all encoder and try either to follow "popular" settings" or to match the same approximate bitrate for all. You said that:
“1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR”
Then use V1. The deviation would be much lower (min = 191 kbps and max 200 kbps => ~4...5%) and even perfectible (by increasing MPC bitrate).

I rounded q-parameters for Nero, Ogg, MPC:

1.  mp3: Lame 3.97b2 [--preset fast standard or –V1] 173.1 or 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #42 –
Quote from: Serge Smirnoff on
Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.

Coherency is the most important point IMO. If a majority of people would rather see HE-AAC at 64 kbps instead of LC at 192, would you include the first one among all ~200 kbps encodings?
-V2 is clearly too low bitrate according to your methodology; V1 is marginaly higher. The second one is the best choice if you want to keep the comparison fair.

Other solution: try to build a bitrate table based on various albums. Values may significantly change (and would correspond to a real/daily usage of codecs)

192 kbit/s listening test at SoundExpert

Reply #43 –
Quote from: guruboolez on
Coherency is the most important point IMO. If a majority of people would rather see HE-AAC at 64 kbps instead of LC at 192, would you include the first one among all ~200 kbps encodings?

Why not? Especially if they are comparable in quality. I know at least one ISO listening test with different codecs at different bitrates. So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.

Quote from: guruboolez on
Other solution: try to build a bitrate table based on various albums. Values may significantly change (and would correspond to a real/daily usage of codecs)

Unfortunately this approach is not ideal as well because actual figures will depend on type of albums chosen for bitrate calculations. Add more classical music albums and you’ll get lower values, more hard/metal – higher values. Such figures will be “just for reference” in any case like SE ones.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #44 –
Quote from: Serge Smirnoff on
Why not? Especially if they are comparable in quality. I know at least one ISO listening test with different codecs at different bitrates. So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.

You call it a '192 kbps' test, so you should choose the setting that is closest to that bitrate. If -V1 --vbr-new is closer then you should choose that one. You seem to have chosen the setting for every other codec closest to 192 kbps, so why not for LAME?

Quote
Now I see the only point to be decided - what setting to choose for Lame: “fast standard” or V1? If there is more interest in testing “fast standard”, it could be moved safely to 160 kbit/s group at SE after this test finished.

No it can't. IMO 173.1 is neither close to 160 (8% diff) or 192 (10% diff), both differences are too big. If you want to call your test '~ xxx kbps-test' thats fine, but at least let all contenders respect that bitrate. Otherwise the name of your test is completely irrelevant to what you are actually testing.
"We cannot win against obsession. They care, we don't. They win."

192 kbit/s listening test at SoundExpert

Reply #45 –
Ok. You win, guys:

mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR

The next test will be heterogeneous for sure and will include most popular codecs/settings. I have to think out a proper name for it or make it completely nameless in order to simplify discussion.

This set seems to be final now:

1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #46 –
Quote
As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.


I seriously doubt you've looked closely at those results.  Do so now, and then try and find me a test showing 128k AAC-HE beating all other codecs.  Unless I've missed something, you cannot.  The ranking of AAC-LC and Ogg encoders also looks contrary to previous results.

Quote
So, see the soundexpert tests as additional special tool, to collect informations about the encoders.


Yes, but what information?  Given that they do not correlate well with double blind listening tests, what exactly is being measured?

No one seems to be able to answer that.

Quote
btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...


I'm guessing this was meant to be a clever way to stay inside TOS#8 without having to support your assertion, or really even provide an arguement beyon "I'm right, you're wrong".

192 kbit/s listening test at SoundExpert

Reply #47 –
Quote from: Serge Smirnoff on
1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)

192 kbit/s listening test at SoundExpert

Reply #48 –
Why is WMA the only codec using CBR? why not use VBR on WMA as well?

192 kbit/s listening test at SoundExpert

Reply #49 –
Out of curiosity, I tested LAME, Vorbis, Nero and Musepack bitrates with the same set of 25 test tracks that I have used previously [1] [2].

At first I encoded the files with LAME 3.97b2 @ -V2 --vbr-new. The average bitrate was 196 kbps.

Then I tweaked the other codecs to produce exactly the same average bitrate. After a few encoding rounds I settled with these final Q settings :

aoTuV 4.51b : -q 6.20  (196 kbps)
Nero (2006-05-26): -q 0.552  (196 kbps)
Musepack 1.15v: --quality 5.75  (196 kbps)

I think that Q settings like these would be more comparable and fairer.

I used Mr QuestionMan for bitrate measurements. The MrQ reports are available here: mrq.zip
 
SimplePortal 1.0.0 RC1 © 2008-2020