Skip to main content
Topic: 192 kbit/s listening test at SoundExpert (Read 93514 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

192 kbit/s listening test at SoundExpert

Reply #50
Quote


As example, I recommend to have a look at the 128k soundexpert test, which reveals, that these results confirm some usual knowledge about the qualities of the different encoders.


I seriously doubt you've looked closely at those results.  Do so now, and then try and find me a test showing 128k AAC-HE beating all other codecs.  Unless I've missed something, you cannot.  The ranking of AAC-LC and Ogg encoders also looks contrary to previous results.

Quote
So, see the soundexpert tests as additional special tool, to collect informations about the encoders.


Yes, but what information?  Given that they do not correlate well with double blind listening tests, what exactly is being measured?

No one seems to be able to answer that.

Quote
btw., everybody who thinks, that 128 vbr with modern codec, is transparent HiFi, should visit an ear doctor and avoid listening to 10 €/$ pc speakers/ear-/headphones... and avoid listening to live concerts with unprotected ears...


I'm guessing this was meant to be a clever way to stay inside TOS#8 without having to support your assertion, or really even provide an arguement beyon "I'm right, you're wrong".


Why can't we just cut the offtopic? If you want to discuss the test methodology start a new thread as already requested by Serge in this very thread.
People insist and insist and insist in polluting this thread just because they don't agree with the methodology. What an annoying behaviour....

192 kbit/s listening test at SoundExpert

Reply #51
Mike can voice his opinions, he was just posting his comments against those of User.  Granted, I think the testing methedology needs to be discussed elsewhere but I don't agree with User's comments, especially the last quote in dealing with 128kbps VBR files.  I think anyone who makes that blatent of a statement needs to have their heads checked.  The audio world does not revolve around one pair of ears.  Yes, I perceive that 128kbps VBR iTunes AAC is transparent.  No, I don't have a "Hi-Fi" system but I am not about to go out and spend $3,000 on something that I don't need.  I would rather spend $200 and get a 1000 watt Logitech speaker system that does a pretty good job.  The closest thing I have to Hi-Fi is the iPod Hi-Fi and we all know that really isn't Hi-Fi at all.

Anyways, back to the test.  I am really glad that iTunes AAC was included.  I think the formats are now final.  Some bitrate tweaks might need to be made in order to get the bitrates to equal one another but this is a very small issue.  I do feal bad about advising -V 2 now especially when people did not like the 20kbps difference from the target bitrate.  Oh well.  With most music, I imagine -V 2 would have been fine but I guess the samples being used must be "less complex" from past samples used.

Still, methedology asside, I will be awaiting these results.  I think the results will be interesting for a test conducted in this manner.

Edit: Spelling

192 kbit/s listening test at SoundExpert

Reply #52
Why not? Especially if they are comparable in quality.
So, what - "coherency" or practical usefulness? And last but not the least: more interesting contenders - more participants.

Fine. Then you shoud favor MPC --standard and vorbis -q6. It would lower the bitrate, as -V2 does for LAME (but probably not as much of course)

Quote
Unfortunately this approach is not ideal as well because actual figures will depend on type of albums chosen for bitrate calculations. Add more classical music albums and you’ll get lower values, more hard/metal – higher values. Such figures will be “just for reference” in any case like SE ones.

I know. Nevertheless, I guess that the average bitrate would be more representative. As example, LAME seems to give a lower than average bitrate for classical. But my own bitrate table [16 hours of music, 150 different tracks] gives me 180 kbps for -V2 [--vbr-new]; you got 173 kbps. People listening other kind of music reported ~190...200 kbps with -V2 with peaks at 220 kbps and more.
173 kbps according to your methodology seems to be significantly different from what people reported in the past several times.
I'm sure that a set of various disc would give good results for a bitrate table. Not necessary perfect, but at least worth to try

EDIT: at last, I found again the message I was looking for:

http://www.hydrogenaudio.org/forums/index....ndpost&p=367334

Quote
- Musepack restriction was set by recommendations of musepack.net people - non-"integral" quality values are not as well-tested and are hardly ever used.


So testing something between -q5 and -q6 is not recommended by the musepack.net maintainers. Moreover, it's not representative (it's likely that people are using the old "presets").[/color]

192 kbit/s listening test at SoundExpert

Reply #53
Out of curiosity, I tested LAME, Vorbis, Nero and Musepack bitrates with the same set of 25 test tracks that I have used previously [1] [2].

At first I encoded the files with LAME 3.97b2 @ -V2 --vbr-new. The average bitrate was 196 kbps.
As example, LAME seems to give a lower than average bitrate for classical. But my own bitrate table [16 hours of music, 150 different tracks] gives me 180 kbps for -V2 [--vbr-new]; you got 173 kbps. People listening other kind of music reported ~190...200 kbps with -V2 with peaks at 220 kbps and more.
173 kbps according to your methodology seems to be significantly different from what people reported in the past several times.
I'm sure that a set of various disc would give good results for a bitrate table. Not necessary perfect, but at least worth to try

With all respect to your actual bitrate findings SE bitrate calculations have its own “pros”. And the most important one is possibility of testing codecs in precisely close conditions because SE FBR values correspond directly to the samples being used for testing. I would rather offer some kind of coefficients for users in order they could recalculate SE FBR for classical music, pop/rock, electronics, trance, ambient, new age, speech and so on. So I don’s see a serious reason to change now SE bitrate calculation method.

So testing something between -q5 and -q6 is not recommended by the musepack.net maintainers. Moreover, it's not representative (it's likely that people are using the old "presets").[/color]

I've asked for help from musepack.net people.

IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)

Not bag idea. What do you think, people.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #54
Why is WMA the only codec using CBR? why not use VBR on WMA as well?

The nearest WMA vbr mode (188 kbit/s) could be achieved only in WMEncoder by using bitrate based VBR 192. Anybody interested in? BTW, WMPlayer presets will be tested later.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #55
With all respect to your actual bitrate findings SE bitrate calculations have its own “pros”. And the most important one is possibility of testing codecs in precisely close conditions because SE FBR values correspond directly to the samples being used for testing. I would rather offer some kind of coefficients for users in order they could recalculate SE FBR for
classical music, pop/rock, electronics, trance, ambient, new age, speech and so on. So I don’s see a serious reason to change now SE bitrate calculation method.

Since your approach is to fit the test samples as close as possible to a given disk space, naturally you should continue doing that.

As we know, the other approach would be to fit a good representation of a typical complete audio library to a given disk space. This might not be as complex as one would like to think. After a certain amount of various tracks the bitrates seem to harmonize nicely and give consistent and comparable results over a wide variety of codecs and VBR settings.

Measuring just the test samples is not suitable if the test tries to be as useful as possible for general public. Even if a certain short sample is generally "difficult" or "easy" for all codecs, each individual codec presents usually a very different reaction. The resulting VBR setting may differ considerably from what is a commonly used VBR setting for a given target. (Like the LAME V1, V2 and possibly also Nero bitrates exhibit now).

The third approach would be to match the average bitrate with a given streaming bandwidth. Naturally, then the settings that produce an even average bitrate with different tracks (or samples) should be used.

Quote
Musepack restriction was set by recommendations of musepack.net people - non-"integral" quality values are not as well-tested and are hardly ever used.

Interesting. I have not heard anything negative about the "in-between" Musepack settings before.

Quote
The nearest WMA vbr mode (188 kbit/s) could be achieved only in WMEncoder by using bitrate based VBR 192. Anybody interested in? BTW, WMPlayer presets will be tested later.

This is a typical WMA VBR problem. VBR75 is too low and VBR90 produces surprisingly high bitrates. (This is most likely an unfixed VBR mode bug. The previous WM codec version produced much lower VBR90 bitrates.)

I guess that the best current WMA codec for the test target would be WMA Pro in VBR 192 kbps 2-pass mode, so if you accept the Windows Media Encoder route then you could as well use WMA Pro. (BTW, dBpowerAMP has the same options in a more usable GUI.)


Edit: typo

192 kbit/s listening test at SoundExpert

Reply #56
1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

Looks good to me  I can't wait to see the results.

For all the people questioning he-aac at 192kbps... keep in mind the high rate encoder is being used.  The LC portion is probably lowpassed around 18khz **guess** and SBR is only used above that.  Since many people can't hear much above 16khz, the SBR wouldn't be noticable to them.
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

192 kbit/s listening test at SoundExpert

Reply #57
About using V1 or V2 for lame.  I've found V2 to be closer at 192kbs then V1.  I did this by comparing the file size of lame (3.97 b2) with itunes aac.  I found that lame and itunes were very close.  The two only being different by about 100kb at the most while a couple files were the same.  At V1 all lames files were larger.

192 kbit/s listening test at SoundExpert

Reply #58
So testing something between -q5 and -q6 is not recommended by the musepack.net maintainers. Moreover, it's not representative (it's likely that people are using the old "presets").[/color]

This danger turned out to be exaggerated. According to clarification from musepack.net forum: “Fractional parameters are well tuned. The only non "tuned" parameters are those other than --quality n or --preset name.”

IMHO, you should change the test target to 198 kbps and tweak Vorbis, Muspack and Nero to produce exactly 198 kbps. Then all contenders would be within +- 2.6 kbps of the target. (195.4 - 200.5)

Not bag idea. What do you think, people.

People seem to be indifferent to this idea. Also taking into account that new contenders will be added to the test later I decided NOT TO increase target bitrate from 192 to 198, though it seems natural for present set of codec contenders.

Here is final set of codecs to be tested:

1. mp3: Lame 3.97b2 [-V1 --vbr-new] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.63] 193.3 kbit/s FBR
3. aac: iTunes [192 kbit/s, VBR] 197.8 kbit/s FBR
4. wma 9.1 std: WMPlayer10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,4] 192.8 kbit/s FBR
6. mpc: v1.15v [--quality 5.5] 193.8 kbit/s FBR
7. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR

The listening test will start July 15.

Thank you very much for the discussion. I propose to use this thread further for talking about problems during testing and its results.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #59
For all the people questioning he-aac at 192kbps... keep in mind the high rate encoder is being used.  The LC portion is probably lowpassed around 18khz **guess** and SBR is only used above that.  Since many people can't hear much above 16khz, the SBR wouldn't be noticable to them.


Even if this were true, it still leads the the question: Why use HE-AAC then, if you aren't going to use the SBR part? To slow down the encoding and decoding and reduce compatibility? Using SBR for >16kHz is a gigantic waste of resources and is probably less efficient than direct coding.

I maintain it's a spectacularly stupid idea.

192 kbit/s listening test at SoundExpert

Reply #60

For all the people questioning he-aac at 192kbps... keep in mind the high rate encoder is being used.  The LC portion is probably lowpassed around 18khz **guess** and SBR is only used above that.  Since many people can't hear much above 16khz, the SBR wouldn't be noticable to them.


Even if this were true, it still leads the the question: Why use HE-AAC then, if you aren't going to use the SBR part? To slow down the encoding and decoding and reduce compatibility? Using SBR for >16kHz is a gigantic waste of resources and is probably less efficient than direct coding.

I maintain it's a spectacularly stupid idea.


Little question

1) Upscaling can help for Source Quality (44.1 Khz -> 88.2 or 96 Khz) ???

2) If yes why not high quality upsampling at 88.2 or 96 Khz conversion and aac+ encoding at 192 Kbps ???

But perhaps really stupid idea ... ???


192 kbit/s listening test at SoundExpert

Reply #62
Why use HE-AAC then, if you aren't going to use the SBR part?
The SBR part is used, its just that joe average can't hear it.

Quote
Using SBR for >16kHz is a gigantic waste of resources and is probably less efficient than direct coding.
There should be a listening test to determine the validity of that assumption 
..and does your speculation still hold at ~100kbps?
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

192 kbit/s listening test at SoundExpert

Reply #63
Using SBR for only the part above 16 kHz and operating on a sampling rate of 44100 Hz both rule out each other since the LC part operates on half the sampling rate (and thus can only cover 0-11 kHz at max).

192 kbit/s listening test at SoundExpert

Reply #64
No, actually with 192 kbps you would prolly use oversampled SBR, where SBR codec operates on 88.2 kHz - and LC-AAC core operates untouched at 44.1 kHz.

I cannot recall, but I think since the frequency resolution ror placing the SBR borders at 88.2 kHz is very coarse you could either start-off with SBR at around 14 kHz (too low as SBR would introduce pre-echo artifacts of its own, even with two times higher time resolution) or somewhere at 18 kHz (irrelevant, inaudible)  - so it is not too useful at 192 kbps and would prolly just waste couple of kbits/s.

Quote
..and does your speculation still hold at ~100kbps?


It is not a speculation - it is an ABX verified claim by people that actually invented SBR in their own AES paper 6199 "aacPlus, Only a Low-Bitrate Codec?"

http://www.aes.org/publications/preprints/lists/117.cfm

Because of the AES copyrights I am in no position to publish results of those ABX tests, but they prove that "Downsampled SBR" (operating at 88.2 kHz) brings no benefits to the overall quality compared to "plain" SBR or LC-AAC at 96 kbps.  It does, however, bring significant increase in complexity of the decoders.

I see no possible way how it can help at higher bit rates where LC-AAC has much more bits in disposal for transparent coding in its own frequency range.

But if this discussion continues in the direction asking for proofs of this I might as well ask AES to give HA permission to publish the ABX test results if possible.

192 kbit/s listening test at SoundExpert

Reply #65
But if this discussion continues in the direction asking for proofs of this I might as well ask AES to give HA permission to publish the ABX test results if possible.


Facts are not copyrightable in the United States. You can freely report the test results and procedures. The graphs, verbatim text and other specific items are copyrightable, and thus require permission(s) from the copyright holder(s).

-Chris

192 kbit/s listening test at SoundExpert

Reply #66
Problem is - there are no numbers, but just print-outs of excel tables with confidence intervals

The only possibility is that I redraw them - which is not the best way I think.

192 kbit/s listening test at SoundExpert

Reply #67
Problem is - there are no numbers, but just print-outs of excel tables with confidence intervals

The only possibility is that I redraw them - which is not the best way I think.
  I wanted to see the pretty pictures.
Quote
or somewhere at 18 kHz
  So I guess my 'guess' might not be too far from the truth.

I did a quick test to discover what the lowpass frequency was on CT lc-aac @ 192kbps, analfreq showed that it did not use any lowpass at this rate.

Looking at the high rate encoder's frequency at 192kbps, my guess would be that the SBR starts at about 17.5-18khz (long-term average showed the frequencies above 17.5khz to be slightly louder).

Just thought I'd share.  And yea, IMHO SBR should be pretty worthless at such a high bitrate.  I wonder if the claims at soundexpert about downsampled sbr has anything to do with the 'effect' that sbr has on how something sounds (kind of like how one would use an equalizer to have a more enjoyable listening experience).
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune

192 kbit/s listening test at SoundExpert

Reply #68
Serge --

Since I usually target about 192 kbps for my average rip, the next test will be very interesting for me.  I'm hoping everything is well above 5.0 at that rate, so I can feel safe in just choosing whatever codec is most convenient to use or has the best features (gapless / tagging / compatability etc.) for a particular use.  I fiind the above-5.0 results very interesting, especially since I use EQ, so that a little extra margin beyond bare transparency is a very good thing.

I would like to have seen Apple Itunes 6 VBR mp3, maybe 160 kpbs vbr mp3 highest quailty (which with itunes 6 will oten result in about 192 kbps).  Such a test would be very interesting to joe sixpack (who is more likely to use itunes than anything else), I think, but I probably should have asked earlier.

Anyway, thanks for your efforts!!!

 

192 kbit/s listening test at SoundExpert

Reply #69
I would like to have seen Apple Itunes 6 VBR mp3, maybe 160 kpbs vbr mp3 highest quailty (which with itunes 6 will oten result in about 192 kbps).  Such a test would be very interesting to joe sixpack (who is more likely to use itunes than anything else), I think, but I probably should have asked earlier.

Sorry, it’s too late now to change codec contenders as the test will start today/tomorrow. But the next one will be something like “top 5 most popular compression modes” without any particular target format or bitrate (just to see how they compare with each other). MP3 from iTunes has a good chance to be included.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #70
The test is opened.

First results (not reliable though) will appear on this page immediately after first grades is returned by participants. Each rating needs about 300 grades. It might seem that a lot of work is required but fortunately a single testing session (one test file) is short and simple. So it won’t be too hard to test 3-5 test files at a time. You are welcome … and thank you in advance!

In order to participate in this test, please, download and grade a test file.
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #71
From those first results I don't understand what it means when the number is above.  Is it the higher the number the better or the number that is closer to 5?

192 kbit/s listening test at SoundExpert

Reply #72
From those first results I don't understand what it means when the number is above.  Is it the higher the number the better or the number that is closer to 5?

The higher the better (more perceived quality margin).
keeping audio clear together - soundexpert.org

192 kbit/s listening test at SoundExpert

Reply #73
In the case, I know that these results are not reliable but wow I wasn't expecting wma to be on top.  I was expecting it to be around last or second to last.

192 kbit/s listening test at SoundExpert

Reply #74
Serge --

Since I usually target about 192 kbps for my average rip, the next test will be very interesting for me.  I'm hoping everything is well above 5.0 at that rate, so I can feel safe in just choosing whatever codec is most convenient to use or has the best features (gapless / tagging / compatability etc.) for a particular use.  I fiind the above-5.0 results very interesting, especially since I use EQ, so that a little extra margin beyond bare transparency is a very good thing.

I would like to have seen Apple Itunes 6 VBR mp3, maybe 160 kpbs vbr mp3 highest quailty (which with itunes 6 will oten result in about 192 kbps).  Such a test would be very interesting to joe sixpack (who is more likely to use itunes than anything else), I think, but I probably should have asked earlier.

Anyway, thanks for your efforts!!!


You might want to read the comments me, Ivan, Woodinville and SebastianG gave regarding this testing methodology first, before reaching any "conclusions".

 
SimplePortal 1.0.0 RC1 © 2008-2019