HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: rjamorim on 2003-07-24 00:30:32

Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-24 00:30:32
Hello.

I'd like to announce the start of the 128kbps extension test, comparing the AAC winner (QuickTime) to Musepack, Vorbis, WMA Pro and Lame MP3.

Instructions are available at the announcement page:
http://audio.ciara.us/test/128extension/pr...esentation.html (http://audio.ciara.us/test/128extension/presentation.html)

If questions or issues arise, please post at this thread.

Thank-you.

Regards;

Roberto Amorim
Title: 128kbps Extension Test - OPEN
Post by: ExUser on 2003-07-24 02:53:54
Any .torrents going to be made available this time?

And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)

I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-24 03:09:31
Quote
Any .torrents going to be made available this time?

Probably, yes.

But Dibrom has been too busy lately, and he plans to create the torrents after HA moves to another server.

Quote
And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)


Yes, it would be very welcome.

Unfortunately, I'm not a member of Slashdot or Kuro5hin, so I don't know about article submissions. I would be very grateful if some insider can post about this test there, pointing to the presentation page.

Quote
I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.


It will only be mentioned at the results page

That way, people won't be confused, since it's there for perspective purposes, not really to be compared.

Regards;

Roberto.
Title: 128kbps Extension Test - OPEN
Post by: dev0 on 2003-07-24 06:48:11
Just submitted it to slashdot.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 07:45:17
Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio

ff123
Title: 128kbps Extension Test - OPEN
Post by: Artemis3 on 2003-07-24 07:47:44
Made a little comment (http://arstechnica.infopop.net/OpenTopic/page?a=tpc&s=50009562&f=174096756&m=5610976675&r=5600996675#5600996675) at Ars Technica (http://www.arstechnica.com/).
Title: 128kbps Extension Test - OPEN
Post by: Cobra on 2003-07-24 09:03:20
How it`s done? All files are VBR but calculated averange 128kbps? So you encoed each file in each format in VBR mode unless acheving e.g. 127-129 kbps averange? IMO it`s best choice to test >>>128kbps VBR<<< files.

EDIT:
"Ogg Vorbis 1.0 post-CVS -q 4.25" - it`s not `128kbps every time, you can`t say that it is TRUE 128kbps test!
Title: 128kbps Extension Test - OPEN
Post by: ilikedirtthe2nd on 2003-07-24 09:06:03
see this threat:

http://www.hydrogenaudio.org/forums/index....showtopic=11134 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134)

regards; ilikedirt
Title: 128kbps Extension Test - OPEN
Post by: Cobra on 2003-07-24 09:16:17
Quote
ogg q 4.25

122.70901287553649, 131.1936462778568, 123.71958369470946, 126.4998828125, 124.25014547160957, 135.24413014700758, 125.75588978120831, 130.33543256174835, 128.77903642773208, 121.00131765659843
127.599113763
- these samples (exept one) are not in 128kbps...
Title: 128kbps Extension Test - OPEN
Post by: ilikedirtthe2nd on 2003-07-24 09:27:50
Quote
Quote
ogg q 4.25

122.70901287553649, 131.1936462778568, 123.71958369470946, 126.4998828125, 124.25014547160957, 135.24413014700758, 125.75588978120831, 130.33543256174835, 128.77903642773208, 121.00131765659843
127.599113763
- these samples (exept one) are not in 128kbps...

nobody said that, but their average bitrates are closesed to 128 from the tested ogg settings (~127).

i think this method was chosen, because vbr encoders get used this way (through quality settings). usually nobody would encode a files several times with vbr to get the bitrate closest to 128kb 

regards; ilikedirt
Title: 128kbps Extension Test - OPEN
Post by: Cobra on 2003-07-24 09:32:14
Of course nobody will encode in such way! But we test _128kbps_  files.  After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?

I just want to say that normal users behaviour is completely not important in 128kbps test.
Title: 128kbps Extension Test - OPEN
Post by: ilikedirtthe2nd on 2003-07-24 09:37:06
of course i see your point here. maybe rjamorim can tell you why he decided to test this way...

regards; ilikedirt
Title: 128kbps Extension Test - OPEN
Post by: Cobra on 2003-07-24 09:47:23
I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?
Title: 128kbps Extension Test - OPEN
Post by: Gabriel on 2003-07-24 09:53:02
Quote
Of course nobody will encode in such way! But we test _128kbps_ files. After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?


What is important is that the overall bitrate is about the same for every codec. If for a specific track codec A is using an average bitrate of 130 and codec B an average bitrate of 110, it is just the decision made by the codec. It is programmed this way, an no one manually forced codec A to use an higher bitrate.

Manually forcing each codec to reach exactly 128kbps for each sample would be unfair, as the codecs are not designed to exhibit such a behaviour. What is important is the overall bitrate.
Title: 128kbps Extension Test - OPEN
Post by: Mac on 2003-07-24 10:06:32
As Gabriel points out - this way you are also testing a codecs ability to judge how hard a sample is.  If Vorbis only gives the sample 110kbs and ends up sounding bad because of it - that is the fault of Vorbis's VBR handling, and so it should be punished

As far as I can tell, you can see which files are which because you load them into ABC-HR - which does all the blinding for you  Otherwise stick a biro in your eyes and you will be blind!  (joke)
Title: 128kbps Extension Test - OPEN
Post by: Volcano on 2003-07-24 10:19:01
Quote
I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?

Have you looked at the contents of the zip archives, or even read the readme?

The abc-hr_bin.zip package includes various CLI en-/decoders - and each sample package comes with a batch file which uses these CLI en-/decoders to convert the supplied FLAC file to a lossy format (and decode back to WAV again for ABC/HR to accept it). For those formats where this isn't possible because of the lack of CLI encoders (WMA, MP4), readily encoded files are supplied. This method saves loads of bandwidth.
Title: 128kbps Extension Test - OPEN
Post by: Daybreak on 2003-07-24 10:54:51
Okay I'm new to this thing....


Does using the ABX tool and comparing the original and sample x repeatedly before giving an actual rating skew the results?


Or are we supposed to simply listen from the rating table itself?
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 11:02:07
just one thing to say:

thanks a lot for this great test, rjamorim!!!
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 12:01:24
Two tests done... death2 and waiting.

/EDIT\
No, comparing two samples before deciding on the result doesn't skew the results.

ABX testing proves your results, especially if the difference is subtle.
\EDIT/
Title: 128kbps Extension Test - OPEN
Post by: Daybreak on 2003-07-24 12:25:10
Another question on this testing thing ...

Rating 1-5 is against the original file right? Not against each other right? So assuming two codecs sound the same to you in comparision with the original file, they would rate about the same score? ( that's what I get from reading the Pratice With ABC/HR page )

Or is scoring done differently?

Also, are comments absolutely necessary?
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 12:32:56
The tests are rating against original.

Comments aren't necessary, but useful for others.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-24 13:02:36
Quote
Just submitted it to slashdot.

Quote
Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio


Quote
Made a little comment (http://arstechnica.infopop.net/OpenTopic/page?a=tpc&s=50009562&f=174096756&m=5610976675&r=5600996675#5600996675) at Ars Technica (http://www.arstechnica.com/).


Thanks a lot!

And thanks to everybody that is taking care or answering the questions that arise.

Regards;

Roberto.
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 13:05:49
Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-24 13:17:52
Quote
Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.

This time they will. I changed the readme for that purpose.

Only when the test finishes though.
Title: 128kbps Extension Test - OPEN
Post by: Night Rain on 2003-07-24 13:32:44
Submitted to Warp2Search.
Title: 128kbps Extension Test - OPEN
Post by: Jens Rex on 2003-07-24 14:28:54
Why mppenc 1.14 and not 1.15r? As far as I know, it has been very well tested.
Title: 128kbps Extension Test - OPEN
Post by: dev0 on 2003-07-24 14:56:36
Quote
Just submitted it to slashdot.

Code: [Select]
2003-07-24 05:41:19 HydrogenAudio 128kbps Extension test started (articles,music) (rejected)


Someone else should try it...

dev0
Title: 128kbps Extension Test - OPEN
Post by: Jens Rex on 2003-07-24 15:03:29
Submitted to Ars Technica news desk.
Title: 128kbps Extension Test - OPEN
Post by: bawjaws on 2003-07-24 15:04:20
If you're submitting to slashdot you should make clear that this is not another duplicate submission of the original 128kbps test which they accidentally reposted a few days ago.

http://slashdot.org/article.pl?sid=03/07/2...tid=181&tid=188 (http://slashdot.org/article.pl?sid=03/07/23/1221226&mode=thread&tid=126&tid=141&tid=181&tid=188)

No-one seemed to notice, though the comments seem of even lower quality.
Title: 128kbps Extension Test - OPEN
Post by: jrbamford on 2003-07-24 15:04:32
had a quick go... good god its hard... only really able to pick out 1-2 ones that sound different.. my hearing must be very bad  i DID spot preecho on fatboy at 320kbs --alt-preset insane once but had to turn it up so high as it got so quiet at that bitrate... I just dont know what i am looking for in here... death2 is the only one so far where i've heard something on 2 files...

Are most people able to pick out easily on a lot of them? these codecs sure are good at 128k (or there abouts)
Title: 128kbps Extension Test - OPEN
Post by: den on 2003-07-24 15:24:38
Ummm, I think we have a problem! 

I just ran the sample3.bat, and a couple of weird numbers caught my eye, so I changed the bat file so that it didn't delete the vorbis and mpc files...

Apples and oranges!

According to foobar, Bachpsichord.mpc - bitrate=198
Bachpsichord_ogg.ogg - bitrate=172

I would have thought it was a bit pointless directly comparing these against the .mp3 and .mp4 which both come in at 128 kbit according to foobar.

 

Den.
Title: 128kbps Extension Test - OPEN
Post by: ilikedirtthe2nd on 2003-07-24 15:25:27
Quote
had a quick go... good god its hard... only really able to pick out 1-2 ones that sound different.. my hearing must be very bad  i DID spot preecho on fatboy at 320kbs --alt-preset insane once but had to turn it up so high as it got so quiet at that bitrate... I just dont know what i am looking for in here... death2 is the only one so far where i've heard something on 2 files...

Are most people able to pick out easily on a lot of them? these codecs sure are good at 128k (or there abouts)

there is always one blade mp3 encode in the range. i try to track it first, abx it 10/10 (  ) and then take a look at the harder ones...

i personaly find sample 12 and sample 07 (here only the initial applaud) to be quite easy.

regards; ilikedirt
Title: 128kbps Extension Test - OPEN
Post by: den on 2003-07-24 15:29:50
Also sample1's .mpc is 166 kbits.

I would have thought that this is not close enough to 128...

Den.
Title: 128kbps Extension Test - OPEN
Post by: Daybreak on 2003-07-24 15:49:08
Thankfully I'm not the only deaf one... Only sample 1 seems easy for me.. the rest are pretty difficult..
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 15:51:05
Quote
Ummm, I think we have a problem!  
According to foobar, Bachpsichord.mpc - bitrate=198
Bachpsichord_ogg.ogg - bitrate=172

i can confirm that

winamp tells me:
Bachpsichord_ogg.ogg - Average bitrate : 164 kbps - Nominal bitrate : 128 kbps
Bachpsichord.mpc - Bitrate: VBR  197.9 kbps

the others seem fine

Quote
Just submitted it to slashdot.
Code: [Select]
2003-07-24 05:41:19 HydrogenAudio 128kbps Extension test started (articles,music) (rejected)

i wouldnt call it extension test!
call it something like hydrogenaudio 128kbps audio codec comparison test

but perhaps it would be wise to wait




and rjamorim can change the ogg and mpc settings if necessary (?) so that they are around 128kbps
if yes, i think it should help do rename the output files in sampleXX.bat to something like
"Sample_ogg2.wav" so that rjamorim can see in the results if the old ogg/mpc was tested or the newer ones...
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 16:05:52
due to the ","/"." failure in oggenc all .ogg files were encoded with "-q 4" (and not with "-q 4,25"):

sample1:
41_30sec_ogg.ogg - Average bitrate : 136 kbps
41_30sec.mpc - Bitrate: VBR  166.3 kbps
41_30sec.wav.mp3 - 126kbit (vbr) lame

sample3:
Bachpsichord_ogg.ogg - Average bitrate : 164 kbps
Bachpsichord.mpc - Bitrate: VBR 197.9 kbps
Bachpsichord.wav.mp3 - 125kbit (vbr) lame

sample7:
Layla_ogg.ogg - Average bitrate : 148 kbps
Layla.mpc - Bitrate: VBR  151.1 kbps
Layla.wav.mp3 - 130kbit (vbr) lame

sample11:
TheSource_ogg.ogg - Average bitrate : 121 kbps
TheSource.mpc - Bitrate: VBR  128.0 kbps
TheSource.wav.mp3 - 134kbit (VBR) lame

sample12:
Waiting_ogg.ogg - Average bitrate : 131 kbps
Waiting.mpc - Bitrate: VBR  147.6 kbps
Waiting.wav.mp3 - 122kbit (VBR) lame


perhaps in the end rjamorim should divide the points given to a codec through the bits used and than calculate an average for 128kbps 
Title: 128kbps Extension Test - OPEN
Post by: KikeG on 2003-07-24 16:06:49
Quote
Does using the ABX tool and comparing the original and sample x repeatedly before giving an actual rating skew the results?

No, but it's better if you do not know which is the codec you are ABXing, so that you can't possibly learn the sonic signature of it for that sample. If you do, this could lead to identifying each codec in the ABC/HR test, and would weaken the "blindness" of the procedure. So, just use the built-in ABX comparator in ABC/HR, but not an external one.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-24 16:34:28
Quote
Why mppenc 1.14 and not 1.15r? As far as I know, it has been very well tested.

In my opinion, it doesn't really matter. 1.15r quality is really close to 1.14 performance. In rare cases, benefit is audible (amnesia for exemple). But in others, 1.14 may be better. And this applies to --standard profile only...

But a most important thing to note is : 1.15r encodings are bigger than 1.14. Especially on some samples, as harpsicord (+10-15 kbps). And I doesn't ear any difference... For a mid-bitrate listening test, this small difference is maybe more annoying than in a archive perspective.


______

For people that are surprised with some bitrate deviation, don't forget two things :

- first, this is perfectly normal when you're testing the same VBR setting with various samples. You can't expect a constant value for different complexity samples : that's against VBR logical.

- second, you can't take a 30 seconds samples as a basis. For exemple, Bachpsichord 20 seconds are probably the highest ones of the whole double-disc of English Suites. Others samples where selected, and cutted, for their encoding difficulties. We can't expect anything else than a serious bitrate inflation with a well-tuned VBR setting (most famous exemple : first seconds of Kalifornia from Fatboy Slim).

Just take a whole Metallica album : bitrate will be near ~128 kbps with mppenc --radio. Isolate samples will probably reach 170-180 samples. Will people be annoyed by it ? Will they even notice it, without cutting a small part of the original PCM file and encode it ? No...

As a consequence, we haven't to be bothered by bitrate values of different encodings. It's a non-sense to criticize the bitrate amplitude of each VBR format on isolated samples. If you want 128 kbps for each format, choose CBR. If you choose VBR, enjoy the amplitude !
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 16:39:11
Quote
Ummm, I think we have a problem!  

I just ran the sample3.bat, and a couple of weird numbers caught my eye, so I changed the bat file so that it didn't delete the vorbis and mpc files...

Apples and oranges!

According to foobar, Bachpsichord.mpc - bitrate=198
Bachpsichord_ogg.ogg - bitrate=172

I would have thought it was a bit pointless directly comparing these against the .mp3 and .mp4 which both come in at 128 kbit according to foobar.

 

Den.

The point of the thread listed below was to find the appropriate settings for ogg and mpc which would average about 128 kbit/s across many album's worth of music:

http://www.hydrogenaudio.org/forums/index....showtopic=11134 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134)

But being VBR, they will sometimes have much higher bitrates on samples with lots of transients.  It's not fair to reduce the bitrates on these samples down to 128 kbit/s because that's not the way they'd actually be used.

ff123
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 16:46:30
hm, i see

ok, let's continue listening
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 17:07:47
Quote
had a quick go... good god its hard... only really able to pick out 1-2 ones that sound different.. my hearing must be very bad  i DID spot preecho on fatboy at 320kbs --alt-preset insane once but had to turn it up so high as it got so quiet at that bitrate... I just dont know what i am looking for in here... death2 is the only one so far where i've heard something on 2 files...

Are most people able to pick out easily on a lot of them? these codecs sure are good at 128k (or there abouts)

It's frustrating to leave some of the codecs at a rating of 5, isn't it?  At least it is to me.

Here's how I usually listen (it takes some time):

First, I make sure everything's as quiet as I can get it around the house (last night and tonight are good, because my wife and kids are away).

For a particular sample, I listen to each codec in its entirety (unfortunately, that means listening to each of the hidden references too).  I can usually pick out the 1 or 2 worst entries this way.  Then I'll go through the sample a short section at a time to listen for subtle differences I might not have picked up in the whole-sample listening.  Most of the time I'll pick up a problem in one codec in one section, but another problem in another codec in a different section.

For very subtle differences, ABX'ing usually helps to hone my ability to hear a defect.

Still, this comparison a lot more difficult than I think some people may have thought, given that this is "only" 128 kbit/s.

ff123
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 17:08:10
Quote
The point of the thread listed below was to find the appropriate settings for ogg and mpc which would average about 128 kbit/s across many album's worth of music:

http://www.hydrogenaudio.org/forums/index....showtopic=11134 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134)

But being VBR, they will sometimes have much higher bitrates on samples with lots of transients.  It's not fair to reduce the bitrates on these samples down to 128 kbit/s because that's not the way they'd actually be used.

ff123

Thanks ff123 for patiently explaining this to people who obviously missed Gabriel's explanation on the first page.. I prolly wouldn't have patience for as cool answer for a question which was answered once in this thread already and in a separate thread..
I hope nobody asks this for the 4th time in this thread..............

Maybe Roberto should put the explanation to his listening test page as well.
Title: 128kbps Extension Test - OPEN
Post by: voltron on 2003-07-24 17:11:48
I for one am having a hard time even finding the Blade encoded file. This test will take a lot longer than the few hours I had planned to devote to it.
Title: 128kbps Extension Test - OPEN
Post by: elmar3rd on 2003-07-24 17:12:06
Quote
The point of the thread listed below was to find the appropriate settings for ogg and mpc which would average about 128 kbit/s across many album's worth of music:

http://www.hydrogenaudio.org/forums/index....showtopic=11134 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134)

But being VBR, they will sometimes have much higher bitrates on samples with lots of transients.  It's not fair to reduce the bitrates on these samples down to 128 kbit/s because that's not the way they'd actually be used.

ff123

Regarding LAME alt-preset 128, which is ABR not VBR:

To have a fair competition for the ABR samples, we have to encode a complete file containing a problem-sample. The whole file must have an average bitrate of 128 kbps. Then we have to cut off the part with the problem sample for the listening-test. Only in this case, we give ABR-settings like alt-preset 128 the chance to increase the bitrate if needed like the VBR-settings do.

[Hope someone understand my bad english]
Title: 128kbps Extension Test - OPEN
Post by: Mac on 2003-07-24 17:16:47
I've found this test quite dis-heartening.  Out of 5 samples I have tried, I could only pick out Blade one 1 of them.  I can't even ABX another codec in any of them    [span style='font-size:7pt;line-height:100%'](why isn't there a crying emoticon?)[/span]
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 17:31:13
Quote
Regarding LAME alt-preset 128, which is ABR not VBR:

To have a fair competition for the ABR samples, we have to encode a complete file containing a problem-sample. The whole file must have an average bitrate of 128 kbps. Then we have to cut off the part with the problem sample for the listening-test. Only in this case, we give ABR-settings like alt-preset 128 the chance to increase the bitrate if needed like the VBR-settings do.

[Hope someone understand my bad english]

Hmm.. I don't know if this is very big issue with Lame ABR since it's 1 pass encode and ABR has unlimited bit reservour anyway.

If someone has time, check how the bit allocation goes in these 2 situations for example with encspot.

Edit. Hmm.. now that I actually think of it, yeah, this might be an issue exactly because of the unlimited bit reservour. 
Title: 128kbps Extension Test - OPEN
Post by: Case on 2003-07-24 17:34:45
Old issue with oggenc rises the head again:

Quote
C:\Temp\128kbps\Bin>oggenc -q 4.25 ..\Sample01\41_30sec.wav --output=..\Sample01
\41_30sec_ogg.ogg
Opening with wav module: WAV file reader
Encoding "..\Sample01\41_30sec.wav" to
         "..\Sample01\41_30sec_ogg.ogg"
at quality 4,00


Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 17:38:21
Quote
Old issue with oggenc rises the head again:

Quote
C:\Temp\128kbps\Bin>oggenc -q 4.25 ..\Sample01\41_30sec.wav --output=..\Sample01
\41_30sec_ogg.ogg
Opening with wav module: WAV file reader
Encoding "..\Sample01\41_30sec.wav" to
         "..\Sample01\41_30sec_ogg.ogg"
at quality 4,00


Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.

Ouch.. this is definitely a problem, although not catastrophic. -q4 is officially 128kbps nominal anyway.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 17:40:46
Quote
Regarding LAME alt-preset 128, which is ABR not VBR:

To have a fair competition for the ABR samples, we have to encode a complete file containing a problem-sample. The whole file must have an average bitrate of 128 kbps. Then we have to cut off the part with the problem sample for the listening-test. Only in this case, we give ABR-settings like alt-preset 128 the chance to increase the bitrate if needed like the VBR-settings do.

[Hope someone understand my bad english]

One could almost say the same thing about any codec which uses a bit-reservoir.

In practice, the only codec where it might matter whether or not the sample was "sliced from" a whole song, post-encoding, would be WMA9Pro 2-pass VBR.  Ideally, one would perform the 2-passes on the entire song (if not the entire album) and then slice out the sample afterwards.

But that's not very practical for this test, among other reasons being that Roberto doesn't have copies of the entire songs for the test suite.  So this test ends up using the 2-pass on just the 20 second samples.  Note that plain 1-pass VBR for WMA9Pro was found to be too variable in the bitrate thread mentioned above.

ff123
Title: 128kbps Extension Test - OPEN
Post by: Case on 2003-07-24 17:46:46
I uploaded modified (http://www.saunalahti.fi/cse/oggenc.zip) oggenc.exe that will use proper quality level, it uses recent CVS libraries.
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 17:51:53
Quote
Old issue with oggenc rises the head again:
Quote
C:\Temp\128kbps\Bin>oggenc -q 4.25 ..\Sample01\41_30sec.wav --output=..\Sample01
\41_30sec_ogg.ogg
Opening with wav module: WAV file reader
Encoding "..\Sample01\41_30sec.wav" to
         "..\Sample01\41_30sec_ogg.ogg"
at quality 4,00
Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.

wah, the same for me (and i already wondered about the results)


Quote
I've found this test quite dis-heartening.  Out of 5 samples I have tried, I could only pick out Blade one 1 of them.  I can't even ABX another codec in any of them    [span style='font-size:7pt;line-height:100%'](why isn't there a crying emoticon?)[/span]

you can have a look at the coments on the samples in the aac listening test (pretty much the same as now) here (http://rarewares.hydrogenaudio.org/test/aac128test/comments.html)
there you can find hints on what you should listen in each sample

first time i tried it i couldnt also hear many differences, but now i listened to 5 samples and i could hear differences in every sample (voted 5 only twice till now)
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 18:01:49
Quote
I uploaded modified (http://www.saunalahti.fi/cse/oggenc.zip) oggenc.exe that will use proper quality level, it uses recent CVS libraries.

thanks but is this the same cvs version rjamorim used?

he used "Ogg Vorbis 1.0 post-CVS". what does "post-cvs" mean?


edit:
according to the filesize case's oggenc produces exactly the same output as the original oggenc with -q 4!

->
i would recommend changing the *.bat files by hand if "," is used as a comma in your country!
Title: 128kbps Extension Test - OPEN
Post by: Case on 2003-07-24 18:13:49
Quote
thanks but is this the same cvs version rjamorim used?



edit:
according to the filesize case's oggenc produces exactly the same output as the original oggenc with -q 4!

->
i would recommend changing the *.bat files by hand if "," is used as a comma in your country!

I suppose so, except library version string isn't modified like john33 did.

Quote
he used "Ogg Vorbis 1.0 post-CVS". what does "post-cvs" mean?

I think it should be "post 1.0 CVS".
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 18:27:09
some one here who has access to the presentation page to update it with this (not unimportant) info?
Title: 128kbps Extension Test - OPEN
Post by: askoff on 2003-07-24 18:40:22
Should this test to be called Medium quality Extension test? If all samples are'nt close enough at 128kbps. Yes i've read all comments, but still this is not the right way i think. If MP3, AAC and WMA are forced to be near as possible to 128kbps and ogg and mpc can be what ever they want to bee...
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 19:06:04
Quote
Should this test to be called Medium quality Extension test? If all samples are'nt close enough at 128kbps. Yes i've read all comments, but still this is not the right way i think. If MP3, AAC and WMA are forced to be near as possible to 128kbps and ogg and mpc can be what ever they want to bee...

And what would be the right way in your opinion? Forcing vbr samples separately to 128kbps? I hope you understand that in that case you'd have 12 different settings for one vbr codec, so what would the results tell then about the codec's quality at certain setting - nothing...
Simply put: VBR is used with average bitrate basis here, that's the whole idea and the only sain thing to do. I'm sorry if you don't understand that, but that is the most fair way to test vbr.
Lame MP3 is ABR because it performs better than Lame MP3 vbr near the 128kbps. Quicktime is CBR because there's no VBR/ABR setting for it.
Title: 128kbps Extension Test - OPEN
Post by: Volcano on 2003-07-24 20:07:23
Quote
Quote
Old issue with oggenc rises the head again:

[...]

Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.

Ouch.. this is definitely a problem, although not catastrophic. -q4 is officially 128kbps nominal anyway.

Bummer!  If only someone had spotted this before the articles were submitted to those large sites...

I guess I'm lucky that at least on one of the two samples I've submitted results for so far, the Vorbis encoded version was totally transparent to me even though it was only encoded at -q 4, so I won't have to re-test that one.

So far, I am very surprised by the test... on the samples I've tested so far, the differences were very hard to detect and ABX most of the time (the two MP3 competitors - OK, Blade isn't really a "competitor"  - aside). Nice to see how the encoders have improved over time.


Quote
Still, this comparison a lot more difficult than I think some people may have thought, given that this is "only" 128 kbit/s.

I can still vaguely remember your last 128kbps test more than 1 1/2 years ago. Yeah, that was indeed a *lot* easier.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 20:20:05
I suggest that Roberto verifies with each person who submits results that their ogg ratings weren't affected by the comma bug (only a potential problem if they use the regional comma setting *and* happened to rate ogg files worse than the reference).

Meanwhile, I guess I'll append a note to my Usenet postings.

ff123
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-24 20:23:31
I'm a bit afraid to see that everybody could check immediatly after a test their results. Names are now explicit. It's easy to 'correct' some results, in order why not to give to our favorite format the best place... It's stupid I know, but there is a disease here called zealotry, and we can fear some cheating...

It's time for encrypted result log file...
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 20:34:00
i would say that this would be the best way:

upload as fast as possible a new abc-hr_bin.zip package instead (!) of the old one

the following changes should be done in the package:

1) exchange the old oggenc.exe with a version that isnt "," or "." sensitive like the one from case (although i am not sure if his version produces really 4.25 output in every case)
2) change the sampleXX.bat files in a way that the .ogg.wav files created with this updated package have a different name than the original ones (for example append a "_2" to the ogg filename so that rjamorim knows that there is the possibility that this file (and the rating) is flawed
-> than rjamorim can verify with the person who sent this package if the comma bug was present (so he doesnt have to check it with every person)
3) it will be than also possible to give the files "anonymous" names (like in the aac test) like guruboolez suggested


but the most important part will be that this happens as fast as possbile!!!
were is rjamorim or does anbody else has access to upload an updated package?
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 20:52:55
Okay, if required to, I can redo three tests I've sent. Got the bug.

The batch file would need to be unreadable and have no output.
Which isn't possible, of course.

/EDIT\
Oh well, it is... with a tool called Bat2Exe.
Now, how to suppress the output of all encoders?

Of course, you'll need to use anonymous names for all flacs and output files.
\EDIT/


You might want to write a dedicated program for this.

Encryption would be nice, but it would require modified version of ABC/HR.
And lots of work!
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 21:08:01
Quote
Okay, if required to, I can redo three tests I've sent. Got the bug.

The batch file would need to be unreadable and have no output.
Which isn't possible, of course. /EDIT\ No, it is... know a tool called Bat2Exe? \EDIT/
But you might want to write a dedicated program for this.

Encryption would be nice, but it would require modified version of ABC/HR.
And lots of work!

Unfortunately, I don't really have time anymore to modify ABC/HR.  Although the source is open, in practice, only the original authors seem to dare to modify such programs.

A program to modify the batch files should not be hard to gin up using something like python.  Let's see what Roberto wants to do.

ff123
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 21:15:47
Reread my post, please.
Modifying the batches won't help - how Roberto will know which file is which?

It has to be set earlier.
Additionally, encoding/decoding order should be 'randomized' (in the batch files, of course).
Title: 128kbps Extension Test - OPEN
Post by: spoon on 2003-07-24 21:27:17
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 21:52:37
Quote
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117956)
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117779)
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117895)

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 21:57:45
Here is the sample batch - you need to make an executable from it!
/EDIT\
Okay, final, working version!
\EDIT/
Code: [Select]
@echo off
flac -d --silent -o ..\Sample05\orig.wav ..\Sample05\orig.flac >> NUL:

rem Don't forget to randomize names and order of the parts!

rem Lossless part
copy /Y ..\Sample05\orig.wav ..\Sample05\5.wav >> NUL:
rem End

rem WMA part
flac -d --silent -o ..\Sample05\2.wav ..\Sample05\wma.flac >> NUL:
rem End

rem MP4 part
rem FAAD does support neither silent encoding nor pipes (weird, it should work with pipes...)
faad -o ..\Sample05\3.wav ..\Sample05\mp4.mp4 >> NUL:
rem End

rem MPC part
rem Workaround for MPPENC showing its version
mppenc --quality 4 --xlevel --silent ..\Sample05\orig.wav ..\Sample05\orig.mpc >> NUL:
mppdec --silent ..\Sample05\orig.mpc ..\Sample05\1.wav >> NUL:
del ..\Sample05\orig.mpc >> NUL:
rem End

rem Vorbis part
oggenc -Q -q 4.25 ..\Sample05\orig.wav --output=..\Sample05\orig.ogg >> NUL:
rem OggDec doesn't support silent decoding... here's a workaround
del ..\Sample05\4.wav >> NUL:
oggdec -o ..\Sample05\orig.ogg >> ..\Sample05\4.wav
del ..\Sample05\orig.ogg >> NUL:
rem End

rem LAME part
lame --alt-preset 128 --scale 1 --silent ..\Sample05\orig.wav ..\Sample05\orig.mp3 >> NUL:
lame --decode --silent ..\Sample05\orig.mp3 ..\Sample05\7.wav >> NUL:
del ..\Sample05\orig.mp3 >> NUL:
rem End

rem Blade part
bladeenc -q -quiet ..\Sample05\orig.wav ..\Sample05\orig.mp3 >> NUL:
lame --decode --silent ..\Sample05\orig.mp3 ..\Sample05\6.wav >> NUL:
del ..\Sample05\orig.mp3 >> NUL:
rem End

rem Cleanup
rem You may ommit this if you want, but this increases the security slightly
del ..\Sample05\orig.flac >> NUL:
del ..\Sample05\wma.flac >> NUL:
del ..\Sample05\mp4.mp4 >> NUL:
del ..\Sample05\orig.wav >> NUL:
Title: 128kbps Extension Test - OPEN
Post by: spoon on 2003-07-24 22:04:14
Quote
Quote
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117956)
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117779)
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117895)

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.

Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 22:13:50
Quote
Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?

Pretty much yeah, because it has been explained many times now, and frankly this is an issue which should be pretty much clear that you are getting no interpretable results about vbr codec's quality with the method you are suggesting..

So for example regarding Vorbis (just example, not real):
sample 1: -q2
sample 2: -q6
sample 3: -q3
sample 4: -q2.5
sample 5: -q3.3
sample 6: -q1.8
sample 7: -q4
sample 8: -q4.2
sample 9: -q5
sample10: -q6.5
sample11: -q2.3
sample12: -q3.5

Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...

And posting here is not any "democratic right". It's a priviledge which is possible to be taken away.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-24 22:26:06
Quote
Quote
Quote
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117956)
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117779)
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=2&t=11585&hl=&view=findpost&p=117895)

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.

Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?


In a fair world, you had to cut each track of each album in small 5 seconds parts, encode them manually with proper setting in order to reach x kbps, then merge all small parts into a complete track again. By doing that you will obtain VBR file, with each difficult or easy part encoded at x kbps. In other words, CBR with bit reservoir...

If some people doesn't like the kbps amplitude phenomenon, they just had to avoid VBR, and chose CBR encodings. Quality is worse, but bitrate will be constant... You can't expect choosing VBR for its quality and fighting against its own logical : more bitrate when needed. Is it so hard to understand ?
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 22:38:57
so perhaps we will see mpc as winner (or on a top position)  just because it's smart enough to "cheat" the comparison by using as much bitrate as possible to reach great quality (and everybody just wanted to add it for "academical reasons") 

"hey, i always had the opinion that mpc was the best codec @128kbps and the test also came to the same conlcusion" (although mpc hardly ever really used 128kbps)

 

in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 22:46:17
Quote
so perhaps we will see mpc as winner (or on a top position)  just because it's smart enough to "cheat" the comparison by using as much bitrate as possible to reach great quality (and everybody just wanted to add it for "academical reasons") 

"hey, i always had the opinion that mpc was the best codec @128kbps and the test also came to the same conlcusion" (although mpc hardly ever really used 128kbps)

 

in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...

Read this:
http://www.hydrogenaudio.org/forums/index....showtopic=11134 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134)

The reason why mpc and vorbis give high bitrates is because some of the samples are short and hard to encode in order to hear some difference between the codec qualities. I don't understand why vbr codecs should be punished, just because the test samples are short and have been chosen to be such that listeners may have easier job to distinguish differences..
Title: 128kbps Extension Test - OPEN
Post by: voltron on 2003-07-24 22:48:07
The encoder doesn't cheat.. it's encoding algorithms are such that when more bits are needed, more bits will be used. It is pointless to discriminate against one encoder by cutting out "problematic" parts of a song. This 128kbs test is for the very purpose of seeing which encoder can produce the best sound while using a relatively (depending on who) low bitrate. If it takes --quality 4 or whatever is being used to attain ~128 (ABOUT 128kbs), so be it.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-24 23:04:51
Quote
in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...

Encoding a whole album in order to reach 128 kbps, and let the same setting for a microscopic sample, is something fair. This is what Roberto asked us to do, by comparing mpc, ogg and wma at different settings on our favorite discs. Musepack --radio is close to 128 kbps, and for oggenc it's -b 4,25.

If you're familiar with your encoder, it's possible to anticipate some of its behaviour, and to chose different setting according to the disc you had to encode.
If I had a piano disc to encode with mpc at ~130 kbps, I will set the encoder at --quality 4.5 without trying --quality 4. For an harpsichord disc, I won't go beyond --quality 3. For orchestral, --quality 4 have the most chances to be close to the targeted bitrate...
Title: 128kbps Extension Test - OPEN
Post by: spoon on 2003-07-24 23:07:27
Still have my priviledge

Quote
Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...


The reason VBR exists is that a codec can be advanced enough to lower its bitrate and up its bit rate, but for this to be a fair test - especially with different sample types - for all we know on the harpsicord: ogg will go to an average of 200Kbps, whilst WMA goes to 128Kbps. Now you could say, tough luck WMA for not matching Ogg when it goes to 200Kbps, but you could also say that WMA is better programmed because it stays within its quality range and does not vary wildly. *** these codecs and numbers are totally made up ***

I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-24 23:11:35
My batch file is finished!
Now I'm going to check if Bat2Exe works correctly with it.

/EDIT\
It does! Of course there are still ways to detect which file is which, of course...
But these aren't that easy.
\EDIT/
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-24 23:23:25
Quote
Still have my priviledge

Quote
Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...


The reason VBR exists is that a codec can be advanced enough to lower its bitrate and up its bit rate, but for this to be a fair test - especially with different sample types - for all we know on the harpsicord: ogg will go to an average of 200Kbps, whilst WMA goes to 128Kbps. Now you could say, tough luck WMA for not matching Ogg when it goes to 200Kbps, but you could also say that WMA is better programmed because it stays within its quality range and does not vary wildly. *** these codecs and numbers are totally made up ***

I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

In testing different vbr codecs nothing is ideal, but the current method is hell of lot better than tweaking short hard-to-encode samples forcefully to 128kbps and then try to make some statistical conclusions about that vbr codec's quality which should potray in someway the average quality relative to other codecs.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-24 23:26:00
Quote
I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

Nice idea, to test Vorbis ABR, completely untweaked (I suppose), completely unused (I'm quite sure), slower and probably worse quality...

Or how doing a listening quality test by forcing the best encoders to be worse than they are...


EDIT :


PS : it's mabe time to split this thread. I'm a bit sad to see whole Roberto's work contested here. It's his test, he worked for it - we worked too for him. It's isn't time anymore to contest the methodology. There were a PRE-TEST DISCUSSION for it.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-24 23:33:29
Quote
Reread my post, please.
Modifying the batches won't help - how Roberto will know which file is which?

It has to be set earlier.
Additionally, encoding/decoding order should be 'randomized' (in the batch files, of course).

Sorry, I thought you were talking about a batch file to fix the comma bug.  As far as making the encoding process silent, I wonder if it's worthwhile to insert that in at this point.  What's the point of randomizing the encoding process?

ff123
Title: 128kbps Extension Test - OPEN
Post by: spoon on 2003-07-24 23:35:16
The 128Kbps title of this test is slightly missleading, we will stick with the Harpsichord...for sake of argument if Ogg went to a final Average of 200Kbps whilst all the other Codecs were 130Kbps, it would be fair to say that Ogg would 'win' that one, but what are you telling people? if you had 10 samples to be tested then oggs result would IMHO be 1/10th higher than it should, unless the whole world listens to harpsichord music, which I don't think it does


Edit: Indeed, please split the thread, I have the upmost respect for Roberto
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-24 23:43:21
one time the vbr codecs are discriminated and the other time the abr codecs (the old thing with apples and oranges)...

but i wouldnt know what to think if mpc will be voted on 1-3 place 

let's wait for the results
Title: 128kbps Extension Test - OPEN
Post by: voltron on 2003-07-24 23:47:07
Quote
I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

I suppose the reason ABR was not used where applicable is because not all codecs do ABR. Hence, VBR was used since all the formats tested are able to encode in it.
Title: 128kbps Extension Test - OPEN
Post by: elmar3rd on 2003-07-25 00:06:43
Discussions like this are always necessary for the interpretation.
Title: 128kbps Extension Test - OPEN
Post by: S_O on 2003-07-25 00:15:17
I already finished three samples, with quite suprising results for me, for example that a encoder that is inperceptable for me at one file completly sucks at another.
But I have a problem now: The files at audiocoding.com are not downloadable.
They are not down, but server does not respond.
Because I have unlimited traffic and 10MB free, very fast, space I uploaded the first file. If you don´t have it yet and audiocoding doesn´t work for you, too, download here:
Sample01.zip (41_30sec) (6,93MB) (http://l.b.oltmanns.bei.t-online.de/Sample01.zip)
It would be great if someone else could upload sample02, 03 and 04.
Quote
Hence, VBR was used since all the formats tested are able to encode in it.
Also Quicktime AAC? I thought it´s CBR only.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-25 00:45:54
I had some problems too with the first four packages last hours. My download software can now DL them. Problem seems to be corrected.
Title: 128kbps Extension Test - OPEN
Post by: den on 2003-07-25 01:24:53
@guru, JohnV and others

I appreciate your response, but please don't just curse and link to threads I have already read multiple times. It achieves nothing.

I completely understand the logic behind the choice of codecs settings etc, and I completely understand the use of quality settings in the real world, etc. You should credit the average member of HA with a bit more intelligence. 

I also feel that Roberto has done an excellent job organising a complex public test like this one. My query about bitrate was in no way a reflection on his outstanding efforts and attention to detail.

But at the same time, here we are promoting this as a public, open test, slashdotting here, putting other notices up there, but when the results are interpreted, to many readers the results will be bogus because of the huge descrepancy in the bit rates for a test titles "128k Extension Test". 197k is not 128k in any one's language.

OK, so we should proceed and see how the results turn out, but I can just see that if mpc gets up as the best encoder for example, the vorbis zealots will have an absolute field day attacking the validity of the results, and vice versa.

There are some ways that this concern can be alleviated. First, place a statement clearly explaining that the codecs are VBR, and for some samples, some codecs will exceed 128 k by some margin. Make sure that it is publicly known that the samples are not all the same size. Also, as part of the test results, show the average bitrate for each codec across all the samples, so that at least when people read through the results, they can conclude that a certain encoder may be better for their own needs, but it uses additional bitrate as required, etc. so at least the bitrate bloat for some of these encoders at this setting is not hidden.

Den.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-25 01:58:53
In the ideal, here is how I think a VBR test suite would be constructed:

There would be a mix of difficult and easy samples, and one encoding setting is used for the entire test suite such that when the average bitrate was calculated for all the samples, it worked out to 128 kbit/s.  Of course, this would be after verifying that multiple album encodes yielded 128 kbit/s on average too, also using that same encoding setting.  Also, we would do the "slicing" the sample out after encoding an entire song/album routine.  I think this would get rid of some of the criticism we're seeing here.

Real world:

Probably to achieve the average bitrate of 128 kbit/s, we would have to encode many more easy samples than difficult samples, and the test suite would grow very large while not being significantly better at telling us which codec is better (the idea is that difficult samples are best for discriminating quality differences).

The assumption being made is that the relative codec ranking stays the same with the easy samples as it does with the difficult samples, or at least that all the codecs are transparent if the samples are easy to encode.

But I think that making this assumption is far more reasonable than limiting the VBR codecs to 128 kbit/s by changing the encoding settings per sample.  That's not how the codecs are used in real life, and it's nullifying the purpose of using VBR.


Regarding slicing:

The problem with slicing out samples after encoding is that we'd have to assemble another test suite, where the entire songs are available.  This is actually possible to do, though, so it's just a problem of practicality for this test.

ff123

P.S.  I agree that this whole discussion must be made very clear in the results page, because if there is argument and criticism here, it will be ten times worse on other forums.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 02:18:21
Quote
Why mppenc 1.14 and not 1.15r? As far as I know, it has been very well tested.

Well, I was under the impression that using alphas wouldn't be very safe.

Quote
Ummm, I think we have a problem! 


The joys of VBR...

Quote
Maybe Roberto should put the explanation to his listening test page as well.


Thanks for the idea. I'll do that now

Quote
call it something like hydrogenaudio 128kbps audio codec comparison test


Blah. And since when this is HydrogenAudio's test?

Sorry about being anal here, but I'd rather not have my tests directly connected to any community.

Quote
due to the ","/"." failure in oggenc all .ogg files were encoded with "-q 4" (and not with "-q 4,25")


I don't deserve this >_<

Quote
Ouch.. this is definitely a problem, although not catastrophic. -q4 is officially 128kbps nominal anyway.


Right. Quality shouldn't be much worse either.

Quote
I uploaded modified oggenc.exe that will use proper quality level, it uses recent CVS libraries.


Thanks a lot, Case.

I'll add it to the abc-hr_bin package and reupload it. No point in asking people to retake the test though, IMO.

So, it now just accepts either dot or comma as separator?
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 02:31:10
Quote
I think it should be "post 1.0 CVS".


True. Fixed that on the page.

Quote
some one here who has access to the presentation page to update it with this (not unimportant) info?


Just FYI, people with access to the rarewares account are John33, Dibrom, Ruairi (rc55) and me. You can PM one of them if you need something to be corrected urgently while I'm away.


About the oggenc comma issue:

People, it's only 0.25 (or 0,25  ) difference in the encoded file. I can't be arsed to believe 2-3kbps will make a huge difference.

And as JohnV pointed out, I'm using nominal 128 anyway.


The updated abc-hr_bin package is being uploaded already though.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 02:33:15
Quote
were is rjamorim

I spent the whole day with my GF. It was about time I gave her some attention, didn't do that since I returned from my trip.

Quote
or does anbody else has access to upload an updated package?


One of them is located at Audiocoding. Menno has access to that account.

The other one is being hosted by Paul Harris (verloren). Also, John33 has access to that account.
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-25 02:40:34
Quote
Just FYI, people with access to the rarewares account are John33, Dibrom, Ruairi (rc55) and me. You can PM one of them if you need something to be corrected urgently while I'm away.

*cough*  I happen to have access too..
Title: 128kbps Extension Test - OPEN
Post by: Case on 2003-07-25 02:41:16
Quote
So, it now just accepts either dot or comma as separator?

This fix isn't as advanced as the one I gave to oggenc author when the issue was originally found some months ago, I now just removed locale changing for the encoder and it only accepts dot.
Title: 128kbps Extension Test - OPEN
Post by: phong on 2003-07-25 02:43:46
I just got started.    Poor blade, it's like a grizzly bear trying to do ballet.  :x  I hope nobody comes to my apartment and sees me locked in the bathroom with extension cords running under the door.  Unfortunately I can still hear the cheetah in my Windoze box seeking, even through the wall.  >_<

I agree an explanation about the VBR up front (before results are compiled) will be key to appeasing the skeptics.  All the codecs used settings that produce an average of about 128kbps over a large number of albums (and you've got the data to back that up.)  VBR codecs can't be punished for spending those bits where they'll do the most good.  Unfortunately, with the festering hole Slashdot has become, there will still be plenty of people who won't accept the results no matter how carefully the test is conducted.  The recent thread on the AAC test was almost entirely flaming by people who either didn't read or didn't understand the test.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 02:48:29
@JohnV: Sorry, I forgot you got root access.

Quote
But I have a problem now: The files at audiocoding.com are not downloadable.
They are not down, but server does not respond.


Audiocoding is actually sourceforge. Their servers aren't very reliable, but I would be a bastard if I complained about something I get for free. Please, just try again later.

Quote
OK, so we should proceed and see how the results turn out, but I can just see that if mpc gets up as the best encoder for example, the vorbis zealots will have an absolute field day attacking the validity of the results, and vice versa.


Ok, here's my take on the "I think your test is wrong" I expect some critics to come up with. As you guys know, I'm a very blunt person

[span style='font-size:12pt;line-height:100%']I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.[/span]

Of course, I accept constructive criticism. But I will really ask people that come to me saying my test sucks without providing a better way to do it to go fuck themselves. I actually learned to be blunt from my experiences with the AAC test. :B

Quote
There are some ways that this concern can be alleviated. First, place a statement clearly explaining that the codecs are VBR, and for some samples, some codecs will exceed 128 k by some margin.


Almost finished
Title: 128kbps Extension Test - OPEN
Post by: den on 2003-07-25 02:55:21
Quote
Ok, here's my take on the "I think your test is wrong" I expect some critics to come up with. As you guys know, I'm a very blunt person

I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.


I was about to post, saying thank you for responding to my initial concern a little more calmly than some others... 

Quote
Almost finished 


Excellent. I think that should do it. Time to get on with the testing.  B)

Thanks again Roberto.

Den.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 02:57:59
I added this text to the presentation page:

"I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate"

Any remarks?
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 03:02:51
Quote
I was about to post, saying thank you for responding to my initial concern a little more calmly than some others...

Well, that wasn't targeted at you. I was just giving people a sample of what awaits clueless critics.




edit: Just uploaded new abc-hr_bin.zip packages using Case's oggenc (thanks!), all should work as planned now. :B
Title: 128kbps Extension Test - OPEN
Post by: den on 2003-07-25 03:25:48
Quote
Any remarks?


Sweet. Explains the situation without apologising for it.  B)

I know it's early days yet, but some comments about the same, and reported bitrates would be useful when the test results are published too, but I'm sure you're already onto that!

Quote
Well, that wasn't targeted at you. I was just giving people a sample of what awaits clueless critics.


 

Den.
Title: 128kbps Extension Test - OPEN
Post by: phong on 2003-07-25 04:15:54
Quote
Any remarks?

I would add that the quality settings chosen for the VBR codecs were chosen because they average out to about 128kbps over a number of encoded albums.  And to pick a nit, I might say "fairer in one sense" instead of just "fairer' and would add that it would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.

BTW: Let me also say thank you for running this test.  You rule.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 04:31:37
Added your comments (actually, did some copy-pasting  hope you don't mind).
Thanks a lot for your remarks, and your kind words.
Title: 128kbps Extension Test - OPEN
Post by: KikeG on 2003-07-25 09:09:29
About this VBR issue: a similar thing happened with the 64 kbps test. Some people said that that was not the way of testing codecs, but it was just because they didn't understand the issue.

Test designers are not responsible for people being dumb. You can explain why VBR codecs have been used this way. If some people still don't agree with explanation, it's their problem. The only "but" I can accept is that for the type of music someone listens, the average VBR bitrate is higher than 128 kbps. People complaining should provide their VBR average bitrate to support their complaints.
Even when that can happen in some particular cases, it's a particular problem impossible to overcome with a general test, which is still much better than nothing.
Title: 128kbps Extension Test - OPEN
Post by: spoon on 2003-07-25 09:21:36
After some thought: add up all the average bit rates for each of the samples and lets say for arguement that one codec comes out at 128Kbps  and the next 256Kbps, the results of the 256Kbps codec should be scalled back by 50%. Present both sets of results with an explination. If all the Kbps comes within say 5% of 128Kbps then there wouldn't really be a need to adust the figures.
Title: 128kbps Extension Test - OPEN
Post by: Gabriel on 2003-07-25 09:38:59
The currently used test samples are note necessarily representative of overall music. They are chosen test samples. What is important is that the overall bitrate for overall music, using an encoder setting, would be 128kbps on average.

If the average bitrate of those test samples for a specific codec is 180kbps, it is still fair is the overall bitrate for overall music is 128kbps. What a user wants is 128kbps as an overall. A user does not care if 10 seconds of his track are encoded using 250kbps while the next 10 seconds are encoded using 90kbps, as long as overall it is still 128kbps.

regarding vbr codecs vs abr/cbr codecs:
As a Lame developper, I know that Lame in abr mode will be penalized in this test against, as an example, Vorbis which is a vbr codec.
But (unfortunately for Lame) this is perfectly fair as it is representative of codecs abilities.
Lame has currently no high quality vbr mode averaging 128kbps, but this is not a point that should be used to penalize other codes which have such a mode. We should not contrain codecs to use the lowest mode just because some of them are not able to use better modes.
Title: 128kbps Extension Test - OPEN
Post by: bawjaws on 2003-07-25 09:55:22
Quote
I added this text to the presentation page:

"I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate"

Any remarks?

I think the problem some people have with this is they think the VBR codecs are cheating by going higher than 128kbps because they are focusing too tightly on the test alone. You may need to broaden their perspective.

I think it might make sense to people if you explain that the VBR codecs are being rewarded for the savings they make in the easier to encode sections where the 'less intelligent' CBR encoders are wasting bits.

If the bitrate for a given quality level averages 128kbps in ordinary use but is variable then obviously it must go both higher and lower than 128kbps. The reason you don't test on the sections that go lower is because those are the parts that the codec finds easy and so you would be less likely to spot the flaws in the encoder.

Just another angle on this obviously un-intuitive point.
Title: 128kbps Extension Test - OPEN
Post by: askoff on 2003-07-25 10:45:00
Does WMA9 include free VBR mode?
Title: 128kbps Extension Test - OPEN
Post by: danchr on 2003-07-25 10:52:47
A small pat on the back for Roberto:
A search for "listening test" on Google reveals "Roberto's public listening tests" as the best match.

That means there are plenty of links to it out there
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-25 12:01:08
Quote
I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

i wouldnt write it that way as it will make people think that there are many files which are around 200, which isnt the case
i wouldnt write 200kbps also, i would only write:
"I noticed that, on some files, bitrate can go up pretty high. Is that right?"

Quote
Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.

i wouldnt write that, as talking about that is "beautifull" that codecs ignore bitrate goals (whenever possible) isnt really an argument against people who claim that the vbr codecs "are cheating" imho

Quote
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate

1) now this sounds like you think for yourself, that the test is unfair, but also that it has to be unfair as the vbr codecs "shine the most in VBR mode". this is also no argument against people claiming imho
2) i wouldnt tell the reader about his behaviour as everybody behaves different (for example i think that the usual practice in audio encoding for the masses is that people care to stay around a specific bitrate on the long run)

Quote
quality settings for each codec were chosen because they average to 128kbps over a number of encoded albums. It would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.

now this is the most important part! and the only part which should be mentioned on the homepage imho!
as it says that

the vbr codecs will reach, with this settings, normally the average of 128kbps!!!

that's the argument that counts (no need to talk about fairness or user behaviour or the beauty of vbr codecs)
like what gabriel wrote:

Quote
What is important is that the overall bitrate for overall music, using an encoder setting, would be 128kbps on average.

If the average bitrate of those test samples for a specific codec is 180kbps, it is still fair is the overall bitrate for overall music is 128kbps. What a user wants is 128kbps as an overall. A user does not care if 10 seconds of his track are encoded using 250kbps while the next 10 seconds are encoded using 90kbps, as long as overall it is still 128kbps.


sorry for my bad english
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-25 12:02:26
so all in all i would write it that way:

Q: I noticed that, on some files, bitrate can go up pretty high. Is that right?

A: Yes, this can happen for samples with hard to encode content, and this is wanted that way as vbr codecs are able to put more bitrate where it is needed and less bitrate where it is not needed that much.

The settings for each codec were chosen because in the long run they average to 128kbps!
[/li][/list]
nothing less nothin more (a short but clear statement), no need to talk about 200kbps, about fairness, defending vbr codecs against anything, about how users behave (everybody has different opinions here)...

in the long run every codec setting averages to 128kbps - thats the only important message!

(and i wouldnt put it as question 1, as this is the presentation page where people can read how to participate, i would put it as question 5 (after linux) no need to push people on this info  )
Title: 128kbps Extension Test - OPEN
Post by: ilikedirtthe2nd on 2003-07-25 12:39:31
sorry for being off-topic here, but: why were nero/psytel forced to cbr 128 in the prior aac test? the arguments given for this test are also suitable for the aac test. nero -streaming also reaches an average bitrate of 128 on long distance (i suppose).

if you put this into relation, ogg vorbis should have been forced to cbr in this test.

regards; ilikedirt
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-25 12:50:47
Heh, this arguing is of no use

Rjamorim, have you taken my batch file to hide the results?
You'd need to modify and reupload all packages, but cheating in the test won't be as easy.
Title: 128kbps Extension Test - OPEN
Post by: ilikedirtthe2nd on 2003-07-25 13:00:37
Quote
Heh, this arguing is of no use

i am not arguing against this test here, but maybe we are testing against the wrong aac encoder? maybe psytel -streaming would have won the prior competition if it were allowed to show the beauty of it's vbr algorithms 

please don't hit me   
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-25 13:06:21
ilikedirtthe2nd
when you read the test again you will find the info that guruboolez (sorry if i misspelled the name) also tested the nero vbr options and came to the conclusion that quicktime is still better!
you'll also find more info about that on the aac test results page http://rarewares.hydrogenaudio.org/test/aa...st/results.html (http://rarewares.hydrogenaudio.org/test/aac128test/results.html) and especially here: http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/tableau_vbr.txt)
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-25 13:54:05
Oh, when I can continue the test not afraid I'll need to redo it?
Title: 128kbps Extension Test - OPEN
Post by: loophole on 2003-07-25 14:01:33
At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be. Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

If you're against forcing Vorbis to use managed bitrates, maybe you should have encoded them at a given quality level (say they averaged 190kbps), then used the 192kbps setting for the ABR codecs. Seems fairer to me.

Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)

It just does sound unfair to the ABR codecs to me. It seems to me like you're testing the accuracy of the bitrate engine, not which codec sounds better for any given bitrate.

Taken to the extreme, codec x in VBR mode y could allocate whatever it wants, and end up an average of 300kbps but that would be a fair comparison because it's engine was smart enough to allocate more bits. </SARCASM>
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-25 14:17:49
We're testing quality of settings averaging 128kbps on multiple different albums.
Not on short samples. That's all. LAME and Quicktime are using ABR,
because they either don't have VBR modes or VBR modes perform worse than ABR.
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-25 14:18:52
Quote
At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be.

How do you know that it's 50% exta of what the original bitrate was targeted to be, you are testing just short hard-to-encode clips. The targeted quality setting tested produces 128kbps average, and we are testing the quality of specific quality setting of a vbr codec which gives this average bitrate. We are not testing qualities of 12 different quality settings of one vbr codec in one test.

Quote
Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.
Like both Gabriel and me have said in this thread already, so this is prolly 4th time in this thread alone: There's no high quality low bitrate Lame vbr setting. ABR mode in Lame averaging 128kbps performs better.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-25 14:28:58
Quote
At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be. Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

Lame uses ABR because it's generally agreed that ABR sounds best at bitrates which average about 128 kbit/s.  Vorbis uses -q (VBR) because that's the mode it generally sounds best using.

Quote
If you're against forcing Vorbis to use managed bitrates, maybe you should have encoded them at a given quality level (say they averaged 190kbps), then used the 192kbps setting for the ABR codecs. Seems fairer to me.


If one uses the Vorbis quality mode, and for a particular sample it averages 190, then the ABR codecs shouldn't be bumped up to 192 for that sample -- that's exactly the same thing as forcing the VBR codecs down to 128, except in reverse!

Quote
Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)


The purpose of the last test was to find the AAC codec to be used for this test.  Final Cut Pro 4 wasn't tested.

Quote
It just does sound unfair to the ABR codecs to me. It seems to me like you're testing the accuracy of the bitrate engine, not which codec sounds better for any given bitrate.

Taken to the extreme, codec x in VBR mode y could allocate whatever it wants, and end up an average of 300kbps but that would be a fair comparison because it's engine was smart enough to allocate more bits. </SARCASM>


The VBR codecs in this test are quite stable in average bitrate on a per-album basis.  If a VBR codec can manage to produce 128 kbit/s on average, but ends up with 300 kbit/s on a difficult sample, it should not be penalized for doing its job!  That's why VBR exists.  The caveat to this is that the VBR codecs should not sound worse on "easy" samples.

ff123
Title: 128kbps Extension Test - OPEN
Post by: upNorth on 2003-07-25 14:53:16
This reminds me of discussions about "optimized" command lines for encoding, that people that don't know the first thing about compression come up with. Especially the ones that "improve" the -presets. How come people trust the developers (and the people helping them) to develop the codec, but not at all when it comes to using it. It makes no sense to me and I hope this test will be performed as intended and not in an artificial and "fair" way. Everything always seem very easy when you don't know nothing about it...
Quote
I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.
B)
Thanks for arranging this test and keep up the good work 

edit: typo
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-25 15:46:32
I've completed 9 samples, and I'm tracking my preferences:

http://ff123.net/friedman/stats.html (http://ff123.net/friedman/stats.html)

(Blocked ANOVA analysis)

The losers don't surprise me, but my apparent preference does (although it's not clearcut by any means).

ff123
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 16:29:14
Quote
Thanks for arranging this test and keep up the good work 

Thanks
Title: 128kbps Extension Test - OPEN
Post by: atici on 2003-07-25 18:05:08
Maybe ATRAC3 LP2 mode could've been added to this test too. And we could have referred to the results every once in a while MiniDisc zealot shows up in HA
Title: 128kbps Extension Test - OPEN
Post by: Gecko on 2003-07-25 18:19:32
Quote
Does WMA9 include free VBR mode?

After reading this thread, that's what I thought. If you use 2 pass you are forcing the encoder to a specific bitrate. If WMA9 Pro had a proper VBR implementation, it would maybe have increased the bitrate on critical samples as well.

This thread shows one thing: correct testing of VBR codecs, especially when mixed with cbr/managed bitrate ones, is all but trivial. Maybe this should be emphasized on the main page and then quickly explain why this method was chosen.

Here's an idea for future tests. Instead of using regular difficult test samples, you would use samples that all come out at ca the same bitrate. That would make the testfield more leveled and you are testing what a codec can do at a certain bitrate without tweaking.

But of course it will be tedious to find samples that will produce nearly identical bitrates over a range of codecs and you will probably end up with samples that are easy to encode and harder to identify by ear.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 18:30:05
Quote
and you will probably end up with samples that are easy to encode and harder to identify by ear.

That's exactly the biggest issue.

Quote
Maybe ATRAC3 LP2 mode could've been added to this test too. And we could have referred to the results every once in a while MiniDisc zealot shows up in HA


Problem is, I already had my share of bad issues with SonicStage (nearly fuxored my Win2k installation), so I'll only test Atrac3 once I create a VirtualPC partition only for installing and running it. :-P
Title: 128kbps Extension Test - OPEN
Post by: atici on 2003-07-25 18:31:26
Quote
Here's an idea for future tests. Instead of using regular difficult test samples, you would use samples that all come out at ca the same bitrate.


I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 18:38:40
Quote
I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...

JohnV will shoot you
Title: 128kbps Extension Test - OPEN
Post by: JohnV on 2003-07-25 18:44:00
Does anybody here read what ff123, Gabriel, Guruboolez and I are writing in this very thread at least 5 times each so far?  I can't BELIEVE THIS!!

(http://static.hydrogenaudio.org/uploads/av-1367.jpg)(http://www.handykult.de/plaudersmilies.de/rough/camper.gif)
Title: 128kbps Extension Test - OPEN
Post by: upNorth on 2003-07-25 19:19:16
Quote
I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...

What would this prove?
The beauty of a quality setting is that it's the bitrate that changes and not the quality. I still think that what matters, is what kind of bitrates a certain codec at a certain setting, average to in the long run.

At least when I encode music I settle for a specific quality setting (currently MPC -q5) and not a bitrate. Do you really test every track to find the setting that is closest to your desired bitrate, and thereby end up with an album where "all" the tracks are encoded with different settings, just so that all of them has about the same average bitrate? It's fine by me if this gives others the warm fuzzy feeling everyone here talk about, but I don't get it. 

As I see it, a test like this has to settle for some interesting samples to put to the test because you have to limit the amount. I would expect the easiest to spot problems parts of a certain track also might be the same places where a good VBR codec truly shines. If you force it not to be smart and use bits where it thinks they should be used, your test IMHO won't be much worth.

I don't say that this is easy at all, but at least when it's done in this way it applies to real life usage. I don't really care what these Slashdot people say, it's only fun to read it...
Title: 128kbps Extension Test - OPEN
Post by: verloren on 2003-07-25 20:17:06
I have almost the opposite question to most people.  I understand and totally agree with the reasoning that roberto has used for setting the various quality levels.  But from the accounts listed here it seems like there are many samples that are way above the 128kbps nominal value, but few if any that are significantly below (the examples I've seen have been around 122kbps for example).

So I wonder if it would be useful to give some really easy to encode samples, to make sure that when the encoder decides it only needs say 50kbps it is making as good a decision as when it picks 190kbps for a hard passage.

And no, I haven't downloaded the samples as I lack the facilities, so perhaps this is already in there!  If so I claim "official mirror's" right to ask one stupid question

Cheers, Paul
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-25 20:39:03
Quote
I have almost the opposite question to most people.  I understand and totally agree with the reasoning that roberto has used for setting the various quality levels.  But from the accounts listed here it seems like there are many samples that are way above the 128kbps nominal value, but few if any that are significantly below (the examples I've seen have been around 122kbps for example).

So I wonder if it would be useful to give some really easy to encode samples, to make sure that when the encoder decides it only needs say 50kbps it is making as good a decision as when it picks 190kbps for a hard passage.

And no, I haven't downloaded the samples as I lack the facilities, so perhaps this is already in there!  If so I claim "official mirror's" right to ask one stupid question

Cheers, Paul

I would guess that the high and low bitrates are not distributed the same way.  For example, if a codec spends 90 percent of its time at 124 kbit/s, then 10 percent of the time it could grow to 165 kbit/s while still averaging 128 kbit/s overall.  It could be that the VBR codecs never let the bitrates dip down to the extent that they're allowed to increase.

If that's the case (probably a reasonable assumption), and a test suite were to be comprised completely of random samples of music (not chosen at all for degree of difficulty), then most of the time it might be completely transparent to the listeners, and basically useless for trying to discriminate between codecs because of the large number of samples which would be required to simulate a real-world music collection.

One type of music which seems to produce lower bitrates is solo piano.  Roberto mentioned in his first test that he removed this from the test suite because it didn't discriminate well on the 64 kbit/s test.  This implies that the codecs would indeed be transparent at lower bitrates.  Still, maybe one or two "very easy" to encode samples, which produce lower VBR bitrates might be a good thing to include in a future test just to make sure one of the VBR codecs isn't failing badly at those bitrates.

ff123
Title: 128kbps Extension Test - OPEN
Post by: askoff on 2003-07-25 21:00:06
Quote
How do you know that it's 50% exta of what the original bitrate was targeted to be, you are testing just short hard-to-encode clips. The targeted quality setting tested produces 128kbps average, and we are testing the quality of specific quality setting of a vbr codec which gives this average bitrate. We are not testing qualities of 12 different quality settings of one vbr codec in one test.

This is quite odd. Even in the name of this topic clearly says "128kbps Extension Test" and nothing about "128kbps average album quality extension test clips". Well what the hec i'm whining about this subject anymore. I gues nothing will be changed anymore in this test, so i just have to do this how it has been started and try to test later in my way with my self. And why not setup my own public test. After all this is only Roberto's test, and it's as official as anyone else public test. Not the only official test.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-25 21:42:25
If anyone is willing to create his own listening test, I would be very happy to help him set it up. I'm pretty sure ff123 would also be happy.


And no, this test won't be changed. It works the way it is, and lots of people already took it, I won't ask them to retake (specially since there's no reason, really).
Title: 128kbps Extension Test - OPEN
Post by: rpop on 2003-07-25 21:54:15
To all the people whining: where were you during the pre-test dicussion? Don't you think these comments would've been more appropriate and helpful then?
Title: 128kbps Extension Test - OPEN
Post by: verloren on 2003-07-25 21:59:29
Quote
I would guess that the high and low bitrates are not distributed the same way.  For example, if a codec spends 90 percent of its time at 124 kbit/s, then 10 percent of the time it could grow to 165 kbit/s while still averaging 128 kbit/s overall.  It could be that the VBR codecs never let the bitrates dip down to the extent that they're allowed to increase.

Thanks for the response ff123, that sounds very plausible.

Cheers, Paul
Title: 128kbps Extension Test - OPEN
Post by: westgroveg on 2003-07-26 02:03:43
Quote
Still have my priviledge

Quote
Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...


The reason VBR exists is that a codec can be advanced enough to lower its bitrate and up its bit rate, but for this to be a fair test - especially with different sample types - for all we know on the harpsicord: ogg will go to an average of 200Kbps, whilst WMA goes to 128Kbps. Now you could say, tough luck WMA for not matching Ogg when it goes to 200Kbps, but you could also say that WMA is better programmed because it stays within its quality range and does not vary wildly. *** these codecs and numbers are totally made up ***

I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

The way I see it WMA would have failed to adapt & keep the selected QUALITY level.
Title: 128kbps Extension Test - OPEN
Post by: loophole on 2003-07-26 08:28:40
Quote
Quote
Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)


The purpose of the last test was to find the AAC codec to be used for this test.  Final Cut Pro 4 wasn't tested.

Final Cut Pro (by Apple) actually leverages QuickTime (also by Apple), it just displays a different interface which seems to allow VBR modes.
Title: 128kbps Extension Test - OPEN
Post by: ezra2323 on 2003-07-26 19:22:46
Roberto - just curious, whay was WMA 9 not included, only the PRO version? Very few people here use WMA to begin with, and those that do are most likely using it because it has excellent hardware support. WMAPRO does not. I have tried to load these files on to my WMA compliant portables and they are not recognized.

However, I would be very interested to see how 2 pass VBR 128 WMA (not the professional version) stacks up against the competition since this is a very popular format with the new legitimate music sites popping up. Yes, I know they likley are using WMA CBR 128, but could probably be convinced to swithch to 2 pass 128 VBR if the quality gain was sufficient.

It would be interesting to see how the WMA offerings stack up against Apple's AAC offering.
Title: 128kbps Extension Test - OPEN
Post by: ezra2323 on 2003-07-26 19:24:04
BTW - not questioning the test, I think its great! Just requesting WMA (regular, not pro) be added
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-26 21:06:05
A public blind test can't include too much encodings. It will make the notation (and hierarchy) harder...
I did few tests between WMA std and WMA pro (one, on the 12 samples, is available on the OTHER AUDIO FORMATS section). In my mind, the gap between STD and PRO encoder is consequent. Too consequent maybe for keeping any hope on standard codec quality, against stronger formats. On the other side, WMApro is more mysterious. No test are available. No mention on its quality on Hydrogenaudio. How good is it ? Can this new format, created and supported by a giant, compete with Goliath MP4 or David Vorbis, in quality term ? We had to let this format a chance, and to test it, against the best challengers of the moment.

This test include the best formats available, and for each of them, the best codec at the best setting. Only exception (easy to understand) : mp3. It would be interesting to include wma standard format, but then, why not atrac3 ? Fraunhofer Fastenc VBR ~128 ? VQF 2.0... As I said before, a public test couldn't include too much challengers. Some choice were made, with dialogue. IMO, Roberto did the good one. Other people will be disappointed. That's life...

Nevertheless, if you're interested by wma standard performance, you can easily include yourself some encodings in each downloaded package.


About hardware support : I give more chance to WMApro to be widely support in the next two years on DVD/Portable than to vorbis. I hope to be wrong...
Title: 128kbps Extension Test - OPEN
Post by: phong on 2003-07-27 21:30:49
Quote
Added your comments (actually, did some copy-pasting  hope you don't mind).

Honored.  :)

Quick question...  On a couple of the samples, I'm not having too much trouble abxing most or all of the codecs, but others are much harder for me (no surprise obviously).  If time is a limiting factor (I may only be able to set aside a small amount of time this week), which of the following would you prefer people to do:
a) A very careful analysis of a few of the samples, making every reasonable effort to distingush as many from the originals as is possible with their equipment and ears.
B) Try to do all 12 samples, at the expense of a few of the best encodings getting rated as "perfect" where more careful analysis would reveal some minor audible defects in some of them.
c) If you can't do your most careful analysis of all 12 samples, don't bother submitting results at all.
d) Whatever floats one's boat.  Have fun and don't stress too much.

From my interpretation of the readme, I doubt c) is the case.  :)  If a) is prefered, do you have a prefered method of chosing the samples?  Go down the list in order?  Pick randomly?  Do the easiest ones, thereby providing the most possible discriminating data?
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-27 22:19:12
Quote
From my interpretation of the readme, I doubt c) is the case.    If a) is prefered, do you have a prefered method of chosing the samples?  Go down the list in order?  Pick randomly?  Do the easiest ones, thereby providing the most possible discriminating data?

Yes, c) definitely isn't the case.

It's really a matter of whatever floats your boat. Both a) and B) suit me well. And if you decide to go with a), I suggest taking files randomly. If people only do going through the order, I'll have too many 41_30sec samples and maybe too few of others, as happened on the AAC test.

Thanks for participating.

Regards;

Roberto.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-27 22:20:44
Damned be that stupid B) smilie.

Couldn't some admin please replace it with :cool: or something?
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-28 17:16:55
Hello.

Tonight I'll have to pull the files from Paul's mirror because it's reaching the bandwidth consumption limit (http://audio.ciara.us/stats/) of 5Gb.

So, I would like to ask if someone with a reasonably fast server could spare me some 20Mb and some few Gb of bandwidth so that I can keep the files there until sunday. It consumed 5Gb these first 5 days, so I expect to consume as much until the end of the test.

If you can, please PM or mail (http://pessoal.onda.com.br/rjamorim/mail.gif) me. You don't even need to give me login/password, just upload the packages to your server and send me the addresses.

Thank-you very much;

Roberto.
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-28 17:30:00
I'm uploading the samples now to

http://ff123.net/128exten/Samplexx.zip (http://ff123.net/128exten/Samplexx.zip)

I have about 12 GB bandwidth to spare.

I have to leave for work, but they should be uploaded within the next half hour or so.  I'll verify that sample12.zip uploaded properly from work.

ff123

Edit:  changed the path
Title: 128kbps Extension Test - OPEN
Post by: verloren on 2003-07-28 17:31:19
I've also made some more space available to Roberto - who knew it would be quite this popular!  I'm sure he'll let you know the details if he decides to use the space.

Cheers, Paul
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-28 17:37:03
Wow. Thanks a lot, both of you
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-28 17:43:16
I'm happy to set up some space on my own server as a backup mirror. I have, well, virtually unlimited bandwidth to spare on my gigabit uplink

If needed just contact me.

Also one question, I've read most of the comments here but didnt see it (I think)

Why not penalize 'naughty' codecs (say ogg using -q4 generating a 190kbit file) by just multiplying its end score for a particular file with 128/190? Sounds reasonable to me but maybe I'm overlooking something big here. In the end I 'respect' that a codec rightly thinks it needs to use 190kbit for a particular piece to keep the 'Quality level 4' there, but indeed when comparing it to a codec that is perhaps just as advanced but less frivoulous with bitrate allocating isnt fair.

"Yay, vorbis wins all tests by allocating 256kbit continuous for all samples)

p.s. just using vorbis as an example, feel free to replace that word with <your hated codec name here>

(uh for the record (people are using the term 'bandwidth' kinda loosely here).. I can easily deal with sending out 100Gb of data over a week. I can not deal with sending out 1Gb/second  )
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-28 18:33:57
Sorry, but the test is frozen. JohnV will eat you alive! (either he or his tiger)
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-28 18:47:09
Quote
Sorry, but the test is frozen. JohnV will eat you alive! (either he or his tiger)

Ah but Im not asking to change the test

I am just suggesting perhaps it would be interesting to see if the outcome of the test would differ in any meaningful way using the 'puntloos audio correction factors' 
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-28 22:27:42
Quote
Why not penalize 'naughty' codecs (say ogg using -q4 generating a 190kbit file) by just multiplying its end score for a particular file with 128/190? Sounds reasonable to me but maybe I'm overlooking something big here.

Why "penalize" ? A VBR codec had to put more bits on complex signals. The 12 samples are complex, difficult : it's nonsense to expect an average bitrate close to 128 kbps, and completely stupid to punish a VBR codec for doing correctly his job.

I'm happy to see (or hear) mpc putting ~700 kbps frame on complex signals like castanets. I was glad to use --preset standard, introducing a lot of 320 kbps frame when needed, and compensate it on quiet/easy part. Every people here are using great format or setting. Each are VBR : that mean consequent variations, but at the end, an average bitrate -the same for most albums. Is there one reason to applaud on their qualily for listening purpose, and blame or punish them for testing ?
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-28 22:30:12
am i right that the test still hasnt been announced on slashdot?
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-28 22:57:58
Quote
Quote
Why not penalize 'naughty' codecs (say ogg using -q4 generating a 190kbit file) by just multiplying its end score for a particular file with 128/190? Sounds reasonable to me but maybe I'm overlooking something big here.

Why "penalize" ? A VBR codec had to put more bits on complex signals. The 12 samples are complex, difficult : it's nonsense to expect an average bitrate close to 128 kbps, and completely stupid to punish a VBR codec for doing correctly his job.

I'm happy to see (or hear) mpc putting ~700 kbps frame on complex signals like castanets. I was glad to use --preset standard, introducing a lot of 320 kbps frame when needed, and compensate it on quiet/easy part. Every people here are using great format or setting. Each are VBR : that mean consequent variations, but at the end, an average bitrate -the same for most albums. Is there one reason to applaud on their qualily for listening purpose, and blame or punish them for testing ?

Oh but as I said: I agree with a good VBR codec allocating HIGH amounts of bits to complex pieces, no problem there.

My dad taught me to always think in extremes when it comes to physics, so:

Suppose we have a piece with ONLY castanets? And MPC at -q4 (say) would create a 700kbps average file. Would you consider it fair to compare that file to (say) Vorbis that has encoded the same castanets file to 140kbps? Youpi! MPC file sounds better!! 

My point therefore is that even though VBR is a very valid way to encode music, and I have no problem at all if some codec I use goes above the 'indicated bitrate' if it feels it needs to. But when comparing these results I think a -certain- penalty must be given to the 700kbps output file. Im sure you agree that comparing the QUALITY of a file that averages at 700kbps with a file averaging at 140kbps isnt fair and will give skewed results when you try to determine the 'best codec'. I'm not a mathematician so Im not sure if my way of unskewing these results (multiply the 'score' of the 700kbps file with 128/700) is completely fair, but at least it will give results that closer matches 'fairness' in my mind.
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-07-28 23:17:12
this discussion will never end 

once again: the used codec settings (of course also the vbr settings) will average to around 128kbps most of the time when you encode full music clips...
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-28 23:20:24
Quote
Suppose we have a piece with ONLY castanets? And MPC at -q4 (say) would create a 700kbps average file. Would you consider it fair to compare that file to (say) Vorbis that has encoded the same castanets file to 140kbps? Youpi! MPC file sounds better!! 

Yes. In my opinion, it's fair.

The only exception is for people listening castanets -and castanets only- the entire day.
I'm listening a lot of harpsichord. This imply that I know that :
- many encoders are not able to encode it properly at low-mid and even high bitrate
- many VBR encoders will increase statistically mesured bitrate for this instrument.

I perfectly aware that mpc --radio can't match the 128 kbps target. Even --thumb, in many case. Vorbis is in the same case. Then, it would be fair too balance their results.
On the other side, in many discs of mine, solo harpsichord can be heard some seconds only ; average bitrate of these discs is close to 130/140 kbps, even with ponctual 200 kbps parts. Quality is great during the entire disc. But if small parts are totally destroyed by the encoder (mp3 for exemple), I can't enjoy the whole encoding.


Roberto's test is a general one. We can't consider some particular tastes or listening behaviour, and therefore, there's no need to play with some formula. For the great majority of discs, mpc --radio and vorbis -b 4,25 are close to 128 kbps. And all these common discs include complex part as bachpsichord, castanets, Atrain... Lowering performance of VBR codec won't be fair at all.
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-28 23:46:54
No it isn't fair; that's my point 

Look, consider this: My hypothetical castanets file will be (say) 700kbps in mpc at the settings Roberto chose. Anyone who is -surprised- at that castanets.mpc sounds better than castanets.ogg (140kbps) should stop taking drugs.

Now. This says one of 2 things:

1/ The test for 'castenets' is useless. We know the result before listening.

2/ The results should be skewed to make the result fair: Yes, castanets.mpc sounds better but at 5x the bitrate so it needs to be corrected for this The Real result needs to be extrapolated from the choices the codec makes.

Now the castanets or harpsichord examples are -extremes- but still.. consider the hypothetical evil .aac codec maker sent Roberto a aacenc.exe program that adds +2 to the quality setting Roberto told it to use. Roberto wants Quality4? Well I'll make a Quality6 file. Then AAC files would of course always be more like 192kbit, but hey, thats Fair cause VBR (which is, in the end, a mix-bag of codec choices made by humans!) allocates as much bits as it Needs to

See where I'm going with this? evil-AAC wins, by your rationale.

And, it shouldn't win (because of this)-  the results simply have to be skewed, and a size penalty must be given to correct for larger filesizes! I don't want to know which codec starts using an insane amount of bits. I want to know which codec Makes the most efficient use of bits it gets at bitrate 128kbit. THAT is an adequate test result.

Now, if you guys say that in the end, the 'ogg package' of files is exactly the same size as the 'aac' and 'mpc' package of files after encoding, then this already means that 'globally' the test might be fair, the overall quality of a codec then will be measured.

But I submit that the packages probably aren't the same size, and even if they are, comparing them on a file-by-file basis is more informative In that case we can clearly see which codec YOU should use. If you like harpsichord pieces, you will want to know which codec is best at harpsichord pieces! You don't want to know that codec X is best overall only to find out that it is best at everything except harpsichord

Hu..hu.. Done with my rant

Anyway, noone is stopping ME from making a corrected results table. But honestly, I really think my points could be valid here.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-29 00:00:58
I'm KO. Not by your logical, but by my poor english. I can't express what I have in mind.

Anyway, balanced-results or adjusted-settings, performed in order to make thing 'more fair', are for me exactly the same thing. Pardon me, but I'm a bit bored to see, on each page, the same criticism and some variation on this theme.
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-29 00:12:45
Quote
I'm KO. Not by your logical, but by my poor english. I can't express what I have in mind.

Anyway, balanced-results or adjusted-settings, performed in order to make thing 'more fair', are for me exactly the same thing. Pardon me, but I'm a bit bored to see, on each page, the same criticism and some variation on this theme.

No problem - Thank you for trying, maybe I just am missing the obvious point and need some sleep

If people don't agree with me for the next test I will write the PuntCodec™ Lossy Codec (it only loses the .wav tags). It will accept the same quality settings as ogg/aac/mpc and it will *cough* aim[/i] at around 128kbit for -q4, 192kbit for -q6. Only no matter what setting you give it, it will copy 1-on-1 the exact wav data from input to output giving you a VVBR (VeryVariableBitRate) file of oh, ah, 1400kbit/sec.

And when decoding: NO ARTIFACTS! Perfect Codec. PuntCodec™ wins! 
(maybe for v2 I will steal Flac sourcecode! at least 60% compression! Still we win!)
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-29 00:25:45
Sorry, but there was testing to find appriopriate quality setting before.
Your codec wouldn't pass it.

/EDIT\ You really need to get some sleep.
I've written about it at least few times it this thread.
But this time the text is at least amusing. Well done! \EDIT/
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-29 00:31:39
Quote
Sorry, but there was testing to find appriopriate quality setting before.
Your codec wouldn't pass it.

Ratz. Well in that case I would send them an ogg vorbis codec to -test-, and send them the latest optimized alpha *evil laughter* when the test is about to start.

For the record of course Im not accusing any codec maker to cheat like me  but I AM saying that for example I have noticed (havent experimented much I admit) MPC to create larger files than ogg at any quality setting. THIS fact does -not- make MPC a better codec. It makes MPC a codec that creates larger files.

And as I said before: filesize doesn't concern me, the quality/bitrate quotient concerns me. Puntcodec@1400kbit winning everything is not interesting. AAC@100kbit creating a slightly better sounding sample than .ogg@105kbit would be, and mpc@110kbit sounding better than .ogg@100 is inconclusive. That is my whole point.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-29 00:34:55
Quote
Ratz. Well in that case I would send them an ogg vorbis codec to -test-, and send them the latest optimized alpha *evil laughter* when the test is about to start.

I don't use alphas on my tests. That's why MPC is version 1.14
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-29 00:36:36
Anyway, the tester shouldn't accept that alpha without testing average bitrates at given quality levels.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-29 00:42:57
Quote
but I AM saying that for example I have noticed (havent experimented much I admit) MPC to create larger files than ogg at any quality setting. THIS fact does -not- make MPC a better codec. It makes MPC a codec that creates larger files.

Goldberg Variations, played by Glenn Gould (55 minutes, stereo digital record) :

ogg -b 4 : 114 kbps
mpc -q4 : 99 kbps

I can find you more than 100 CD in my library where mpc --radio will be more economic than vorbis -b 4(,25)...

Others :
http://membres.lycos.fr/guruboolez/AUDIO/a...assical_VBR.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/classical_VBR.txt)
http://membres.lycos.fr/guruboolez/AUDIO/a...etailed_VBR.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/classical_detailed_VBR.txt)
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-29 00:44:33
Quote
Quote
Ratz. Well in that case I would send them an ogg vorbis codec to -test-, and send them the latest optimized alpha *evil laughter* when the test is about to start.

I don't use alphas on my tests. That's why MPC is version 1.14 

Fine, fine, I will test my codec once and if it doesn't create a Blue Screen of Death I will call it stable, OK? *Dr. Evil Laugh*

Seriously though, rjamorim (look, I spelled it right!) - don't you agree with my point:

Quote
Filesize doesn't concern me, the quality/bitrate quotient, what I call codec efficiency concerns me. Puntcodec@1400kbit winning everything is not interesting. AAC@100kbit creating a slightly better sounding sample than .ogg@105kbit would be, and mpc@110kbit sounding better than .ogg@100 is inconclusive. That is my whole point.


Or am I being silly? Guruboolez tried to argue with me but English isnt his language, it isn't mine either (Dutch anyone?).. I just don't see a reason not to balance quality settings based on a per file (audio-type) basis instead of based on what 'some guy at some codec coding factory thought would be a good set of parameters'. Who knows, maybe MPC's algorythms are inferior for castanet sounds and it needs 700kbit to compensate this inferiority while all other codecs can do the same with 300kbit? You won't know without weighed results
Title: 128kbps Extension Test - OPEN
Post by: ExUser on 2003-07-29 00:47:47
If I'm summing up the problems properly, they're these:

Some codecs are VBR. This means that they allocate bits based on the difficulty of the sample. In other words, quality remains constant, and file size varies.

Some codecs are ABR. This means that they make the best out of a specified file-size. In other words, file size remains constant, and quality varies.

Roberto's test was designed so that in standard, global usage file size should be approximately equal between the VBR and ABR codecs. However, for this test there are samples that are difficult to psychoacoustically encode. The ABR codecs will thus be hampered by their limited file size, whilst the VBR codecs will excel due to the relaxation of this constraint.

In reverse, there may be samples (I haven't downloaded and examined all packages) where the ABR samples will be of a higher quality than the VBR, due to the fact that the sample is less difficult to encode.

The fact that two different encoding methodologies are being combined into one test will inevitably obfuscate proper analysis, as there was an uncontrolled variable.

It would seem to me that either all codecs should be VBR, or all should be ABR, to ensure uniformity throughout the samples.

I suppose we'll see how combining the two modes affects the results, however.
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-29 00:48:35
MPC is good if it can compensate for its inferior algorithm when it needs to (even switching to lossless)
while everything else is worse, because the difference is very audible.

/EDIT\ Only CBR codec is QuickTime, which is still very good, comparable to mpc.
(for me at least - tried ff123's analyzer)
LAME is using ABR, because its low quality VBR perfoms poorly compared to it. \EDIT/
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-29 00:54:58
Quote
Quote
but I AM saying that for example I have noticed (havent experimented much I admit) MPC to create larger files than ogg at any quality setting. THIS fact does -not- make MPC a better codec. It makes MPC a codec that creates larger files.

Goldberg Variations, played by Glenn Gould (55 minutes, stereo digital record) :

ogg -b 4 : 114 kbps
mpc -q4 : 99 kbps

I can find you more than 100 CD in my library where mpc --radio will be more economic than vorbis -b 4(,25)...

Others :
http://membres.lycos.fr/guruboolez/AUDIO/a...assical_VBR.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/classical_VBR.txt)
http://membres.lycos.fr/guruboolez/AUDIO/a...etailed_VBR.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/classical_detailed_VBR.txt)

OK I stand corrected then. From those tables you showed me the reverse is true. (Doesn't matter to me, I have no preference for ogg or mpc)

So, for -your- kind of music, ogg consistently uses more bits it seems, with little exception. (I only compared q=3.5). Now suppose that the ogg and the mpc output all sounded EXACTLY the same (not perfect or transparent, just -the same quality-) to you. Wouldn't you then pick mpc as the 'best codec' instead of saying ogg is equal to mpc?

Quote
MPC is good if it can compensate for its inferior algorithm when it needs to (even switching to lossless)
while everything else is worse, because the difference is very audible.

/EDIT\ Only CBR codec is QuickTime, which is still very good, comparable to mpc.
(for me at least - tried ff123's analyzer)
LAME is using ABR, because its low quality VBR perfoms poorly compared to it. \EDIT/


PuntCodec™ switches to lossless ALL THE TIME. It needs to cause its only lossy algorythm is 'delete the wav header'. Still sounds the same! And gets 1400kbps. Does this make puntcodec better than mpc? No. Q.E.D.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-29 01:06:34
Quote
Seriously though, rjamorim (look, I spelled it right!) - don't you agree with my point:

Congratulations.

An no, I won't agree with anything at this moment. I want to see how this discussion develops first.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-29 01:17:37
Quote
So, for -your- kind of music, ogg consistently uses more bits it seems, with little exception. (I only compared q=3.5).


Not exactly. Classical music, with vorbis, is generally cool to encode. The quality scale isn't tuned for this musical genre, and -b 4 setting is close to 115-120 kbps (near -b 3 average bitrate). On the contrary, mpc need more bitrate for tonal instruments : violin, flute, cello... But at the same setting, mpc is more sympathic for piano, mono-recordings, and some others... Mpc has more amplitude than Vorbis (Wma9pro VBR is the champion, here).

Quote
Now suppose that the ogg and the mpc output all sounded EXACTLY the same (not perfect or transparent, just -the same quality-) to you. Wouldn't you then pick mpc as the 'best codec' instead of saying ogg is equal to mpc?


Sound can't be the same. Sorry for not supposing what you asked me to suppose, but Vorbis, AAC, WMA, MPC, MP3 simply can't be the same. For one single sample, artifacts are really different, not eaqually distributed, with different amplitude, etc...
But if you really want me to suppose this, yes, i would say mpc is the best (as far as I understand what you exactly mean).
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-29 01:33:30
Quote
Quote
So, for -your- kind of music, ogg consistently uses more bits it seems, with little exception. (I only compared q=3.5).


Not exactly. Classical music, with vorbis, is generally cool to encode. The quality scale isn't tuned for this musical genre, and -b 4 setting is close to 115 kbps (near -b 3 average bitrate). On the contrary, mpc need more bitrate for tonal instruments : violin, flute, cello... But at the same setting, mpc is more sympathic for piano, mono-recordings, and some others... Mpc has more amplitude than Vorbis (Wma9pro VBR is the champion, here).

OK, agreed, of course. Each codec has its own strenghts and weaknesses, and eacho codec has different methods of selecting how many bits it would 'like' to use on a specific type of sample.. piano, rock, silence.. etc
Quote
Quote
Now suppose that the ogg and the mpc output all sounded EXACTLY the same (not perfect or transparent, just -the same quality-) to you. Wouldn't you then pick mpc as the 'best codec' instead of saying ogg is equal to mpc?


Sound can't be the same. Sorry for not supposing what you asked me to suppose, but Vorbis, AAC, WMA, MPC, MP3 simply can't be the same. For one single sample, artifacts are really different, not eaqually distributed, with different amplitude, etc...


I agree. For example I understand that mp3 is such an inferior codec that it CAN'T encode certain types of sounds properly, no matter how many bits you throw at it.

But you forget: we are speaking statistics here. We are doing a test with (say) 100 people who all pull on sliders saying how good/bad a track sounds. Now it is possible that when all the slider positions are added up and divided by 100, that for 'harpsichord.wav' ogg vorbis and mpc get exactly the same score...

Quote
But if you really want me to suppose this, yes, i would say mpc is the best (as far as I understand what you exactly mean).


... and as you say. If for example for that track ogg and mpc get -exactly- the same statistical average score as said by the 100 testers, and the .ogg file is (say) 10% smaller, then ogg vorbis is the best codec, for this small track, for this type of music, according to the human taste in music... And again: yes Im sure the ogg will have sounded different than the mpc because of different coding methods, but if the panel of 100 people say that they liked the tracks equally well, then that means something. It is, after all, psychoacoustic encoding, the human factor is important.

And, it might be that ogg -overall- scores worse than mpc, sure but it is useful information to know that for this specific type of track, even though statistically they sounded identical, the ogg vorbis codec would be the best, because with that information you can perhaps find out what codec is most suitable for which types of music.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-29 01:47:13
Quote
... and as you say. If for example for that track ogg and mpc get -exactly- the same statistical average score as said by the 100 testers, and the .ogg file is (say) 10% smaller, then ogg vorbis is the best codec, for this small track, for this type of music, according to the human taste in music...
(...)
And, it might be that ogg -overall- scores worse than mpc, sure but it is useful information to know that for this specific type of track, even though statistically they sounded identical, the ogg vorbis codec would be the best, because with that information you can perhaps find out what codec is most suitable for which types of music.


One sample isn't necessary representative of the global genre. Especially when you take bitrate into consideration. There are a lot of signal characteristic to take into consideration : noise, stereo separation, volume... For MPC for exemple, single harpsichord notes will produce a very high bitrate ; higher density of attacks will reduce the bitrate. Other encoder can react on the opposite way.
Therefore, you can't extrapolate bachpsichord.wav future result to the entire harpsichord music. It's a good basis, but not an accurate one. I can send another bachpsichord sample, part of the same disc, and bitrate difference would be totally different, and quality maybe too...
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-29 02:25:30
Interesting point..

But in that case there can be a few conclusions:

1/ bachpsicord should be discarded completely since it is a 'freak occurrence' that only shows an abnormal bitrate for one (or a few) codecs and will therefore color the test.
2/ bachpsichord is just a good example where a particular codec (or a few codecs) have trouble.

In case 1, well, discard the sample cause it corrupts the 'overall image'?
In case 2, well it is interesting to see what the same codec would do with those other samples, compared to other codecs. In principle more samples would be needed for more accurate results. Also it wouldn't be fair to only include one 'freak' sample that confuses one codec.. at least one confusing sample per codec should be in there.

But I think in the end we are bound by some 'luck' choosing the samples. It might be that Roberto has chosen totally horrible samples from mpc point of view and totally great for ogg.. there's no way to prevent such faults, really. (unless you start listening to complete music pieces). The best test would be to have one sample of every song ever made  but that's kinda hard to do.

If you assume that Roberto has chosen fairly 'average, representative' samples that represent a lot of the different possible challenges lossy codecs face, then a codec that has more punt-weighted bad problems should receive negative points, the 'representative' choice of -all- samples should cancel out the freak occurrences with the exeption samples.
Title: 128kbps Extension Test - OPEN
Post by: phong on 2003-07-29 06:20:55
puntloos - were you here for the pre-test discussion?  The encoders were tested (the same versions as are being used in the test) on a large number of albums.  The settings chosen were the ones that came closest to 128kbps on average over all those albums.  Your hypothetical encoder would never have been able to achieve that because the same version used in the preliminary run would have been the one used in the actual testing.

Quote
I want to know which codec Makes the most efficient use of bits it gets at bitrate 128kbit. THAT is an adequate test result.

That's EXACTLY what's being done.  Settings were chosen that result in an average of 128kbps over the long run.  VBR codecs try to make the most efficient use of those bits by spending more (sometimes lots more) on hard parts and fewer on easy parts.  ABR and CBR codecs use those bits less efficiently by spending the same amount everywhere (very roughly speaking of course).  In the end, both produce music collections of same size (give or take), with the bits distributed differently.

If you forced VBR codecs to behave like ABR or CBR codecs what would that prove?  That they do poorly when they're not used as they were intended?  Why penalize them?  Should we encode longer files at lower bitrates so they end up the same size as shorter ones?  I don't think too many people are going to try out half a dozen quality settings on each file until they achive an exact bitrate.  They're going to choose a setting that gives them a desired quality level or a desired average bitrate over the whole of their collection.

In the case of the samples for this test, my understanding is that some were chosen BECAUSE they have shown to be difficult to encode.  Naturally, VBR codecs will spend more bits on some of them - that's the goal of a VBR codec.  If easy samples were chosen, they would spend less.  Of course, it would be harder to hear artifacts in easy samples, so they probably wouldn't give very useful results.

This reminds me of a benchmark I recently heard about comparing a Pentium 4s to a G5.  They disabled hyperthreading on the P4 to make the test "fair".  That's like saying if one CPU had an extra fast cache, it should be disabled to make the test "fair".  Nonsense of course.

Now, if we were trying to determine which would sound best for STREAMING at 128kbps, I could understand trying to manage the bitrates on a per-file basis, but that's clearly not what's being tested here.  If it were, I'm sure oggenc's managed bitrate mode (and other encoders' equivalents) would be chosen.
Title: 128kbps Extension Test - OPEN
Post by: Gabriel on 2003-07-29 09:03:29
It seems that a few people here missed the pre-test discussion:

http://www.hydrogenaudio.org/forums/index....showtopic=11383 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11383)

I you have not read it yet, you should do it now. Things will seems more clear after.
Title: 128kbps Extension Test - OPEN
Post by: bexx on 2003-07-29 09:35:59
"PuntCodec™ switches to lossless ALL THE TIME. It needs to cause its only lossy algorythm is 'delete the wav header'. Still sounds the same! And gets 1400kbps. Does this make puntcodec better than mpc? No. Q.E.D."


I'm really having a problem understanding any flaw in this test.

Each codec averages about 128bit.

Therefore each codec is given THE SAME AMOUNT OF BITS TO USE.

Its up to the codec on how to use them.

To test which codec does this the best we zoom in on areas of which are 'very' hard to encode.  If a codec is transparent on the dificult areas it will be transparent on the easy areas (okay a slight assumption however a very good one to make)

We zoom in on the hard parts and test how well each codec does.

MPC can avg 1411bit if it wants and OGG can use 12bits, it doesn't matter they are the SAME SIZE FILE IN THE END.... 128bit.  MPC isn't using more bits, its 128 just like the others.  If MPC was 'smart' enough to use bits where they are needed well then I think thats a good thing.

MPC IS NOT GIVEN MORE BITS THAN ANY OTHER CODEC. Its 128bit avg, just like the rest.  Prove that wrong and you might have a case.

I think the key thing to remember is this test is not about these particular samples, but music in general.

(sorry for adding this too many people have said it already but i dunno)
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-29 10:51:36
Quote
1/ bachpsicord should be discarded completely since it is a 'freak occurrence' that only shows an abnormal bitrate for one (or a few) codecs and will therefore color the test.

Harpsichord is known to be one of the most difficult instrument for many lossy encoders, sensitive to pre-echo (sharp attack) and subjet of heavy distorsions during the development of the note. I see two advantages on it :
- easy to ABX for many people (at least, easier than a piano or a violin)
- interesting to see how will react a psymodel, and bitrate consequences (for exemple, mpc and vorbis will detect the difficulties, and increase the bitrate ; a VBR setting for lame, as --preset-standard, underate difficulties, and will produce an abnormal distorted sound)
And harpsichord isn't a 'freak occurence'. It's just like saying that electric guitar is something rare, and should be removed : pure non-sense. Harpsichord was the most used instrument during at least two century, in the whole Europa. According to my listening tastes, harpsichord is more usual than any kind of cymbals. A listening test had to include a good variety of musical genre and various samples, in order to be 'fair' and representative.

Quote
2/ bachpsichord is just a good example where a particular codec (or a few codecs) have trouble.

Yes, exactly. But not for a single or even few codecs, but for most lossy & perceptual format. I know one exception : wma and wma9pro. I know wma to be bad on most other samples - critical or not. Don't know wma9pro general performance.
Therefore, it's interseting to test various format on a difficult instrument ; mixing VBR and CBR will make thing more intersting (and certainly not unfair), because I know that one pure CBR encoder, at 128 kbps, may sound as good if not better than others at 160-200 kbps.

Quote
But I think in the end we are bound by some 'luck' choosing the samples. It might be that Roberto has chosen totally horrible samples from mpc point of view and totally great for ogg.. there's no way to prevent such faults, really. (unless you start listening to complete music pieces). The best test would be to have one sample of every song ever made but that's kinda hard to do.

If you assume that Roberto has chosen fairly 'average, representative' samples that represent a lot of the different possible challenges lossy codecs face, then a codec that has more punt-weighted bad problems should receive negative points, the 'representative' choice of -all- samples should cancel out the freak occurrences with the exeption samples.

Did you ever made a listening test ? Did you ever took the responsabilty of choosing the adequate and representative samples, easy enough for being detected from original ? What made a sample 'exceptional', and which of the twelve is really 'freak' ? Did you participate to this test ?
Title: 128kbps Extension Test - OPEN
Post by: Gecko on 2003-07-29 20:05:56
A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

I'm not criticizing the test as a whole, but interpreting the results will be very difficult. Maybe the only usefull conclusion we can pull out of this is that vbr is better than managed bitrates which in turn is better than pure cbr.

Take this car anology: You have 2 types of cars with similar motors, one with many gears (vbr) and another with just a single gear (cbr). If you put them on an empty road (easy sample) and adjust the gear of the vbr car so the motor is going approximately as fast as the cbr car's, you won't notice much of a difference. If you put the two in city traffic (difficult sample), the cbr car will have a hard time because of all the varying speeds neccessary that its motor just can't deliver. On average the vbr car's motor is running at the same speed as the cbr car's, but depending on the situation other gears are used to compensate for weak motor output.

So what does this tell you about the quality of the motor? Maybe the vbr car has a worse motor than the cbr car, but uses it more efficient. You'll never know because with this test you don't gain access to this type of info. It will most likely only tell you that having gears in your car is a sensible thing. The anology has its flaws, but I hope you understand my point.

If you choose samples which are known to be difficult to encode then you are treating cbr codecs unfairly. The samples are chosen because they cause trouble and vbr codecs have a way of reacting to troublemakers which the cbr codecs lack.

Heh, I'm creative tonight, so here comes another anology. You are testing two boxers and start hitting them at various strengths. The cbr boxer allways uses the same amount of protection while the vbr boxer is skilled adapts his protection to the force of the blow. Under normal conditions both handle the hits similarly well. On stronger hits the cbr boxer sometimes is a little shaky. Since this isn't really telling you much, you decide to hit them both much stronger. The vbr boxer reacts and pulls out a heavy shield that can deal with even the most forcefull blows, but the cbr boxer simply collapses. Again, what does this tell you?
Title: 128kbps Extension Test - OPEN
Post by: [JAZ] on 2003-07-29 20:15:01
Ok, I think i'll add my grain of salt to why this isn't unfair (or fairer than the opposite).

First, bigger bitrate means better quality, if we stay on the same codec version (except if there's a bug). But this is not a linear progression so high_bitrage_quality/Factor_of_high_bitrate_and_low_bitrate not equal to low_bitrate_quality.
(addenum to this: if a codec has been rated transparent, how do you know that with a lower bitrate isn't still transparent?)

Second, said many times. We are trying to identify the quality that some codecs have around 128kbps.  Quality depends on the content being encoded, and the bitrate being used.  ABR/CBR codecs are constrained to a bitrate, and thus the quality is not constant.
VBR codecs try to maintain the same quality, and thus, deciding when to use more bits or not.

What can we extract from here?
The CBR/ABR codecs will have different qualities for a bitrate depending on the sample. This is why we use many samples, not just one, to get a global idea.
The VBR codecs, in theory, should maintain the same amount of quality, with different bitrate demands so we shouldn't expect big variations on the range (this would be indicating that they have failed on their purpose).[/b]
Thus, with VBR codecs we try to see how good they are, in maintaining the desired quality. There's no interest at all with the bitrate.

Going back to this test, VBR codecs just can't participate in it, because we are already telling them a quality. Why are they included? Because we've checked withing several files the average bitrate that it uses, and if it is getting the same quality for them, we can indicate which is the quality that averages that bitrate.

Since it is fair to test several samples in CBR/ABR to find the average quality that a bitrate produces, it is as fair to test several samples in VBR to find the quality that averages that bitrate. (addenum: you can't judge the bitrate of a VBR coded from a single file just like you can't judge the quality of an ABR/CBR coded from a single file)

I think this last sentence should be strong enough to divise any more discussion about it.
Title: 128kbps Extension Test - OPEN
Post by: [JAZ] on 2003-07-29 20:21:57
Quote
A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

WMAPro in VBR variated too much, and a setting couldn't be choosen. That's why the test uses the ABR mode (2-pass).
Title: 128kbps Extension Test - OPEN
Post by: ff123 on 2003-07-29 20:29:35
Quote
A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

WMA9Pro 2-pass VBR is a difficult case.  The normal WMA9Pro VBR had wild bitrate swings from album to album, and we had to rule out using that in this comparison because the average bitrate across different albums just wasn't stable enough.

So we figured 2-pass VBR would be the next best thing.  However, in the ideal world, the 2-pass would be calculated by encoding an entire album, or at the very least, an entire song, and then the sample would be sliced out afterwards.

Note that there is at least one critical difference between WMA9Pro 2-pass and limiting the bitrate on the other VBR codecs -- Microsoft actually provides a 2-pass mode, and the other codecs don't.  So 2-pass is a mode that listeners might actually choose to use, whereas limiting the bitrates on the other VBR codecs would not be how they would normally be used.

ff123
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-07-29 20:44:13
Just wanted to inform everyone here that the test pages have been moved to Paul's server for practicity reasons.

The new url is:
http://audio.ciara.us/test/ (http://audio.ciara.us/test/)

RareWares will stay here at HA, only the Listening Tests subsection will go.

Sorry for any inconvenience.

Regards;

Roberto.
Title: 128kbps Extension Test - OPEN
Post by: phong on 2003-07-29 21:02:05
Quote
So what does this tell you about the quality of the motor?

Not much, but we're shopping for a whole car, not just a motor.  Who cares how efficient your motor is if you loose all that efficiency in a crappy transmission.

Quote
The vbr boxer reacts and pulls out a heavy shield that can deal with even the most forcefull blows, but the cbr boxer simply collapses. Again, what does this tell you?

It tells us that our glassjaw CBR boxer should find a new line of work before he gets himself killed.

In this analogy, if defense represents adaptability of bitrate, what does "ring generalship" stand for?  Which codec should be accused of ear biting?
Title: 128kbps Extension Test - OPEN
Post by: fileman on 2003-07-29 21:42:22
@rjamorim

Thank you for your great work with that test!

Regards, fileman.
Title: 128kbps Extension Test - OPEN
Post by: puntloos on 2003-07-31 14:02:06
I will just say one more thing before I wave my white flag and/or get banned for being too stubborn, whichever comes first  Please, spend some time on my little rant, one last time.

Another 'hypothetical situation' for you:
Suppose we have a test between 2 encoders, both VBR and only 2 samples.Now we run the encoders with 'quality settings such that they will average at 128kbit, or to put it differently, such that in the end the sum of all filesizes add up to exactly the same: 327680 bytes. Both encoders have used the same amount of bytes to 'play with'.

Now. The results: *drumroll*
Encoder A:192kbit average.
  - Hardrock, encoded at 64kbit average[/li][/list]Encoder B:128kbit average.
  - Hardrock, encoded at 128kbit average.[/li][/list]

Now we run the listening tests, and suppose it turns out that overall, the Encoder A and Encoder B versions of both Bachpsichord and hardrock tracks get the same score. (they don't sound the same, they just sound equally 'pleasing' when you consider overall score). Or if you wish, more skewed: Say that Encoder A gets a slightly higher score than B on bachpsichord, but a slightly lower score than B on hardrock and the average score is the same.

And now my point is:

The result: "well, that's that, Encoder A and Encoder B are equal for 128kbit." is what would be the conclusion of this test like everybody supports here. And I agree! It is Fair.

But.

If you weigh the averages, you can say MORE. There is more information in the results! You can then say:
- Encoder A probably has an inferior coding method for bachpsichord, and needs more bits to reach its target quality.
- Encoder A probably has a superior coding method for hardrock and needed fewer bits to reach its target quality

And this is useful information, wouldn't you say? Isnt this what this test is all about? Finding out as much as we can from listening tests we do, trying to compare the various files? Wouldn't it be nice to be able to say "Well, for Classical music, mpc scored highest, while for hardrock ogg vorbis and AAC are the preferred choices? (just making something up here).

Plus, of course my example is highly stylized. With a lot more encoders, a lot more samples, filesizes and so on, it is not inconceivable that interesting extra results can come to the light. Again, I'm not saying this weighing scheme is rock solid, it, like the test itself, gives some extra data to think about. I really don't see why you would want to ignore this.

As a last comparison, if you are choosing which car to buy, and a test gives Car A and Car B *exactly* the same score and same price, then you don't just pick one blindfolded. No. You go and check what exact features both cars have that you consider important[/i]. One car will go faster while the other is safer. A USEFUL car test won't just say "Car A and B are equal", no, a useful test will say "Car A and B are equal, Car A is faster and more dangerous, car B is slower and safer". Pick what you prefer.
Title: 128kbps Extension Test - OPEN
Post by: bawjaws on 2003-07-31 14:20:13
Quote
This reminds me of a benchmark I recently heard about comparing a Pentium 4s to a G5.  They disabled hyperthreading on the P4 to make the test "fair".  That's like saying if one CPU had an extra fast cache, it should be disabled to make the test "fair".  Nonsense of course.

This is sort of off-topic (except that it shows how difficult benchmarking and testing can be) but:

They disabled hyperthreading on the P4 because the P4 got *worse* scores when it was turned on. What you think was done to make things 'fairer' for the G5 was actually done to benefit the P4.

Weird, huh?
Title: 128kbps Extension Test - OPEN
Post by: AstralStorm on 2003-07-31 14:42:34
Quote
- Encoder A probably has an inferior coding method for bachpsichord, and needs more bits to reach its target quality.
- Encoder A probably has a superior coding method for hardrock and needed fewer bits to reach its target quality

No, it might mean that encoder A has a superior psychoacoustic model, which detected, that first sample will be hard to encode and used less bits for easier second track. It is right if the same codec fails at bachpsichord using less bitrate.

Do you think that 64kbps MP3 is superior to 128kbps AAC, because the latter uses more bits?

Anyway, we're testing QUALITY, not QUALITY/SPACE tradeoff. Go figure.
In the second case, 48kbps MP3 might be better than 64kbps one,
because quality loss is substancially smaller than filesize loss.
Title: 128kbps Extension Test - OPEN
Post by: guruboolez on 2003-07-31 15:32:26
Quote
Wouldn't it be nice to be able to say "Well, for Classical music, mpc scored highest, while for hardrock ogg vorbis and AAC are the preferred choices? (just making something up here).

Yes, conclusions are fine to me.
But you forget one thing. Roberto's test had a precise purpose : building an approximative hierarchy between many audio format, at ~130 kbps, and for general purpose. You can't conclude anything reliable and precise on musical genre, with only 12 samples. If you wan't to know which codec is the best for harpsichord, encode at least three different harpsichord sample (harpsichord didn't have a big dynamic range, but with recording conditions, manufacturing, and music content, results may change). For piano, you should take at least 6 samples : low/mid/high volume, noise or not, precise recording or more distant one... For orchestra, damn, it will be more difficult : strings, wind, full-orchestra, solo instruments, recording conditions, etc... are many variations of a same reality. And you to continue with string quartets, solo recorder, trumpets, wind ensemble, voice of course, organ... And I'm only talking here about "classical" genre.

In Roberto's test, there are two classical samples : macabre (orchestra) and bachpsichord. It doesn't make sense to conclude that mp4 is the must accurate encoder for piano and voice, just because mp4 was the best for macabre.wav. There's more sense to conclude than sample X, winner of the bachpsichord test, should be a privileged choice for solo harpsichord discs (nice : it's <0,01% of the whole discs of this planet), simply because another harpsichord recording should differ too much. For orchestra, I wouldn't conclude anything on macabre.wav results. Of course, blade failed horribly, and had 99,99% chance to be a bad choice on other orchestra parts. But the winner on these 20 seconds, how many chances to keep the first place on a long clarinet solo ? On a tutti ? A string introduction ?

Roberto's test conclusion will give us less answers than lead. If an encoder will fail on electronic sample (#5), and be the best on metal (#10), we can't conclude that this encoder is good for metal and useless for electro. We just have to investigate further to be sure, by testing other similar but different samples. It's just a preliminar hypothesis, that need to be proved by user experience, or by isolated tests.


(Note : for harpsichord, my choice is made. I know for a good time what encoder is the best for my ears. I'm just waiting the end of the test before any comments).
Title: 128kbps Extension Test - OPEN
Post by: bond on 2003-08-03 15:52:34
rjamorim,
i am still listening to some samples, so dont hurry to close the test


btw. does anybody know from where the macabre clip was taken (i mean from which composer, etc...)
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-08-03 16:05:17
Hehe. Well, the test will be closed tonight at midnight brazilian time (4AM of Monday GMT)

So, please send in your results until then
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-08-04 02:00:52
[span style='font-size:14pt;line-height:100%']TEST CLOSED![/span]

No more mails accepted from this moment on.

Expect results in a few hours.
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-09-10 04:18:46
Quote
Here is the sample batch - you need to make an executable from it!

And how do I do that?

I tried a tool called bat2exe, it didn't work (batch execution stops at the middle)
http://nlsn.free.fr/batch-down/Bat2Exe.ZIP (http://nlsn.free.fr/batch-down/Bat2Exe.ZIP)
That tool actually creates .com files.

Can somebody help me out with a good bat to exe tool?
Title: 128kbps Extension Test - OPEN
Post by: rjamorim on 2003-09-10 06:25:49
Just tried this one:
http://www.bdargo.com/info/batcomp.htm (http://www.bdargo.com/info/batcomp.htm)

Compiles the batch files well and these .exe from .bat work perfectly here at home. But they generated GPFs on ff123's PC, and output some very weird errors (could not find C:\bdtmp)