HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Sebastian Mares on 2006-08-11 13:11:26

Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-11 13:11:26
I am back in Germany after some very stressing days in Bucharest, Romania because of the sudden death of my grandmother and the problems related with it (very early burial, very short time to obtain flight tickets, etc.).

Anyways, I see that the crowd wants a new listening test and since the 48 kbps multiformat test was delayed until the new WMA codec is open for testing, we have two reasonable possibilities: test MP3 at 128 kbps and include fast encoders like Helix, or test various formats at 80 kbps. Personally, I don't have any preference so I don't really care which one comes first. What do you guys think? Also, I am open for suggestions about which codecs to include in any of the tests.

Please don't say anything about the 48 kbps test in this thread because I won't change my mind.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-11 13:26:15
i always wanted to see how this free fhg encoder (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_Win32_FCR_20050915.zip&Name=Win32%20mp3%20command%20line%20encoder) compares to others at 128kbps (it has only CBR).

thanks, J.M.
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-08-11 14:06:52
I vote for an mp3 test @ ~128 kbps
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-11 14:24:10
Could we make this topic a poll?  I vote for multiformat.

Although.. maybe I should convince my bias for LAME that it may not be the best encoder?

I have a slight penchant for the multiformat @80, but anything is good.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-11 16:23:05
what about multi-bitrate? does it makes sence?

LAME at 64, 96, 128, 160, 190? To see if at what point is most transparent with the least bytes?
Title: Autumn 2006 Listening Test
Post by: Steve999 on 2006-08-11 17:02:21
I know higher bitrate is a bit of a pain becuase of test design reasons (getting significant results), but I'd like to see multi-format MP3 (including the ubiquitous-in-the-real-world itunes) at the 160 kbps bitrate.
Title: Autumn 2006 Listening Test
Post by: pepoluan on 2006-08-11 17:38:20
You know, after thinking awhile, I agree with multi-MP3-encoder @ 128 kbps.

I mean, we take for granted that LAME is the best.

Reminds me of Musepack...

Title: Autumn 2006 Listening Test
Post by: markanini on 2006-08-11 18:32:19
Another vote for MP3 @ 128. Still the most popular format and bitrate.
Title: Autumn 2006 Listening Test
Post by: stephanV on 2006-08-11 18:49:05
Another vote for 128 kbps MP3.

Quote
test MP3 at 128 kbps and include fast encoders like Helix

Perhaps some benchmarks would be nice too then.
Title: Autumn 2006 Listening Test
Post by: sony666 on 2006-08-11 18:49:31
i always wanted to see how this free fhg encoder (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_Win32_FCR_20050915.zip&Name=Win32%20mp3%20command%20line%20encoder) compares to others at 128kbps (it has only CBR).


This is interesting.. a working, legal, recent FhG comandline encoder. Thanks

edit: omg 64bit, OSX and Linux too

Wow, I completely missed that Thomson has gone into offensive like that. Too bad they don't have a more intelligent lowpass after all these years.
Title: Autumn 2006 Listening Test
Post by: pepoluan on 2006-08-11 19:06:34
Sooooo what encoders will we be testing against each other?

- LAME
- FhG
- Helix

And I second stephanV there... not only listening tests, but while we're at it, some good ol' speed benchmark. Or am I misunderstanding you, stephanV?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-11 19:33:24
Speed benchmark, hmm... We should use full tracks for this I guess.
Title: Autumn 2006 Listening Test
Post by: quas on 2006-08-11 19:36:17
My vote is for MP3 too.

How about:
- LAME
- FhG
- Helix
- iTunes
+ speed benchmarks.
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-11 19:43:31
Speed benchmark, hmm... We should use full tracks for this I guess.

Speed benchmarks should be a different process, IMO.
I think everyone is aware that HELIX is the fastest encoder we have.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-11 19:47:24
How about:
- LAME
- FhG
- Helix
- iTunes
+ speed benchmarks.

well, then maybe gogo (and maybe also xing) could be also included if we are to test something as crappy as itunes mp3
Speed benchmarks should be a different process, IMO.
I think everyone is aware that HELIX is the fastest encoder we have.

the fhg encoder is evil fast too

J.M.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-11 19:48:11
Do you think Gogo should be tested? Also, what about anchors?
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-11 19:51:23
well, then maybe gogo (and maybe also xing) could be also included if we are to test something as crappy as itunes mp3
iTunes updated their mp3 encoder lately... If you want to include something interesting, I would pick Blade.  And Shine, maybe.  But I'd really like Blade to be in there.
Lame and Xing (new - realplayer) are mandatory.  So is FHG (WMP 10). So is iTunes.
Title: Autumn 2006 Listening Test
Post by: eofor on 2006-08-11 19:54:16
Do you think Gogo should be tested? Also, what about anchors?


To keep it strictly MP3, my suggestion:

Low anchor: ye olde Blade @ 128
High anchor: LAME VBR @ ~192
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-11 19:56:13
Quote
iTunes updated their mp3 encoder lately... If you want to include something interesting, I would pick Blade.  And Shine, maybe.  But I'd really like Blade to be in there.
Lame and Xing (new - realplayer) are mandatory.  So is FHG (WMP 10). So is iTunes.


Blade? Come on...
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-11 19:57:25
well, then maybe gogo (and maybe also xing) could be also included if we are to test something as crappy as itunes mp3
iTunes updated their mp3 encoder lately... If you want to include something interesting, I would pick Blade.  And Shine, maybe.  But I'd really like Blade to be in there.
Lame and Xing (new - realplayer) are mandatory.  So is FHG (WMP 10). So is iTunes.

again I forgot that the Helix is Xing successor  (by Xing I meant the "new" xing without short blocks)

I vote for Blade too (low anchor) 

J.M.

(edit- how many times I forget to fix the quote name  )
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-11 19:58:44
As low anchor it's OK, but not as contender. Decide between Blade and Shine. I would go with Shine.
Title: Autumn 2006 Listening Test
Post by: Pio2001 on 2006-08-11 20:29:46
Why not Windows Media Player mp3 encoder ? It should be a very common one.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-11 20:33:36
Why not Windows Media Player mp3 encoder ? It should be a very common one.

It is a FHG one.
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-11 20:35:17
As low anchor it's OK, but not as contender. Decide between Blade and Shine. I would go with Shine.

I want Blade!  I want to hear what I encoded to, "way back when"
Title: Autumn 2006 Listening Test
Post by: gameplaya15143 on 2006-08-11 20:39:38
Blade for low anchor would be good.  It got used a lot in the past.

I'd like to see xing 1.5 and lame 3.90.3/3.93.1 in the test.
I wouldn't complain if l3codecp.acm 1.2 was included too
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-11 20:59:30
There are a lot of FhG encoders: Adobe Audition, WMP, Nero(?), etc. Maybe we can decide which one to use - based on popularity? If popularity is important, we should stick to WMP.

Personally, I am against the idea of including multiple LAME or FhG encoders.
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-11 21:11:40
There are a lot of FhG encoders: Adobe Audition, WMP, Nero(?), etc. Maybe we can decide which one to use - based on popularity? If popularity is important, we should stick to WMP.

Personally, I am against the idea of including multiple LAME or FhG encoders.
I agree.  WMP is most popular, and should be selected, IMO.  Also, tests have already been made to compare LAME 3.90.1 to 3.97b, both by guruboolez and halb27, leaning on the general superiority of the latter.
Title: Autumn 2006 Listening Test
Post by: haregoo on 2006-08-12 04:11:58
I dare say that MP3 test at 96kbps by guruboolez (http://forum.hardware.fr/hardwarefr/VideoSon/MP3-WMA-AAC-OGG-qualite-kbps-evaluation-sujet-84950-1.htm#t921994) suggests the winner. 

I also have no preference at this time without new aoTuV and WMA...
I will go for any listening test as a tester.
Title: Autumn 2006 Listening Test
Post by: vinnie97 on 2006-08-12 08:02:37
And here I was getting excited at the prospect of a 80-kbps multi-format test. 
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-12 10:29:08
And here I was getting excited at the prospect of a 80-kbps multi-format test. 


That will follow shortly after (well, shortly - there has to be some time for the testers to recover).
Title: Autumn 2006 Listening Test
Post by: vinnie97 on 2006-08-12 12:09:50
I can be patient if I try hard enough.

Welcome back, btw...condolences to you in this stressful time with your loss.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-12 12:55:31
This kind of MP3 @ 128 test should be a useful piece of advice for practical usage. Besides the standard audio quality test a table of speed measurements would help a lot when the users consider the pros and cons of the faster encoders.


Some comments about the possible contenders:

LAME

I guess LAME is going to produce best audio quality in this test if -V5 --vbr-new is going to be used again. BTW, has anyone really tested the current LAME version at -V5--vbr-new vs. plain -V5? I think we have only assumed that --vbr-new is slightly better.

LAME is the only encoder that provides gapless playback with certain player programs. For files that are meant to be stored permanently on a computer this probably makes LAME the only sensible choice. Gapless playback is also possible with Rockbox and possibly with some LAN devices. Perhaps LAME deserves a "plus point" for that. However, with "non-gapless" players this does not matter. A faster encoding speed can be a more important factor if the quality is otherwise acceptable. Though, it is always possible to encode only the albums that need gapless playback with LAME and use a faster encoder for the other files.

In general, the interesting point would be how much better audio quality LAME has (if at all) when compared with the current versions of the faster encoders. Is the difference significant?

FhG

As said, perhaps the WMP's MP3 encoder should be used because of its assumed popularity. (Though, most WMP users are probably not aware of the encoder differences and did not actually choose the encoder they use.) Too bad that it does not have a VBR mode. Another popular version of FhG is included with Musicmatch Jukebox. It has a VBR mode.

Helix

As said, this is the current version of Xing. The quality is fine at higher VBR settings (I have personally tried to ABX some Helix VBR settings that average at about 170-200 kbps). Has anyone experimented with Helix settings that produce about 128 kbps VBR files?

Gogo

I have compared Gogo 3.13 "-a -b 192 -m j -q 2" (abr 192 kbps, joint stereo, high quality) against LAME 3.97b2 "-V2 --vbr-new". The very subjective result was that Gogo ABR is good enough for making MP3 discs quickly for my car stereo. LAME was quite transparent with some problem samples when Gogo was not, but in casual listening my Gogo encoded audio tracks have been fine. With these settings Gogo is about 3.8x faster on my PC. ABR is also a nice mode for making MP3 CDs since the resulting size is predictable.

I have not tested anything like "-a -b 133 -m j -q 2", but it might be worth of trying.

iTunes

iTunes VBR bitrate setting "quarantees" the minimum bitrate. I suppose its VBR 128 kbps setting would produce too big files. Hopefully the VBR 112 kbps setting would be close enough to the test target. iTunes has also some additional adjustments that are not properly documented as far as I know.

Roberto mentioned that iTunes should have been tested at CBR 128 instead of VBR in his test result page two and half years ago. Why? Has someone tested iTunes 128 kbps CBR vs. VBR? Has anything changed since then?


- Edited the itunes part a bit.
Title: Autumn 2006 Listening Test
Post by: Jillian on 2006-08-12 16:41:17
I'd like to see LAME@128 vs Multiformat@80 and any SBR+PS@32kbps.
Title: Autumn 2006 Listening Test
Post by: singaiya on 2006-08-12 16:49:17
I'm going to wait for 80kbps multi format. Because my prediction for the mp3@128kbps test is that most if not all the contenders will be transparent and none will be statistically greater or worse than any others.
Title: Autumn 2006 Listening Test
Post by: fpi on 2006-08-12 18:24:53
I'm going to wait for 80kbps multi format. Because my prediction for the mp3@128kbps test is that most if not all the contenders will be transparent and none will be statistically greater or worse than any others.


I agree. Since LAME MP3 in the last listening test was already near transparent I don't see the need to perform another MP3 test at this bitrate, if using LAME we are already near to transparency.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-12 19:14:57
If the test result would show that the much faster encoders are as transparent as LAME that would be good news for those who constantly convert on the fly. By much faster I mean FhG, Helix and Gogo. I have not tested iTunes speed.

However, I seriously doubt that the other encoders can be as good as LAME.

In any case the sample choice would be critical for making the differences show up. This time the samples should be known to be problematic especially for MP3.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-12 19:23:22
I think there should be a mix of killer samples and "normal" samples.
Title: Autumn 2006 Listening Test
Post by: Steve999 on 2006-08-12 23:22:44
It'd be nice if you stuck 160 kbps MP3 itunes CBR in there, if for no other reason than it's probably the most commonly used MP3 coding in the world.  It's the default MP3 setting for itunes.
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-12 23:22:54
Sounds like a good idea to test different mp3 encoders comprehensively.

How about including something like the old blade or the original l3enc (was that the oldest?), to see how much encoders have improved since the birth of the codec.

I also think it's a good idea to compare -vbr-new vs the regular lame, perhaps also the old athaa trick that reportedly still improves sound quality.

I'm all for testing iTunes too, crap or not, people deserve to know where it stands.

How many codecs/settings should the test include? With two lames, 4 other modern encoders, and an old one, the total is 7.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-12 23:25:19
Too much - consider anchors. Or do you see the old encoder as anchor? There shouldn't be more than 5 contenders I would say.

I guess these are in for sure:

LAME
Helix
iTunes
FhG (WMP, MMJB, something else?)

Now what about Gogo? Also, what should the high anchor be? Maybe LAME -V2? What about low anchor - Shine or Blade?
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-12 23:36:31
I say blade or l3enc could double as anchor, shine is not the oldest encoder, and the goal is to show how far the codec has progressed since it's inception, aside from using it as anchor. I would favour l3enc for this purpose as blade's source wasn't really intended to be used as a serious encoder whereas l3enc was. Then again with l3enc you may risk other encoders falling below the anchor - i don't know them well enough to judge the risk of that happening.

But, which version.. the initial 0.99a from the 16th march '94 or perhaps another? I don't know if any had bugs or quality issues or how they differ.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-13 00:54:07
Too much - consider anchors. Or do you see the old encoder as anchor? There shouldn't be more than 5 contenders I would say.

I guess these are in for sure:

LAME
Helix
iTunes
FhG (WMP, MMJB, something else?)

Now what about Gogo? Also, what should the high anchor be? Maybe LAME -V2? What about low anchor - Shine or Blade?

isn't the original the high anchor? i would add gogo.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-13 08:52:50
Of course not. The high anchor is something else. Question is: do we actually need a high anchor? Multiformat at 128 kbps didn't have one.

BTW: Shine, while not being the oldest, is the most simple MP3 encoder.
Title: Autumn 2006 Listening Test
Post by: gaekwad2 on 2006-08-13 10:14:05
A high anchor would most likely be transparent to most people most of the time.
I'm not sure how useful that would be.
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-13 11:09:34
BTW: Shine, while not being the oldest, is the most simple MP3 encoder.


I'm aware of that, but it won't tell you what 12.5 years of development have done for mp3. That's why i think it makes sense to have l3enc double as low anchor.

Shine is almost never used, is it?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-13 12:29:39
A high anchor would most likely be transparent to most people most of the time.
I'm not sure how useful that would be.


You guys should read what anchors are good for then.

And HbG, popularity is not a criterion for choosing a low anchor. A low anchor is supposed to sound really bad, it should be the worst. I am not sure if l3enc might not beat iTunes. On the other hand, if we choose l3enc as low anchor and iTunes or whatever other contender is worse than the anchor, it shows that the contender really sucks.
Title: Autumn 2006 Listening Test
Post by: boombaard on 2006-08-13 13:23:46

A high anchor would most likely be transparent to most people most of the time.
I'm not sure how useful that would be.


You guys should read what anchors are good for then.

And HbG, popularity is not a criterion for choosing a low anchor. A low anchor is supposed to sound really bad, it should be the worst. I am not sure if l3enc might not beat iTunes. On the other hand, if we choose l3enc as low anchor and iTunes or whatever other contender is worse than the anchor, it shows that the contender really sucks.


as i recall, blade@128-160kbps used to give me headaches with all the ringing happening.. i'm all for blade for a low anchor
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-08-13 13:39:03
To make sure the low anchor is clearly worse than the contenders, it would make sense to lower the bit rate. e.g. shine @ 96 kbps or something like that. A high anchor is useless since lame is pretty much transparent @ 128 kbps. It would be too hard to distinguish the anchor from the contenders.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-13 14:15:44
Quote
as i recall, blade@128-160kbps used to give me headaches with all the ringing happening.. i'm all for blade for a low anchor


Oh, I remember having the same feeling after listening to Vengaboys' "We're going to Ibiza" which I encoded for my little cousin five years ago when I didn't even hear about LAME.

Quote
A high anchor is useless since lame is pretty much transparent @ 128 kbps.


But we're going to use some killer samples, too.
Title: Autumn 2006 Listening Test
Post by: PatchWorKs on 2006-08-13 15:24:33
Sincerly i don't know if Kbps is still the right choice... i think that quality settings (something like ones tha Vorbis uses)  could be better.

Anyway, just my opinion
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-13 15:31:46
I don't get your point.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-13 17:11:46
As a quick rehearsal I tried this sample

http://www.hydrogenaudio.org/forums/index....showtopic=47370 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47370)

using the following encoders & settings:

LAME 3.97b2, -V5 --vbr-new
WMP10 / FhG, 128 kbps CBR default settings (I used the acmenc command line interface)
Helix (hmp3enc.exe 23.7.2005), -X2 -U2 -V60 -HF2 -F17000
iTunes 6.0.5.20, 128 kbps, VBR, Quality: Highest, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
iTunes 6.0.5.20, 128 kbps, CBR, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
Gogo 3.13, -a -b 133 -m j -q 2

The first few distorded rock guitar chords sent all other encoders except LAME straight to the Hell.

I ABXed them all 8/8, but LAME was difficult and the others were quick and easy.

In general, LAME was quite good, not annoying at all, FhG was barely usable, the others were bad.

I don't know if the encoder versions and settings I used were optimal, but if the actual test samples behave even partially like this I think we may have a good chance to find a clear winner this time.

Personally, after this rehearsal, I would use only LAME at this bitrate. Even the other samples would be transparent with all encoders a standard rock quitar sound is too important for me.

Edit: fixed the link
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-13 17:24:05
As a quick rehearsal I tried this sample

http://www.hydrogenaudio.org/forums/index....showtopic=47313 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47313)

using the following encoders & settings:

LAME 3.97b2, -V5 --vbr-new
WMP10 / FhG, 128 kbps CBR default settings (I used the acmenc command line interface)
Helix (hmp3enc.exe 23.7.2005), -X2 -U2 -V60 -HF2 -F17000
iTunes 6.0.5.20, 128 kbps, VBR, Quality: Highest, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
iTunes 6.0.5.20, 128 kbps, CBR, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
Gogo 3.13, -a -b 133 -m j -q 2

i always had mismatch in WMP version vs. encoder used version. Can you please enlighten me on which encoder is the WMP10 / FhG ? Is it l3enc based (v1.2.x, slow) or fastenc based (v3.x, very fast)?

thanks, J.M.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-13 17:26:59
Fraunhofer IIS mpeg Layer-3 Codec (professional)

It is very fast.


Edit:

V. 3.3.2 (Build 44)


Edit 2:

http://www.hydrogenaudio.org/forums/index....showtopic=47313
The correct sample link is: http://www.hydrogenaudio.org/forums/index....showtopic=47370 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47370)
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-13 17:37:06
Fraunhofer IIS mpeg Layer-3 Codec (professional)

It is very fast.


Edit:

V. 3.3.2 (Build 44)

thanks (this is my favourite encoder at 128kbps cbr  )
J.M.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-08-14 07:54:34
Because of the problem samples which are included in the test I suggest for the high anchor to use very high bitrate ABR or CBR. Guess most members like to see Lame 3.97b2 as the high anchor encoder, and as for this I suggest to use ABR 256 or similar.

This should still show up typical mp3 problems while not depending on specific weaknesses. The high anchor should really be a high anchor.
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-08-14 09:41:22
It makes no sense using an ABR setting as a high anchor... I don't get it.  It's not recommended anywhere else. I know you have a weak spot for it but I think the rest are quite happy with, let's say -V2.
Title: Autumn 2006 Listening Test
Post by: stephanV on 2006-08-14 09:48:12
ABR is just using brute force, so why not using Lame CBR 320 kbps instead then? Anyway, if a high anchor is used, it should probably be reviewed first.

I would like to remention my interest in speed measurements. I wouldn't mind assisting in doing some benchmarks. (If something is called fast I want to know HOW fast  )
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 10:15:14
I am still wondering if we really need a high anchor since we didn't have one in the multiformat listening test at 128 kbps.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-08-14 11:21:38
ABR is just using brute force, so why not using Lame CBR 320 kbps instead then? ...

Yes, CBR 320 is a good thing for using as an mp3 high anchor IMO.
Title: Autumn 2006 Listening Test
Post by: SirGrey on 2006-08-14 11:24:27
Quote
I am still wondering if we really need a high anchor since we didn't have one in the multiformat listening test at 128 kbps.

High anchor is not for fun, but should be used for a purpose 
If all competitors quality is low, high anchor is needed to make marks to reflect the real situation more closely (w/o it marks could be too high).
IMHO, 128 Kbit for now is rather near to transparency, so high anchor isn't needed on this bitrate...
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-08-14 11:50:06
IIRC, Sebastian Mares wanted the high anchor because of the killer samples. It's quite useless for normal samples since LAME is pretty much transparent @ ~128 kbps (geee... it's the second time I say that). The question is then, will the listening test be split in half or what?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 12:02:59
Yep, that's my problem. Without the killer samples, LAME would already be like a high anchor (it was already transparent and only a little bit worse than the contenders during the multiformat listening test).
Title: Autumn 2006 Listening Test
Post by: SirGrey on 2006-08-14 13:49:21
Ok, point taken.
But, anyway:

1. Are you sure that those samples are really "killer" ? 
(It can be verified)

2. When the competitors are near transparent, may be original should be used to compare, not another codec ?
Title: Autumn 2006 Listening Test
Post by: fpi on 2006-08-14 14:11:09
So why not using, as an high anchor, an encoder that has different types of artifacts, like aoTuV beta 4.51 at a 160 kb/s bitrate?
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-14 14:17:39
The need of a high anchor can be decided later.

The contenders, fair encoding settings and samples are more important now.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-08-14 14:47:52
As for the killer samples I suggest to use (among others).
It's all natural music with the special effect that without knowing in advance that these are problem samples one would not easily beleive how big the problems are for encoders and settings that are usually transparent on most music (with the exception of castanets).
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 14:59:55
OK, I decided not to use a high anchor at all. As for the low anchor, I will have a look at l3enc.

Now to the contenders (which shouldn't be more than 5 IMO).

LAME is for sure LAME 3.97b2 -V5 --vbr-new.
FhG - WMP's CBR encoder or MMJB's VBR encoder?
iTunes - CBR or VBR?
Gogo and Helix - will look into it ASAP
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-08-14 15:17:57
Hmm... I'd really like to see lame 3.98. It seems like it has made som progress since 3.97b2. But I know, it's still alpha... but perhaps there's a beta coming up pretty soon?
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-14 15:19:46
...
FhG - WMP's CBR encoder or MMJB's VBR encoder?

I'm sure FHG performs better in cbr (I have yet to see a FHG encoder that doesnt force plain Stereo in VBR). so I would choose the WMP's CBR ACM encoder (also because I think it is very good quality and would like to see how it compares to Lame).

J.M.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 15:29:24
I just talked / am talking to Roberto who claims that iTunes also performs better in CBR mode. It also seems sensible that Apple focused on AAC and didn't tune the VBR mode of their MP3 encoder.

So, with l3enc, FhG and iTunes, we have 3 encoders in CBR mode. LAME, Helix and Gogo will most likely use VBR. I am now waiting for a CBR vs. VBR discussion to start and make this thread explode.
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-14 15:52:43
So, with l3enc, FhG and iTunes, we have 3 encoders in CBR mode. LAME, Helix and Gogo will most likely use VBR. I am now waiting for a CBR vs. VBR discussion to start and make this thread explode.
You know what I've been thinking?  Maybe we should test LAME -b 128 with a high-anchor;  I'd be interested in seeing how LAME performs, "on equal terrain" with the other CBR encoders.  And we could get guaranteed 128kbps bitrate, as opposed to the approximate 135 or so we'll get with vbr...
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 16:23:58
I thought about that, but then imagine how the LAME zealots will react. Gabriel, what do you think?
Title: Autumn 2006 Listening Test
Post by: vlada on 2006-08-14 16:41:54
I think you should always use VBR if it is possible and produces higher quality files then CBR. Why limit one encoder just because others are worse (can't produce VBR). If you decide to use CBR for some encoders, you should encode first all VBR tracks and then set the CBR bitrate as average of the achieved bitrates. I'm affraid all VBR encoders will go above 128 (I'd guess the average will be around 135 kbps). You might also contact developers to suggest you best settings as Doom9 does before his video codecs comparisons.

As for speed testing - everyone can try it by himself. It is an objective test while listening test is subjective.
Title: Autumn 2006 Listening Test
Post by: stephanV on 2006-08-14 17:49:22
Quote
As for speed testing - everyone can try it by himself. It is an objective test while listening test is subjective.

You can't do subjective testing for yourself?

I just thought it would be interesting to gather results from different machines, that's all. Maybe not.
Title: Autumn 2006 Listening Test
Post by: kennedyb4 on 2006-08-14 18:17:12
As low anchor it's OK, but not as contender. Decide between Blade and Shine. I would go with Shine.


Hi. I'm still running into new material encoded at Blade 128 so anything that might help to stop its use would be useful. It should make a good low anchor. Shine would too but this is used rarely it seems.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 19:52:54
l3enc would most probably be the low anchor - Blade won't be featured then.
Title: Autumn 2006 Listening Test
Post by: rjamorim on 2006-08-14 20:49:17
Hi. I'm still running into new material encoded at Blade 128 so anything that might help to stop its use would be useful. It should make a good low anchor. Shine would too but this is used rarely it seems.


Blade was already put to shame 3 years ago (http://www.rjamorim.com/test/128extension/results.html). I think that's all the argument you need to provide to people willing to listen, as I don't think its quality improved ever since 


I agree with Sebastian, it would be awfully cool to test l3enc 1.0, to see how quality improved since the very first encoder. I always wanted to use it as anchor, but on every test people somehow convinced me to use something else.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-14 21:35:00
Roberto, could you ask your Real connections what encoder settings they recommend?

Also, Gabriel, what do you suggest for LAME?

BTW, the encoders featured are going to be: l3enc, LAME, Gogo, iTunes, Helix and WMP FhG. Time to discuss settings now.
Title: Autumn 2006 Listening Test
Post by: /mnt on 2006-08-14 22:13:41
I tried the oldest version of l3enc a while ago to see the improvements of the MP3 formats over the last 10 years.

L3ENC 0.99a still works but it leaves a pop sound on the begaining of the MP3, I have not tried the other versions through.
Title: Autumn 2006 Listening Test
Post by: rjamorim on 2006-08-14 23:08:38
L3ENC 0.99a still works but it leaves a pop sound on the begaining of the MP3, I have not tried the other versions through.


I think it leaves some ancillary crap at the beginning of the file. If the same thing happens on 1.0, it would be wise to chop the first few samples
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-14 23:10:09
I tried the oldest version of l3enc a while ago to see the improvements of the MP3 formats over the last 10 years.

L3ENC 0.99a still works but it leaves a pop sound on the begaining of the MP3, I have not tried the other versions through.

l3enc v. 1.00 has the same problem. This click makes mp3 files that are encoded from wave source files unsuitable for testing purposes. It makes the measured track volume levels incorrect even if the beginning of the file is excluded in the used ABC/HR configuration. However, raw pcm works fine.

I added l3enc v.1.0 samples (from wave & pcm) here: http://www.hydrogenaudio.org/forums/index....st&p=420360 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47370&view=findpost&p=420360)

BTW, l3enc v. 1.0 uses joint stereo, bit reservoir and switches between long and short blocks. It does not sound worse than the other bad encoders with this sample. I guess it is better than Shine.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 11:33:54
Maybe schnofler can change ABC/HR in such a way, that it calculates the volume taking the offset into consideration.
Title: Autumn 2006 Listening Test
Post by: stephanV on 2006-08-15 12:53:38
l3enc v. 1.00 has the same problem. This click makes mp3 files that are encoded from wave source files unsuitable for testing purposes. It makes the measured track volume levels incorrect even if the beginning of the file is excluded in the used ABC/HR configuration. However, raw pcm works fine.

Couldn't it be that the encoder always expects raw pcm and that the click is then the as audio encoded wav header?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 13:09:14
Did you guys use -wav for encoding?

Quote
1.1 PCM audio input file
The first command line argument specifies the name for the PCM audio
data file. Version 1.00 of the encoder accepts either raw PCM audio
data files or PCM audio data files in RIFF/WAVE format as used by
Microsoft Windows. The samples must be 16 bit signed integer values.
The sampling rate must be 44.1 kHz.
 
A) raw PCM audio data
    By default the input file is assumed to contain raw PCM audio data.
    Stereo audio data is input in interleaved format, the first channel
    beeing the left channel.
      <sample #1 channel #1> <s. #1 ch. #2> <s.#2 ch.#1> <s.#2 ch.#2> ...
    Mono audio data has the format
      <sample #1> <sample #2> <sample #3> ....
    Whether the input file is treated as mono or stereo audio data is set
    by the encoding mode parameter (1.3). Default is stereo.

B) RIFF/WAVE format
    If the '-wav' option is specified, the input file is assumed to contain
    16 bit PCM audio data in RIFF/WAVE format as used by Microsoft Windows.
    Audio parameters are extracted from the Wave header and checked against
    the settings of the encoder. If not supported options are found
    (e.g. 8 bits/sample), the encoding process is aborted. The encoding
    mode (mono or stereo) is determined by the settings in the WAVE header.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-15 13:38:03
Actually, the -wav switch didn't work. I had to remove it for making the encoder work with a standard wave file.

However, making pcm copies of the wav samples should not be a problem. I did it with WaveLab.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 14:21:16
Hmm, true - it says it doesn't understand the format.

OK, I use Audacity to remove the header of my Jethro Tull sample and encoded using l3enc. In this way, I don't have to remove audio samples from the test samples.
Title: Autumn 2006 Listening Test
Post by: pest on 2006-08-15 17:07:15
what about "--abr 128" with Lame?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 17:34:30
-V5 --vbr-new is just fine.

Edit: (If Gabriel agrees with that.)
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-15 19:46:28
We already proved that setting was transparent. I'd like to test -b 128...
Title: Autumn 2006 Listening Test
Post by: greynol on 2006-08-15 21:17:03
I hate to argue but -V 5 --vbr-new is not transparent to everyone.

Still, I think it is appropriate to conduct a 128kbit test using samples at 128kbit.  I don't think it's appropriate to allow one contender to use more than 128kbits to encode a sample and not another because it's CBR.

If it's a test based on quality settings, call it as such.  Reserve 128 kbps for contenders @128 kbps.

...then I suppose there's a case for and against including ABR.
Title: Autumn 2006 Listening Test
Post by: benski on 2006-08-15 21:29:50
I think it'd be interesting to see LAME at 128 CBR.  I think it's well proven that the VBR algorithm is excellent, but how does it stand up without it?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 22:32:17
Well, this would mean to tie LAME's hands. Also, one argument for running this test will lose its strength: most people who use -V5 --vbr-new wanted to see how faster codecs perform at similar settings. l3enc being CBR only is not a big problem, since it's also supposed to be the low anchor. As for iTunes - developers probably focused on tuning CBR and then moved to AAC because CBR is most popular for their target group.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-15 22:55:24
My interest for a MP3-listening-test-only is not very high, but my interest for a MP3 test using CBR or ABR is really low. I'd personally focus on VBR performance of various MP3 encoders. List of possible contenders allowing VBR:
- LAME
- Helix
- Fraunhofer Fastenc
- iTunes
- gogo

It's not really surprising but these encoders are also the most interesting MP3 encoders available. I don't really see the point of testing them at an unefficient setting. If we force LAME to use ABR/CBR then we should logically force all other competitors to use the same kind of setting (otherwise the test will look as a very dubious one).

If I had to perform myself such test (as a personal exercise), I would:
1/ select all encoders mentioned above
2/ build a bitrate table to see if they all offer a setting close to 128 kbps and of course which one may work [three encoders are finely tunable then it shouldn't be a problem; LAME has -V5 which correspong to ~130 kbps; iTunes is maybe harder to include for this reason]
3/ think about the samples
4/ think about anchors


I would try to avoid VBR/CBR confrontation (and as I said, I don't consider a CBR-only test as a useful solution). I'd tend to forget the idea of making a "popular encoders" listening test for that reason (assuming that one at least would be CBR only, like WMP encoder IIRC).

It's just my personal point of view on this subject.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 23:17:48
iTunes is said to perform better at CBR than VBR. Forcing iTunes to use VBR was one of Roberto's mistakes in his test.
Title: Autumn 2006 Listening Test
Post by: greynol on 2006-08-15 23:20:47
If we force LAME to use ABR/CBR then we should logically force all other competitors to use the same kind of setting (otherwise the test will look as a very dubious one).
...
I would try to avoid VBR/CBR confrontation (and as I said, I don't consider a CBR-only test as a useful solution).

Exactly!

I have no problem with a VBR 128-ish test, but I do have a problem with being concerned about tying Lame's hands but not being concerned with tying the hands of the other contenders.  This test does not compare codecs with "similar settings" rather it's quite the contrary.

But if you're going to run a 128-ish VBR test, don't call it 128 kbps.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-15 23:32:04
iTunes is said to perform better at CBR than VBR. Forcing iTunes to use VBR was one of Roberto's mistakes in his test.

And people used to repeat LAME's poor performance with VBR at ~130 kbps until well-grounded listening tests proved the opposite
Problem with the mentionned listening test isn't VBR by itself, but the choice of setting (112 kbps + VBR IIRC) which lead Roberto to include iTunes encodings at a significantly lower bitrate (119 kbps (http://www.rjamorim.com/test/mp3-128/results.html) on average for 'difficult samples' when bitrate should be higher than 128 kbps using VBR on such material).
I may be wrong and the choice of CBR instead of VBR would make sense if VBR settings are not tuned enough. But are there proofs for this?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 23:32:54
What other hands are we tying? iTunes' VBR mode is worse than its CBR mode. As for FhG, MMJB could be an option if VBR is needed. Aren't both WMP's and MMJB's MP3 encoders based on FastEnc?
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-15 23:35:02
But if you're going to run a 128-ish VBR test, don't call it 128 kbps.

But you can call it as a ~130 kbps listening test as long as all tested settings are leading to ~130 kbps in the long term (e.g. by publishing complete bitrate tables including several hours of various kind of music).
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 23:38:27
I think we are nitpicking regarding the title. I used the term "@ 128 kbps" because of the tradition. I can also call it "@ ca. 43 * Pi kbps".
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-15 23:53:24
So, for the encoder decision and settings, do you think some pre-tests are appropriate? Obviously, not all encoders can be included to have a comparison between iTunes CBR and iTunes VBR, FhG CBR and FhG VBR, etc. What I know is that iTunes VBR performed very bad in Roberto's test. That's why I thought CBR should be used now. As for FhG, I always thought their CBR engine would also be better than their VBR one, but as you guys said, this has to be checked. However, this has to be checked in some pre-tests because there are already enough contenders.

I am hitting the sack now.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-15 23:54:55
iTunes' VBR mode is worse than its CBR mode.

Did I miss the listening test(s) proving that? Do you have a link?
The problem with iTunes (but correct me if I'm wrong) MP3 VBR is the following one:
1/ iTunes doesn't offer a scalable VBR engine (like HELIX or Fastenc does); you can only use "VBR" as option for fixed bitrate settings (like 96 kbps VBR, 112 kbps VBR, 128 kbps VBR etc... with nothing in the middle). It's a bit like current LAME VBR scale.
2/ But iTunes also uses the "target" bitrate as the minimal one per frame. In clear, if you select "192 kbps VBR", the bitrate will vary from... 192 kbps up to 320 kbps. "Low complexity" samples won't compensate for the bloating bitrate of "complex" samples. As a consequence the average bitrate of this setting could only be higher than 192 kbps.

In our case, "128 kbps + VBR" iTunes setting will for sure correspond to an average bitrate superior to 128 kbps. How greater? I can't say. But it will be superior.
An alternative choice would be "112 kbps + VBR" which would correspond to a bitrate higher than 112 kbps and --who knows-- maybe as close to 128 kbps than the previous setting. But as Roberto explained it this choice would handicap iTunes encoder (as it handicap it in 2004).
So assuming that iTunes 112@VBR is lower quality than 128@CBR, we still don't know how iTunes 128@VBR would perform against CBR@128. As I said, VBR@128 won't go below 128 kbps. Therefore it's hard to imaging this setting performing poorer than 128@CBR! So what we really need to know is the average bitrate of iTunes VBR at "128 kbps". If the average bitrate isn't too far from 128 kbps then iTunes VBR would easily be include in the test. If the bitrate is a bit too high (138...140 kbps) then we could easily increase a bit the setting for HELIX, FHG (and also GOGO). Then we would perform a listening test at ~135...140 kbps instead of a ~128...132 kbps one. Not a big issue in my opinion.


Quote
As for FhG, MMJB could be an option if VBR is needed. Aren't both WMP's and MMJB's MP3 encoders based on FastEnc?
Adobe Audition also uses Fhg encoder with a nice VBR scale.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 00:01:00
I find it hard to believe that 6 kbps difference between iTunes and LAME can make such a difference in quality. And if iTunes VBR implementation is so good, why didn't it allocate more bits for hard parts like LAME did?

Anyways, I will let iTunes encode my music collection (electronic, rock, instrumental, pop, could add some classical music too) and see what it comes up with.
Title: Autumn 2006 Listening Test
Post by: greynol on 2006-08-16 00:03:02
I think we are nitpicking regarding the title. I used the term "@ 128 kbps" because of the tradition. I can also call it "@ ca. 43 * Pi kbps".

Again, I happen to agree with guruboolez.  I don't believe we're picking nits.  I think that 128 kbps implies CBR and is therefore misleading.  130, OTOH, is clearly VBR.  IMO, tradition is usually never a good reason to do anything and I have taken issue with this idea of passing apples off as oranges with previous tests but I don't believe I've said anything about it until now.  I apologize for voicing my opion too strongly.

As far as genres and killer samples go, bitrates can vary.  I don't think you'd want to change the quality setting just to achieve a certain bitrate because of a certain genre.  This also reinforces the point of not putting CBR up against VBR.  Perhaps using samples of metal can demonstrate this, especially where the use of VBR will result in bitrate increases of greater than 10%?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 00:12:04
Man, didn't I say I wanted to hit the sack. Now it's 1 AM again and I am still here.

OK, I will change the title - no problem.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 00:15:38
So, for the encoder decision and settings, do you think some pre-tests are appropriate?

Pre-test would enlight us a lot if they're done properly (i.e. several samples, several people). A big pre-test performed by a single person  has a well-known limitation, and a small-one performed by several people but on a small selection of samples may also lead to false conclusion.
And what we need here is more than one pre-test. Suggestions for pre-tests:
- iTunes CBR vs iTunes VBR
- Helix VBR vs Helix CBR
- Helix default setting vs Helix 'tuned' setting
- Fhg VBR vs Fhg CBR
- Fhg from {one software} vs Fhg from {another software}
- LAME -V5 vs LAME -V5 --vbr-new
- LAME 3.97 vs LAME 3.98

If someone start a pre-test to "extract" the best setting for one encoder (let's say iTunes as example) then it would be unfair to refuse the same treatment for all other competitors (excepted LAME maybe which is the one we're more familiar with). iTunes, Helix, Fhg... our lack of knowledge is the same for all these encoders and I wouldn't really see as fair to privilege one over the other just before the test begins. On the other side, I can't imagine how great the enthousiasm would be to the idea of performing three pre-tests as a prelude to another one dedicated to MP3 encoders. 

Not to mention than even if pre-tests are done some people would still attack your choice, especially if a fight against CBR vs VBR will occur :/
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 00:20:54
Jesus, that's a lot of pre-tests. I am thinking of another possibility... Only thing that comes into my mind would be running two tests: one with all encoders using CBR and one with all encoders using VBR. However, this is rather complicated. We would have to use the same samples, but then the factor tester could have an impact on the results (different testers or even moods = different results).
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 00:42:22
I find it hard to believe that 6 kbps difference between iTunes and LAME can make such a difference in quality.

Don't forget that the bitrate table is based on short samples (selected ones, who all tend to use more bits than average). It implies that iTunes 112@VBR average bitrate would logically be lower and therefore that the difference with LAME ABR bitrate is therefore higher than 6 kbps.

Quote
And if iTunes VBR implementation is so good, why didn't it allocate more bits for hard parts like LAME did?
It's indeed a good question. Someone said once as hypothesis that the limited bitrate's variations of this encoder is linked to the (old) iPod CPU clock issue. I tend to believe that this limitation simply correspond to the will of keeping the bitrate as close to the mentioned one (128@VBR with iTunes is indeed close to 128 kbps whatever the situation and it's probably what most users are expecting from this setting).

N.B. what is interesting us at the moment is not to see if iTunes VBR is "so good" but rather if it isn't "poor" (in order to see if CBR would be more suitable than VBR). One advantage of iTunes trick (i.e. using the target bitrate as the minimal floor per frame) is at least to prevent the encoder of big encoding mistakes. I expect from iTunes CBR and VBR to be very close each others (quality-wise) simply because both modes are always leading to the same bitrate (approximately). With LAME on the contrary you can expect from VBR to sound much better on critical material (some samples could reach 200 kbps with -V5) but also to sound poorer in some cases (it was the case with quiet samples - problem partially fixed with --athaa-sensitivity 1 switch and now close to be totally fixed with 3.98 alpha 6).



Jesus, that's a lot of pre-tests. I am thinking of another possibility... Only thing that comes into my mind would be running two tests: one with all encoders using CBR and one with all encoders using VBR. However, this is rather complicated. We would have to use the same samples, but then the factor tester could have an impact on the results (different testers or even moods = different results).

I have something more easy: let people having choice of testing either CBR-only settings or VBR-only ones. A poll could be an answer.

If a majority of people are rather interested by 128 kbps CBR => test everything with CBR. Then no bitrate table fight. Only problem: people interested to optimize the quality for their DivX/Portable player... at moderate bitrate won't learn anything by a test focusing on... unoptimized settings (CBR instead of VBR).

If a majority of people are rather interested by 128 kbps VBR => test everything with VBR and discard all encoder which doesn't offer either VBR coding or simply a ~128 kbps VBR setting.

Simple, isn't it? 
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-16 00:47:14
Thanks for accepting l3enc.  Very long ago i had a few mp3's with a short burst of noise at the start, now i know why.

As for the other contenders, since we are testing encoders i believe that each encoder should contend at the most optimal settings for the target bitrate, and it doesn't matter if that's cbr or vbr, just whatever is best.

Indeed a pre-test may be a good way to determine the most optimal settings, but that really means we're doing the developers' homework.
Title: Autumn 2006 Listening Test
Post by: Dologan on 2006-08-16 04:41:01
Ahhh, am I too late?? I wanted to suggest slipping a standard iTunes AAC @ 128 kbps as "pseudo-high" anchor.  Might be interesting to see if you could say AAC really is "out of MP3's league" or if MP3 in general (...ok, Lame ) is a good contender. Ok, shut me up...

(Hey! First post in many many months! Yay!)
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-16 04:56:48
Ahhh, am I too late?? I wanted to suggest slipping a standard iTunes AAC @ 128 kbps as "pseudo-high" anchor.  Might be interesting to see if you could say AAC really is "out of MP3's league" or if MP3 in general (...ok, Lame ) is a good contender. Ok, shut me up...

(Hey! First post in many many months! Yay!)

In the multiformat test, iTunes AAC was statistically tied with LAME -V5 --vbr-new. sorry
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-16 08:36:26
As I mentioned above, the FHG Fastenc encoder forces simple stereo in VBR mode. So I suggest testing it in CBR (even if all other contenders are in VBR), because I'm sure it will perform better with CBR.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 09:00:45
Guru, a poll would ease up everything, but in the end, we won't know if CBR or VBR was better for certain encoders. For this purpose, we really have the two options only: lots of pre-tests or two big listening tests.
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-16 10:23:24
If you view the test as a study that should have some authorative value over which encoder is best and how they compare, a poll isn't very scientific.

What about mailing the developers of each encoder and asking them?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 10:29:04
Well, for small groups or individual developers (LAME, AoTuV, etc.) this is an option, but who is responsible for iTunes' MP3 encoder? Who is responsible for WMPs encoder? And I already made the experience with writing mails to big companies and the time it takes to receive a reply.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 10:53:54
Going to encode to iTunes VBR @ 128 kbps to see what the average bitrate is. Should the other settings be like in Roberto's test?
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 11:06:46
Who is responsible for WMPs encoder?

I think they are: Audio & Multimedia (http://www.iis.fraunhofer.de/contact/index.html?bf=amm). But what's not clear with their encoder?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 11:21:20
While the encoder is based on FastEnc, I am sure MS did some own tuning. And what is unclear? If we should use VBR or CBR.
Title: Autumn 2006 Listening Test
Post by: fpi on 2006-08-16 11:25:06
I don't understand the point in testing CBR when a VBR option is available. As far as I know the only advantage of VBR should be its superior quality at the same average bitrate of CBR. If, for a particular codec, VBR has a lower quality than CBR, then it's a developers fault, in that case they should have provided only the CBR mode. So I don't see the point in testing CBR when a VBR option is available and that VBR option is not tagged by the developers as "experimental" or "you should use CBR, VBR option is provided here only to be useful for that particular case....".
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 11:36:29
WMP doesn't come with a VBR option in first place. Maybe MS tuned the CBR encoder only and then focused on WMA - no idea. The problem with FhG encoders - as far as I can tell - is that each company might have done their own tunings unless this was prohibited from FhG's side. Therefore, you cannot say that FhG performed like this or that.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 11:37:10
As I mentioned above, the FHG Fastenc encoder forces simple stereo in VBR mode. So I suggest testing it in CBR (even if all other contenders are in VBR), because I'm sure it will perform better with CBR.

It's maybe for a good reason, who knows... The feature, or this bug, is present for years. Even Audition 2.0 (Trial version) embedd a MP3 encoder forcing joint-stereo. On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

Quote
we won't know if CBR or VBR was better for certain encoders.

Quote
If you view the test as a study that should have some authorative value over which encoder is best and how they compare, a poll isn't very scientific.

It doesn't have a scientific purpose. And it's not intended to tell us which kind of coding is supposed to be better for each encoder. It's just a way for us to make a decision and to avoid inextricable problems. And yes, final decision would appear as an arbitrary one
If people are interested to see how various MP3 encoders perform compared to LAME, I'd tend to say that it wouldn't be a bad idea to test them all with similar settings: VBR against VBR, CBR against CBR. VBR is per default supposed to offer a better quality (or efficiency) than CBR, especially on hard-to-encode samples (precisely what we're used to include in listening tests). HELIX, Apple, Fraunhofer... have all put energy and money to develop a VBR mode; this mode is offered to the public; there is no contraindication nor warning against VBR in the manual telling to users that VBR isn't tuned enough. And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).

I'm not against the idea of questionning developers themselves, but as you said, answer may take a long time. I'm not against making a pre-test, but it looks like filling ourselves a hole in the encoder/software's manual (and I'm very sceptical about the success of such pre-tests -- which are requiring as energy as the final test: we can't count on a pre-test to help us and spoil it by a lack of samples/testers). The idea of having two distinct tests (one for CBR, another for VBR) is also a good one, but again I'm not sure that many people are interested to spend free time to test twice what is often considered as an outdated format (MP3), including what is often perceived as outdated encoders (Fhg, iTunes), with one test dedicated to what is considered with right as a wrong coding method (CBR).

... just my 2 ct
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 11:38:58
Isn't the stereo collapse bug something in the unofficial encoder(s) only?
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-16 11:43:11
... On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

This bug is just in the Fastencc from rarewares, it is not present in the newer (WMP) fastenc.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 11:44:32
And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).


WMAPro is developed entirely by Microsoft and WMP / WME offer VBR encoding. WMP's MP3 encoder on the other hand is CBR only. There is no VBR option provided. Using MMJB or AA is an option, but just like for WMP and its CBR encoder, the encoders are not only FhG, but FhG + company tunings.



... On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

This bug is just in the Fastencc from rarewares, it is not present in the newer (WMP) fastenc.


Yes, and the fastencc from RRW is not an official build as far as I know. Some developer published the libraries and some other developers wrote a front end for them (AFAIK)
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 12:01:58
iTunes encoded 47 tracks already and none of them has a bitrate higher than 140 kbps. The encoding speed (WAV to MP3) on my Pentium 4, 3 GHz, Hyperthreading and 1 GB RAM is 20.8x.

For longer tracks (over 4 minutes), the speed reaches 24x.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 12:10:27

... On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

This bug is just in the Fastencc from rarewares, it is not present in the newer (WMP) fastenc.

Not only in this unofficial encoder; see http://www.hydrogenaudio.org/forums/index....ost&p=13607 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=16&t=1438&hl=collapse&view=findpost&p=13607)
Official product including "SlowEnc" were also affected. But it doesn't really matter, because we're mostly interested I suppose by the fast "Fastenc", and not the legacy one which was so slow (and buggy).
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-08-16 12:22:07
Regarding Apple, if there are specific questions I could direct them to the right persons.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 12:22:20
WMP's MP3 encoder on the other hand is CBR only. There is no VBR option provided. Using MMJB or AA is an option, but just like for WMP and its CBR encoder, the encoders are not only FhG, but FhG + company tunings.

Are you sure that these encoders are also tuned by these companies?
Anyway, if the Fhg "generic" encoder really benefits from external tuning, the question isn't only to see which mode (VBR or CBR) is better but also which companie is providing the best Fhg-mod encoder. In this case, we don't have the choice : we can't mail to the developer (hello, is your encoder better than your direct competitor?... guess the answer!  ) and we're forced to perform intensive pre-tests... and just for Fhg encoder! Then iTunes, then Helix...

Now you can see the problem with pre-test: it's a neverending task.
Title: Autumn 2006 Listening Test
Post by: vlada on 2006-08-16 12:31:44
An alternative choice would be "112 kbps + VBR" which would correspond to a bitrate higher than 112 kbps and --who knows-- maybe as close to 128 kbps than the previous setting. But as Roberto explained it this choice would handicap iTunes encoder (as it handicap it in 2004).


Why sould this handicap the iTunes encoder? If this is the best setting you can set to get files with average bitrate ~130kbps, then it is their faul the didn't implement a better VBR. Or use 128 CBR if it will produce better results.

So assuming that iTunes 112@VBR is lower quality than 128@CBR, we still don't know how iTunes 128@VBR would perform against CBR@128. As I said, VBR@128 won't go below 128 kbps. Therefore it's hard to imaging this setting performing poorer than 128@CBR! So what we really need to know is the average bitrate of iTunes VBR at "128 kbps". If the average bitrate isn't too far from 128 kbps then iTunes VBR would easily be include in the test. If the bitrate is a bit too high (138...140 kbps) then we could easily increase a bit the setting for HELIX, FHG (and also GOGO). Then we would perform a listening test at ~135...140 kbps instead of a ~128...132 kbps one. Not a big issue in my opinion.


I think if iTunes 112 kbps + VBR would produce bitrate which would be close to 128 kbps, I would go with it. If you have a VBR encoder which would produce lower bitrates then 128 kbps with 112 kbps set as minimum bitrate, I wouldn't call such encoder VBR. It is a hybrid between CBR and VBR, but much closer to CBR.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 12:43:39
Why sould this handicap the iTunes encoder? If this is the best setting you can set to get files with average bitrate ~130kbps, then it is their faul the didn't implement a better VBR. Or use 128 CBR if it will produce better results.

The handicap would come from a too low bitrate (it was the case in 2004; situation is maybe different with latest iTunes). But if bitrate is indeed close to 130 kbps then no problem
Title: Autumn 2006 Listening Test
Post by: fpi on 2006-08-16 12:48:49
If people are interested to see how various MP3 encoders perform compared to LAME, I'd tend to say that it wouldn't be a bad idea to test them all with similar settings: VBR against VBR, CBR against CBR. VBR is per default supposed to offer a better quality (or efficiency) than CBR, especially on hard-to-encode samples (precisely what we're used to include in listening tests). HELIX, Apple, Fraunhofer... have all put energy and money to develop a VBR mode; this mode is offered to the public; there is no contraindication nor warning against VBR in the manual telling to users that VBR isn't tuned enough. And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).

I'm not against the idea of questionning developers themselves, but as you said, answer may take a long time. I'm not against making a pre-test, but it looks like filling ourselves a hole in the encoder/software's manual (and I'm very sceptical about the success of such pre-tests -- which are requiring as energy as the final test: we can't count on a pre-test to help us and spoil it by a lack of samples/testers). The idea of having two distinct tests (one for CBR, another for VBR) is also a good one, but again I'm not sure that many people are interested to spend free time to test twice what is often considered as an outdated format (MP3), including what is often perceived as outdated encoders (Fhg, iTunes), with one test dedicated to what is considered with right as a wrong coding method (CBR).


Totally agree with that
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 13:04:27
Regarding Apple, if there are specific questions I could direct them to the right persons.


Well, it would be nice to know if they recommend CBR or VBR.

iTunes is encoding track 261/519.

WMP's MP3 encoder on the other hand is CBR only. There is no VBR option provided. Using MMJB or AA is an option, but just like for WMP and its CBR encoder, the encoders are not only FhG, but FhG + company tunings.

Are you sure that these encoders are also tuned by these companies?
Anyway, if the Fhg "generic" encoder really benefits from external tuning, the question isn't only to see which mode (VBR or CBR) is better but also which companie is providing the best Fhg-mod encoder. In this case, we don't have the choice : we can't mail to the developer (hello, is your encoder better than your direct competitor?... guess the answer!  ) and we're forced to perform intensive pre-tests... and just for Fhg encoder! Then iTunes, then Helix...

Now you can see the problem with pre-test: it's a neverending task.


The FhG problem came into my mind just now. I cannot say for sure if MS really tweaked the FhG encoder. I cannot say for sure if MusicMatch or Adobe did. But it's possible. If you look at it now, deciding which codec to use based on popularity is not a bad idea for this specific case. Adobe Audition is not freeware and it's not something you would use to encode CDs. MMJB and WMP are better programs for such tasks and are also more user friendly. Since WMP is installed with Windows, I thought it would be best to test that. MMJB on the other hand offers VBR and is also good for casual usage.

Food is waiting for me.

Edit: BTW, I contacted someone from Real and asked what settings he recommends. I am waiting for the reply, though. Gabriel, what do you recommend for LAME? -V5 --vbr-new?
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 13:16:37
If you look at it now, deciding which codec to use based on popularity is not a bad idea for this specific case.

I agree. The subject of this test (MP3, ~128 kbps) is (or was?) also something popular. The choice of a popular and unexpensive software makes more sense than a professional editing tool.

But correct me if I'm wrong: to access to the MP3 encoder within WMP (at 128 kbps), you have either to pay or to manually change the name of one .dll, right? In this case, could we consider WMP encoder as a popular one?
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-16 13:25:06
...
But correct me if I'm wrong: to access to the MP3 encoder within WMP (at 128 kbps), you have either to pay or to manually change the name of one .dll, right? In this case, could we consider WMP encoder as a popular one?

it is an ACM codec (most probably fastenc based).
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 13:26:50
I didn't edit anything and I got it with WMP 10.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 13:42:59
You must be right. I didn't remember well the debate on this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=26956). MP3 encoder is available in WMP10; manual changes are only necessary to make it available for all softwares using the ACM engine.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 13:55:51
IIRC, you also had to add or modify some registry keys to make WMP 9 encode to MP3.

BTW, iTunes is encoding track 485/519 and as far as I can tell by looking at the reported speed, it's doing so at ~22x. CPU has 48 °C in the meanwhile.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-16 13:56:24
I updated my 128 kbps bitrate table with itunes VBR 128 kbps:

http://www.hydrogenaudio.org/forums/index....st&p=421284 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=38955&view=findpost&p=421284)

I used iTunes v. 6.05.20 with these settings:

stereo bitrate 128 kbps, vbr: enabled, quality: highest, stereo mode: joint, smart adjustments: enabled, filter below 10 Hz: enabled

I checked the bitrates with Mr Questionman v.0.81b and foobar2000 v.0.9.3.1.


Edit: fixed MrQ version
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 13:57:57
Thanks Alex, I am also going to post my bitrate table in a few minutes (iTunes only).
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 14:20:21
I didn't edit anything and I got it with WMP 10.

There is a newer 3.4.0.0 version of l3codec.acm that comes with WMP11 beta (it is possible to extract the binary without actual wmp installation). I tried to encode with 128k Low and 128k High using EAC, and then checked the resulting files. foobar2000 said "No differences in decoded data found" and Mr QuestionMan said the files were encoded with "MPEG 1 Layer III FhG".
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-16 14:22:21
Has decoder clipping ever considered as a factor that may have some effect to the listening test results?

Some of my bitrate test tracks produce extremely high peak values with iTunes VBR MP3.

This is the worst of them:

Code: [Select]
File Name : Garbage - Bleed Like Me.mp3
File Path : D:\test\iTunes_VBR128\Garbage - Bleed Like Me.mp3
Subsong Index : 0
File Size : 4 135 339 bytes
Last Modified : 2006-08-16 16:10:26
Duration : 4:01.934 (10669295 samples)
Sample Rate : 44100 Hz
Channels : 2
Bitrate : 137 kbps
Codec : MP3
Encoding : lossy
Tag Type : id3v2|id3v1
Track Gain : -9.61 dB
Track Peak : 1.642079
<ENC_DELAY> : 0
<ENC_PADDING> : 0
<EXTRAINFO> : VBR
<MP3_ACCURATE_LENGTH> : yes
<MP3_STEREO_MODE> : joint stereo

The source file looks like this:

Code: [Select]
File Name : Garbage - Bleed Like Me.ape
File Path : E:\test\Convert\LL\Garbage - Bleed Like Me.ape
Subsong Index : 0
File Size : 27 576 655 bytes
Last Modified : 2006-03-23 18:37:42
Duration : 4:01.867 (10666320 samples)
Sample Rate : 44100 Hz
Channels : 2
Bits Per Sample : 16
Bitrate : 912 kbps
Codec : Monkey's Audio (Normal)
Encoding : lossless
Tag Type : apev2
Embedded Cuesheet : no
Audio MD5 : C7D86603166482C328E9BF0015A27C79
Track Gain : -9.78 dB
Track Peak : 0.999969
<FLAGS> : 32
<VERSION> : 3.99
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 14:27:27
http://www.hydrogenaudio.org/forums/index....showtopic=47452 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47452)

Average bitrate of iTunes is 134 kbps. 143 kbps was the highest bitrate.

Going to get some fresh air now that the damn rain finally stopped after one week (not that it rained one week without pause, but chances are good that the weather will finally get sunny and warm).
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-08-16 14:48:06
Thank you for the tables (I can't build one at the moment: I'm on my notebook, iTunes doesn't work anymore and I don't have access to my usual samples/tracks gallery).
From both tables 128 kbps appears as the minimal bitrate. I suppose that iTunes MP3 VBR encoders still uses a minimal bitrate floor corresponding to the selected bitrate). Average bitrate is near 135 kbps. What matters in this case isn't the gap between iTunes' average bitrate and 128 kbps (+ 7 kbps) but the gap with LAME -V5. The latter also tend to be 130...135 kbps. In other words, iTunes 128@VBR and LAME@V5 are very close each others (bitrate-wise); a comparison would be interesting IMO. With both HELIX and FHG finely tunable VBR mode, I believe that we should be able to start a VBR-only listening test with very limited bitrate discrepencies between all competitors (if people agree with this idea).
Title: Autumn 2006 Listening Test
Post by: AtaqueEG on 2006-08-16 15:05:28
Regarding Apple, if there are specific questions I could direct them to the right persons.


When are we getting gapless playback on iPods?

(Off-topic, I know, but I couldn't help myself)
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 15:16:25
[...] FHG finely tunable VBR mode, I believe that we should be able to start a VBR-only listening test with very limited bitrate discrepencies between all competitors (if people agree with this idea).

I believe that FhG "CBR Joint Stereo" would perform better than FHG "VBR Stereo" in the planned bitrate range.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-16 15:19:12
[...] FHG finely tunable VBR mode, I believe that we should be able to start a VBR-only listening test with very limited bitrate discrepencies between all competitors (if people agree with this idea).

I believe that FhG "CBR Joint Stereo" would perform better than FHG "VBR Stereo" in the planned bitrate range.

This is also my thinking. I've posted this two times so far and it seems like nobody is listening to me
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-16 15:30:41
Believing and thinking do not count here.

Tests?

EDIT

If no usable or trustable results from public tests are available you should provide a sample that can demonstrate the difference and include ABX logs and the encoder & settings details.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 15:42:15
Believing and thinking do not count here.

Tests?

What kind of tests do you need? I can easily (10/10) ABX FhG 3.4.0.0 128k from the original with pop music, but I don't have FhG VBR encoder to compare. If I had one, how would I compare then? (BTW, there may be another satisfactory explanation why JS@128 will be better than Stereo@128  ).
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-16 15:52:20
Believing and thinking do not count here.

Tests?

What kind of tests do you need? I can easily (10/10) ABX FhG 3.4.0.0 128k from the original with pop music, but I don't have FhG VBR encoder to compare. If I had one, how would I compare then? (BTW, there may be another satisfactory explanation why JS@128 will be better than Stereo@128  ).
Check my edit.

In general, we should not claim anything if others cannot try to reproduce the findings. (This is not my personal "rule" - you know what I mean.)
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 16:00:14
(This is not my personal "rule" - you know what I mean.)

But requiring to verify if JS@128 is superior to Stereo@128 is senseless and simply absurd.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-08-16 16:10:32
okay, I decided to do a little test:
Encoder: EasyMP3 (uses Fastenc)
CBR: 128kbps
VBR: VBR quality 4

Sample: airscape1 VBR bitrate: 128kbps ABX: 7/7 (VBR has more pre-echo)
Sample: airscape VBR bitrate: 123kbps ABX: 6/7 (sounds quite similar)
Sample: latenight VBR bitrate: 114kbps ABX: 7/7 (VBR has more pre-echo and smearing)
Sample: scooter2 VBR bitrate: 133kbps can't ABX
Sample: scooter_fixed VBR bitrate: 116kbps ABX 7/7 (VBR has some additional noise and pre-echo)
Sample: time VBR bitrate: 120kbps ABX: 7/7 (noise and distortions in VBR)

edit- of course this test was CBR against VBR...

edit2- it's worth mentioning that the difference was sometimes obvious
Title: Autumn 2006 Listening Test
Post by: SirGrey on 2006-08-16 16:21:13
guruboolez
Quote
With both HELIX and FHG finely tunable VBR mode, I believe that we should be able to start a VBR-only listening test with very limited bitrate discrepencies between all competitors (if people agree with this idea).

You have my vote.

Egor
Quote
I believe that FhG "CBR Joint Stereo" would perform better than FHG "VBR Stereo" in the planned bitrate range.

Egor, I strongly recommend to use your "believing system" locally, but not on this board, because you simply violating TOS.
Please, stop trolling in this thread.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-16 16:25:57
But requiring to verify if JS@128 is superior to Stereo@128 is senseless and simply absurd.

Is it? We are not speaking about LAME.

Also, it was JS 128 CBR vs. stereo 128 VBR

Actually, depending on the used program the possible main choices are:

- JS CBR
- stereo CBR
- JS VBR
- stereo VBR

+ some additional options like the "general quality" and lowpass settings (if these happen to work properly).

If I recall correctly, also intensity stereo is possible with some programs that use FhG encoder.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 16:33:29
Egor, I strongly recommend to use your "believing system" locally, but not on this board, because you simply violating TOS.

I believe you've misinterpreted me. 
Quote
believe (verb)
...
2. judge or regard; look upon; judge; "I think he is very smart"; "I believe her to be very smart"; "I think that he is her boyfriend"; "The racist conceives such people to be inferior"
Syn.: think, consider, conceive
...


Edit.
Actually, depending on the used program the possible main choices are:

...
- JS VBR

OK, what software does support this?
Title: Autumn 2006 Listening Test
Post by: fpi on 2006-08-16 17:06:16
guruboolez
Quote
With both HELIX and FHG finely tunable VBR mode, I believe that we should be able to start a VBR-only listening test with very limited bitrate discrepencies between all competitors (if people agree with this idea).

You have my vote.


I also agree 
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 19:11:48
Nero is CT, right? Sorry if this is a dumb question, but is CT's MP3 encoder based on FhG, or did they build one from scratch?

BTW, I am asking because Nero's medium VBR settings produces files with ca. 132 kbps and also supports JS. I have to check if all frames are SS, though.

Edit: Forget it, while JS is used, all frames are SS.
Title: Autumn 2006 Listening Test
Post by: SirGrey on 2006-08-16 20:54:13
>>Sorry if this is a dumb question, but is CT's MP3 encoder based on FhG, or did they build one from scratch?
Write PM or mail to Ivan or Menno, they should know for sure.
Sebastian, as I understand, you generally dislike an idea to test vbr modes of all encoders, am I right ?
EDIT (offtopic)
>>I believe you've misinterpreted me.
Confirmed.  But anyway, TOS should not be violated.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 21:25:47
No, I don't dislike it in general, but I thought it would be good to test the best settings and considered that iTunes' VBR implementation is sub-optimal because of its design in first place. I see that iTunes VBR at 128 kbps produces an average bitrate of 134 kbps which is still acceptable, so I am going to use it in order to avoid any additional confrontation with people and stress my nerves. It's my personal opinion that not the bitrate selection (112 kbps) is to be blamed for iTunes' bad results in Roberto's test. 6 kbps shouldn't have such a big impact on quality and a true VBR encoder should be smart enough to allocate enough bits if it feels the need of it. The way I see it, iTunes developers focused on AAC and added MP3 for compatibility reasons. Since most users who utilize iTunes don't care about VBR, I also think Apple focused on CBR. I cannot say this for sure - I understand this is a problem (and it really is).

Edit: BTW, Ivan said he thinks CT is based on FhG.

As for FhG, this is a bit more complex since FhG is used in several products which might or might not have special tunings made by the companies which develop the various products. Also, some products have some features disabled (Microsoft disabled VBR encoding in WMP).
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-16 21:55:26
I don't know if somebody with golden years (maybe guru?) has the time/will to do a quick tests between CBR/VBR. I don't see any problem on using itunes in CBR, if it's better prepared with it.

Garf offered to ask the question to the right apple people.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 21:58:30
It was Gabriel IIRC (too lazy to go back and see). I am looking forward to hearing from him soon.
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-16 21:59:24
Perhaps MS were the only ones who tested it and found CBR was better, dropped VBR for simplification and to make sure users wouldn't make poor encodes.

Simply because WMP is so common and commonly used for encoding mp3's by the general public, i'd say it's not a bad idea to take it and test it at 128 cbr. It'll make the test relevant to a broader audience.

Or they just wanted to give WMA an advantage...
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 22:02:46
The problem here is that we all speculate. I do regarding iTunes, you do regarding MS and MP3... Nothing is certain unless we do some tests that show this. Unfortunately, we don't have the time and resources to conduct such pre-tests.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-16 22:18:58
Just quickly tested the mp3surround encoder (http://www.all4mp3.com/tools/sw_fhg_cl.html) in stereo mode and it turned out, that it offers even better quality than 3.4.0.0 FhG ACM encoder - it requires more careful listening to ABX/distinguish a pop song @128k from the original.
Mp3sEncoder.exe allows to encode only in CBR mode too, but it surely comes from FhG:
Quote
Fraunhofer IIS MP3 Surround Commandline Encoder V1.0

Encoder-Library V04.00.03 (build 2005-09-14)

© 1996 - 2005 Fraunhofer IIS
© 2004 Fraunhofer IIS and Agere Systems Inc.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-16 22:19:25
One important thing is to determine what is the test for:

1) to see how the usual population encoding habits perform against each other.
2) to test the best encoder option for each encoder.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-16 22:28:01
What is that supposed to mean? I understand your post, but I don't see what you want to point out. There are several FhG implementations and you cannot say "we use program A because it's better than B" because you don't know if it really is better (related to problem WMP vs. MMJB vs. Adobe Audition vs. Nero MP3 vs. whatever). Therefore, you have to choose another criterion otherwise you cannot test FhG at all. Therefore, popularity is second criterion.

As for iTunes, the problem with best option is that we don't know what the best option is.
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-16 23:03:32
The problem here is that we all speculate. I do regarding iTunes, you do regarding MS and MP3... Nothing is certain unless we do some tests that show this. Unfortunately, we don't have the time and resources to conduct such pre-tests.


There's nothing wrong with speculating if the situation leaves you little choice. As long as you apply some logic consistently, there'll be less of a basis for criticism after the test. I mean serious, founded criticism, not the usual crap.

My proposal for test method would be:
Test the most common software encoders feature in, if they offer VBR and don't recommend against it in the manual, test VBR because it should be better, else CBR.

By this logic you test Helix, Lame, iTunes in VBR, wmp's fhg in CBR.

edit: just curious, but what's mmjb's market share and what is it's default setting?
Title: Autumn 2006 Listening Test
Post by: greynol on 2006-08-16 23:24:05
Test the most common software encoders feature in, if they offer VBR and don't recommend against it in the manual, test VBR because it should be better, else CBR.

Unless js is used in CBR but not in VBR, then it isn't as simple as that.
Title: Autumn 2006 Listening Test
Post by: Steve999 on 2006-08-17 00:11:10
I wouldn't mess with recent MMJB.  I used to like it very much for 160 -- 220 kbps ripping, but recent versions (9 and 10) are so awful in design that they are unusable for me, and from what I read, many other people.  Yahoo bought it, I think, and has really dropped the ball and done some bad things, including deceptive upgrade offers and making old keys expire without prior notice.  I had a hard time getting it to do anything in MP3 other than 128 CBR, for inexplicable and unfathomable reasons.  I would imagine their market share has plummeted.

edit: just curious, but what's mmjb's market share and what is it's default setting?
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 02:20:18
What is that supposed to mean?


If we want to (1) to see how the usual population encoding habits perform against each other.
We would probably end up with a mixture of CBR against VBR. And it would probably not be the best settings, but the most used.

If we want to (2) to test the best encoder option for each encoder.
we would have to test to see at each encoder, what setting is the best.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-17 09:18:13
kwanbis, did you read my whole post? 2 is simply not possible, so we have to mix 1 and 2.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-17 09:57:39
Actually, depending on the used program the possible main choices are:
...
- JS VBR

OK, what software does support this?

At least Musicmatch Jukebox supports joint stereo. However, it seems that MMJB chooses the used mode automatically (there's not option for selecting JS).

I encoded my test file set with MMJB 8.10.169. (Yes, it is old, I know. It is the last version I have wanted to keep as a sort of reference. I tried MMJB 10 when it was new and at least the visible MP3 encoding options were unchanged.)

I used the VBR 55% setting. Processing Level was set to "Normal", Maximum Bandwidth to "Let encoder choose".

22 of the 25 test tracks seem to be Joint Stereo. Here is how EncSpot Pro sees the files:

(http://www.adart.pp.fi/ha/pix/mmjb55.png)


The average bitrate is nicely 135.2 kbps (about the same as LAME -V5 --vbr-new produces). EncSpot's "complete scan" is needed for checking this because it cannot use the VBRI header info. Also, most other programs show wrong bitrates with FhG VBR files.

I have only one other program that can encode FhG VBR. It's WaveLab. It has only five predefined VBR quality levels from "lowest" to "highest". It has also an option for joint stereo (allow mid/side coding), but that option does not seem to stick. If the options window is opened again "allow mid/side coding" is always resetted to disabled. Also it seems that WaveLab does not write VBRI headers. The resulting files do not contain VBR headers at all.

In general, I think that FhG VBR is a can of worms that probably should not be opened. A lot more reliable information would be needed for making the correct front program and encoder settings choices.


Edit: a couple of typos
Edit 2: image link fixed
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 14:30:22
kwanbis, did you read my whole post? 2 is simply not possible, so we have to mix 1 and 2.

If 2 is not an option, *i* think we should try 1, not a mix.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-17 14:42:02
Well, this would mean using 128 kbps CBR for everything.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 17:11:14
no, i think people use LAME with vbr, for example, and iTunes is ABR(?)?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-17 17:11:59
iTunes seems to be ABR, but if you are going with popularity only, most people use iTunes in CBR mode I guess.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 17:45:43
tought call
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-17 18:34:52
Once again, on my part, I vote for CBR only.  You could add the LAME -v5 --vbr-new as a high anchor, if you want, since we already proved it's transparent to most users.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 20:12:50
that could be a good middle-road solution ... and also LAME CBR 128? I allways wondered how it compared.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-17 20:23:40
I'd say we test everything in VBR mode. If there should be demand for another midrange bitrate MP3 test, we could use CBR then. For FhG based, I think I will use MMJB or Nero.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-17 20:34:07
I already asked about Helix VBR settings. No one has answered yet. The encoder is nicely configurable, but without testing I can only try to guess what would be suitable.

Sebastian, have you asked for advice at https://helixcommunity.org/ (https://helixcommunity.org/) ?
Title: Autumn 2006 Listening Test
Post by: rjamorim on 2006-08-17 20:53:51
I am strongly against a CBR-only test. if LAME shines most on VBR, and people concerned with quality (the ones that care about listening tests) use VBR, why test CBR? That would yield useless results.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 21:35:40
I am strongly against a CBR-only test. if LAME shines most on VBR, and people concerned with quality (the ones that care about listening tests) use VBR, why test CBR? That would yield useless results.

I have to agree with roberto. 
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-17 22:32:20
Funny, a few moments before you suggested we should use CBR.

Roberto, what's your opinion regarding FhG? Would you use MMJB / Nero because they offer VBR mode, or would you use WMP and CBR?
Since iTunes averages 135 kbps when set to encode to 128 kbps VBR, it has a similar bitrate to LAME.
Title: Autumn 2006 Listening Test
Post by: kwanbis on 2006-08-17 23:00:15
Funny, a few moments before you suggested we should use CBR.

No, i did not, i said that if you didn't wanted option 1, i would do option 2 and not a mix. But i didn't wanted to mean not to mix vbr/cbr, but not to mix criteria for selecting. But doesn't means i'm right.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-17 23:04:12

Funny, a few moments before you suggested we should use CBR.

No, i did not, i said that if you didn't wanted option 1, i would do option 2 and not a mix. But i didn't wanted to mean not to mix vbr/cbr, but not to mix criteria for selecting. But doesn't means i'm right.


that could be a good middle-road solution ... and also LAME CBR 128? I allways wondered how it compared.


Anyways, forget about it - didn't want to offend you, sorry.

Does anyone know if the Helix encoder hosted on RW is the latest and greatest version?
Title: Autumn 2006 Listening Test
Post by: rjamorim on 2006-08-18 02:58:53
Roberto, what's your opinion regarding FhG? Would you use MMJB / Nero because they offer VBR mode, or would you use WMP and CBR?.


I didn't even know MMJB was still alive...

As far as popularity is concerned, I would go with WMP in a heartbeat. Nero must offer one of the hardest, most arcane ways to rip a CD to MP3, so I don't think people are flocking to it for their ripping needs (it actually took me quite some time to realize that Extras -> Save tracks meant "CD ripping")

Of course, the ideal would be doing a pre-test to see which one is better. But honestly, that's a waste of effort for an encoder that will probably lose anyway...

Does anyone know if the Helix encoder hosted on RW is the latest and greatest version?


Should be. You can always ask Karl... have you already mailed him about encoding hints?
Title: Autumn 2006 Listening Test
Post by: Steve999 on 2006-08-18 03:50:24
I really like the idea of 128 kbps+ vbr mp3.  Actually I'd prefer 160 kbps+ vbr MP3, but I think I'm the lone wolf on that one.

I always encode in MP3 (and will until there's another universal format) and strive for fast, convenient, well-tagged transparency on a normal recording at a reasonably low bitrate.  I think I'm a pretty normal Joe that way.  Anything to help along those lines in this test will get a lot of play on the net, IMHO.
Title: Autumn 2006 Listening Test
Post by: greynol on 2006-08-18 06:29:09
Nero must offer one of the hardest, most arcane ways to rip a CD to MP3, so I don't think people are flocking to it for their ripping needs (it actually took me quite some time to realize that Extras -> Save tracks meant "CD ripping")

I wouldn't be so sure about that.

EDIT: COLOR added.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-18 07:39:25
You can always ask Karl... have you already mailed him about encoding hints?


Of course, a few days ago.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-18 10:43:47

You can always ask Karl... have you already mailed him about encoding hints?

Of course, a few days ago.

Oh, I didn't know that. Hopefully they have some idea what should be used. One option would be to trust the default and only adjust the basic VBR quality setting.


Any comments on the decoder clipping issue I mentioned? I don't know how audible the problem is and how much the encoders differ from each other, but sometimes the difference in a decoded wave file can be quite big. Some samples can be decoded without clipping and some show heavy clipping.

As we know, this possible decoder clipping may or may not be something that users experience outside a listening test depending on if they use a decoder that can prevent from clipping or if they use MP3Gain for that.

Naturally, the gain adjustment that is applied with ABC/HR after decoding to 16-bit integer does not prevent from decoder clipping.

I made some experiments with the Garbage's "Bleed Like Me" track that I encoded with iTunes VBR 128. I converted the file with and without fb2k's clipping protection to wave and wave gained the resulting files to have the same overall volume. I cannot ABX test the files just now, but this is how the files show up in a wave editor:

(http://kotisivu.mtv3.fi/alexb/ha/grabage.png)

I'll try to ABX these and some other test tracks during the weekend and perhaps post samples and start another thread for discussion.
Title: Autumn 2006 Listening Test
Post by: rjamorim on 2006-08-18 11:21:20
Nero must offer one of the hardest, most arcane ways to rip a CD to MP3, so I don't think people are flocking to it for their ripping needs (it actually took me quite some time to realize that Extras -> Save tracks meant "CD ripping")
I wouldn't be so sure about that.


I didn't say I was sure. I said I think. There's a difference in there, if you care to look.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-18 11:43:07
http://www.freedb.org/freedb_stats.php?typ...p;topic=clients (http://www.freedb.org/freedb_stats.php?type=yearly&topic=clients)

EDIT:

Though, naturally programs like WMP or iTunes are not included. Also, we cannot know if Nero is used mainly for making direct duplicates instead of ripping.
Title: Autumn 2006 Listening Test
Post by: greynol on 2006-08-18 17:17:41
I didn't say I was sure. I said I think. There's a difference in there, if you care to look.

If I care to look?

I highlighted it.

Your opinion holds great weight though in this case I felt your projection deserved to be challenged as I have seen Nero used for things for which there are better options.

@Alex B:  Thank you.
Title: Autumn 2006 Listening Test
Post by: rjamorim on 2006-08-18 18:16:45
Your opinion holds great weight though in this case I felt your projection deserved to be challenged as I have seen Nero used for things for which there are better options.


Pretty much all features in Nero have better options elsewhere. But people tend to keep using Nero because of status quo and ease of use. Audio ripping and encoding, in particular, is an area where Nero is neither easy nor it enjoys a great status. So there.
Title: Autumn 2006 Listening Test
Post by: ezra2323 on 2006-08-21 00:45:34
Quote
Just quickly tested the mp3surround encoder in stereo mode and it turned out, that it offers even better quality than 3.4.0.0 FhG ACM encoder - it requires more careful listening to ABX/distinguish a pop song @128k from the original.


While I respect Egor and think he is a valuable participant at HA, my ears have to disagree on htis one.

Listening to Breaking Benjamin's Phobia CD (tracks 1-4) today, encdoded at 192 kbps with the Fhg Surround Sound encoder and Fhg 3.4 from WMP 11 - I found 3.4 much better. Some of the cymbals while listening to the surround sound encoder made me cringe. Maybe this only applies to grunge/metal however. His test was based on a pop song.

If anyone else has the Phobia CD, check my hearing. Maybe I am imagining it. I do not think so however.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-24 17:59:33
Sent two e-mails to Karl and Ken and no reply yet. I think I will use the Helix encoder with something like -V65 -X2. Do you guys have any recommendations?

Problem is that I am running out of time since I have to leave Germany and drive to Romania on September 2nd. I will be there for three weeks and have no idea if my dad manages to get a notebook from the company he's working for.
Title: Autumn 2006 Listening Test
Post by: Diow on 2006-08-24 19:07:38

I am strongly against a CBR-only test. if LAME shines most on VBR, and people concerned with quality (the ones that care about listening tests) use VBR, why test CBR? That would yield useless results.

I have to agree with roberto. 


I agree with you too but i'm think that's interesting make an test with cbr to prove (in flesh and blood) that vbr is better than cbr, and how much it's really is.The results will shock ANYBODY that stick use cbr instead of vbr.  And use some problematics samples will be some interesting... 
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-24 19:17:58
I agree with you too but i'm think that's interesting make an test with cbr to prove (in flesh and blood) that vbr is better than cbr, and how much it's really is.The results will shock ANYBODY that stick use cbr instead of vbr.  And use some problematics samples will be some interesting... 
Same here;  We have proved that VBR is good; now we must prove that CBR is bad, and we'll be able to deduce that VBR is better than CBR.  Perhaps including one Lame -V5 --vbr-new and one LAME -b 128, just to see how people react about them both (in double-blind)
I think it's important also to have matching bitrates.  It's been a long time since we've had a CBR test...
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-24 19:36:58
I think I already stated what encoders are going to be used. Adding a new LAME setting won't work because there are enough contenders. One (if not the main) goal of this test was to see how traditional encoders like LAME perform against fast encoders such as Helix, Gogo and FastEnc.
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-08-24 21:25:18
@ Sebastian:

This thread has some command lines that seemed to work:

http://www.hydrogenaudio.org/forums/index....531&st=169# (http://www.hydrogenaudio.org/forums/index.php?showtopic=35531&st=169#)

...if I can get the link right...

Edit: Of course you have to adjust the -V value to match the opted bit rate.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-24 22:05:46
Thanks. "seemed to work" - well, they do if they have the right syntax, but do people agree that they sound well? Are they recommended settings?
And do you guys think I should use some cryptic command line arguments or should I stick to the simple -V setting (and -X2 to write a Xing header with TOC)?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-26 17:46:18
Damn, still no reply from Helix / Real. Going to contact Karl on his private mail address. If I still don't get any answer, I think I am going with the VBR switch only (-Vwhatever -X2).

BTW, should I choose LAME 3.97b2 or b3? Gabriel?

One more thing - for Gogo, I tried encoding using -v 5 and -v 4. -v 5 averages 140 kbps and -v 4 120 kbps (20 tracks tested). The difference between Gogo and LAME seems to be around 5 kbps. Guess I will go with -v 5. Is this OK?
Title: Autumn 2006 Listening Test
Post by: pepoluan on 2006-08-26 19:01:17
And do you guys think I should use some cryptic command line arguments or should I stick to the simple -V setting (and -X2 to write a Xing header with TOC)?
Will using -X2 affect quality in some ways?
Title: Autumn 2006 Listening Test
Post by: Shade[ST] on 2006-08-26 19:17:09
One more thing - for Gogo, I tried encoding using -v 5 and -v 4. -v 5 averages 140 kbps and -v 4 120 kbps (20 tracks tested). The difference between Gogo and LAME seems to be around 5 kbps. Guess I will go with -v 5. Is this OK?
Fine with me!
I'd go with lame 3.97b3.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-26 19:23:20
And do you guys think I should use some cryptic command line arguments or should I stick to the simple -V setting (and -X2 to write a Xing header with TOC)?
Will using -X2 affect quality in some ways?


According to the documentation, it only activates writing of Xing header with TOC. So, no, it should not affect quality.

Karl just replied:

Quote
sorry for not replying sooner. Audio encoding is simply not my area of
expertise. I would ask Jon Recker <#######@real.com>, one of our two
audio codec experts, but I spoke with him, and even he has not used this
MP3 encoder much, and I would not be too hopeful of an answer from him
either. Perhaps just a forum thread, the original thread for the release
of the encoder would be the best place to find an answer from the community.

Karl.
Title: Autumn 2006 Listening Test
Post by: HbG on 2006-08-26 19:35:30
One more thing - for Gogo, I tried encoding using -v 5 and -v 4. -v 5 averages 140 kbps and -v 4 120 kbps (20 tracks tested). The difference between Gogo and LAME seems to be around 5 kbps. Guess I will go with -v 5. Is this OK?


Hmm, 140 is 12 kbps over the nominal rate of 128, 120 is 8 under it. Wouldn't it be logical to pick the rate closest to nominal?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-26 20:21:04
I think greynol was right saying that the title should be changed. Basically, the test is at ~135 kbps since that is what LAME and most other encoders output.
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-08-27 10:02:26
Quote
BTW, should I choose LAME 3.97b2 or b3? Gabriel?

3.97b3
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-28 21:32:15
I just noticed that Nero does not write a VBRI header to MP3s with variable bitrate. Wondering if it's still based on FhG - still guess so, though.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-29 12:51:53
I was away over a week. I am trying to catch up this thread now.


Quote
One more thing - for Gogo, I tried encoding using -v 5 and -v 4. -v 5 averages 140 kbps and -v 4 120 kbps (20 tracks tested). The difference between Gogo and LAME seems to be around 5 kbps. Guess I will go with -v 5. Is this OK?

I have had an impression that Gogo should be used with ABR at a bitrate like ~130 kbps. It is based on LAME 3.88 and VBR wasn't very matured at that time. Even LAME 3.9x ABR has been used at lower bitrates for example by Guruboolez until the most recent versions. I mentioned earlier that I have used command lines like this: "-a -b 192 -m j -q 2". For this test the command line could be like "-a -b 135 -m j" and the quality option 0, 1 or 2. (I have used -q 2 without any testing just because LAME 3.88 -h was mapped with the -q 2 option and it produced a good balance between the quality and speed).


Quote
sorry for not replying sooner. Audio encoding is simply not my area of
expertise. I would ask Jon Recker <#######@real.com>, one of our two
audio codec experts, but I spoke with him, and even he has not used this
MP3 encoder much, and I would not be too hopeful of an answer from him
either. Perhaps just a forum thread, the original thread for the release
of the encoder would be the best place to find an answer from the community.

I was afraid that this would be the answer. Perhaps it is then best to use the default VBR setting if it is not possible to pre-test encoders.


Quote
I just noticed that Nero does not write a VBRI header to MP3s with variable bitrate. Wondering if it's still based on FhG - still guess so, though.

Also the FhG encoder in Wavelab does not write a VBRI header for some reason.


It seems that no one has commented the decoder clipping issue I wrote about. Isn't it interesting or am I totally wrong in assuming that it may have some uncontrolled effect to the test results? (I'll try to do the listening test that I promised earlier now.)

Edit: typo
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-29 21:19:03
Well, what can you do about the clipping?
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-08-29 21:44:46
I just noticed that Nero does not write a VBRI header to MP3s with variable bitrate. Wondering if it's still based on FhG - still guess so, though.

Might just be an integration issue. You need to rewind in the stream to write the VBRI header, and this is not always possible.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-08-29 23:42:58
Well, what can you do about the clipping?

That's a good question.

Because the users can avoid the mp3 decoder clipping issue in real life should a test also do that?

In my opinion the test samples that produce clipping should be MP3gained with maximum no-clip replay gain.

Another option would be to wavegain the samples before encoding if needed, but this is more complicated and possibly would result altered MP3 files because the source files would be changed.

In general, HA should advice how the users can avoid decoder clipping for getting the best mp3 decoding results. These instructions should be as strong as the recommended LAME settings. I don't think this is anything too complicated. Similarly HA has advised for years how LAME can be used properly.

Personally, I always apply maximum no-clip album gain after encoding in case the decoded files clip (I don't use MP3Gain's undo tags because I've had problems with converting the APE tags). After that I reanalyze the files with foobar2000 in album mode so that I can use all replay gain playback settings.

It would be nice if foobar2000 9.x had all options that are available in MP3gain. The needed engine is already built-in and the same results can be achieved by manual tweaking, but that is too complex so I just analyze the files twice.
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-08-30 08:29:21
About iTunes's mp3 encoder:
the "vbr" mode can be safely used over cbr in the context of a 128-135kbps test. (according to Apple)
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-30 08:48:34
Thank you for your efforts Gabriel!
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-08-30 10:12:13
Thank you for your efforts Gabriel!

To be honest I have to admit that I only asked about it yesterday evening.
Title: Autumn 2006 Listening Test
Post by: Hollunder on 2006-08-30 10:30:43
Are you going to set the test up before you travel to Romania Sebastian?
That would be great (in case you don't get a laptop there).
I simply can't wait to try out my new old phones in a proper listening test.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-08-30 10:48:48
Happily, I just got a reply from Fraunhofer Audio and Multimedia division:

Quote
thank you for your inquiry and my apologies for not responding earlier
to your email.

I'm not sure if we are in the right position to give a strong
recommendation for your planned listening test.
One reason is, that we're often not really aware which versions of our
mp3 libraries are used in a certain customer's product.

When browsing through the discussion thread a little, we at least would
not recommend to use the famous old hacked versions of our libraries,
like the fastencc, that is a buggy version from the 90ies afaik.

From our point of view, the command-line codecs available at
www.all4mp3.com are using our latest MP3 libraries. Besides the surround
capabilities, they offer CBR-encoding in stereo and mono. The encoder
implementation is optimized for high execution speed on modern machines.
These command-line codecs have been updated last week, mostly adding new
functionality for file handling and surround operation.

All the best and have fun with your listening test

-Johannes
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-08-30 11:34:39
Are you going to set the test up before you travel to Romania Sebastian?
That would be great (in case you don't get a laptop there).
I simply can't wait to try out my new old phones in a proper listening test.


I doubt it. We are leaving in three days and we still have to discuss samples. Anyways, once I get back late September, the test will follow in a few days.
Title: Autumn 2006 Listening Test
Post by: DARcode on 2006-08-30 15:42:29
Happily, I just got a reply from Fraunhofer Audio and Multimedia division:

Quote
thank you for your inquiry and my apologies for not responding earlier
to your email.

I'm not sure if we are in the right position to give a strong
recommendation for your planned listening test.
One reason is, that we're often not really aware which versions of our
mp3 libraries are used in a certain customer's product.

When browsing through the discussion thread a little, we at least would
not recommend to use the famous old hacked versions of our libraries,
like the fastencc, that is a buggy version from the 90ies afaik.

From our point of view, the command-line codecs available at
www.all4mp3.com are using our latest MP3 libraries. Besides the surround
capabilities, they offer CBR-encoding in stereo and mono. The encoder
implementation is optimized for high execution speed on modern machines.
These command-line codecs have been updated last week, mostly adding new
functionality for file handling and surround operation.

All the best and have fun with your listening test

-Johannes

A tad off topic: anyone knows what actually happens (registry entry, file(s) saved somewhere, binary mod, etc.) when you accept the license agreement?
Title: Autumn 2006 Listening Test
Post by: mlb2gm5x on 2006-08-30 16:28:43
A tad off topic: anyone knows what actually happens (registry entry, file(s) saved somewhere, binary mod, etc.) when you accept the license agreement?

C:\Documents and Settings\UserName\Application Data\Fraunhofer\MP3sCmdlEncDec\EulaEnc.txt is created
Title: Autumn 2006 Listening Test
Post by: DARcode on 2006-09-01 01:57:51

A tad off topic: anyone knows what actually happens (registry entry, file(s) saved somewhere, binary mod, etc.) when you accept the license agreement?

C:\Documents and Settings\UserName\Application Data\Fraunhofer\MP3sCmdlEncDec\EulaEnc.txt is created

Many thanks!
Title: Autumn 2006 Listening Test
Post by: AstralStorm on 2006-09-01 10:39:46
Quote
From our point of view, the command-line codecs available at
www.all4mp3.com are using our latest MP3 libraries. Besides the surround
capabilities, they offer CBR-encoding in stereo and mono. The encoder
implementation is optimized for high execution speed on modern machines.
These command-line codecs have been updated last week, mostly adding new
functionality for file handling and surround operation.


That still leaves the case of FhG VBR vs CBR... Could you ask specifically about that?
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-01 11:04:52
That still leaves the case of FhG VBR vs CBR... Could you ask specifically about that?

This encoder doesn't support VBR.
Title: Autumn 2006 Listening Test
Post by: Enig123 on 2006-09-01 14:07:07
New FhG Surround encoder (v1.2) claimed supporting piping from stdin, but I failed use
-if - -of %d -br 128000
with foobar2000's encoder setting for this codec.

I wonder if I uses the wrong command or the support of piping is buggy.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-01 14:13:13
Yeah, piping is buggy, it requires channel count and samplerate options.

Edit. e.g. "-if - -of %d -br 128000 -c 2 -sr 44100"
Title: Autumn 2006 Listening Test
Post by: gameplaya15143 on 2006-09-02 03:08:09
One more thing - for Gogo, I tried encoding using -v 5 and -v 4. -v 5 averages 140 kbps and -v 4 120 kbps (20 tracks tested). The difference between Gogo and LAME seems to be around 5 kbps. Guess I will go with -v 5. Is this OK?
I would go with -v 5 as well.  Your bitrate test results are the same as the results of my own little test.

As for helix -V65 -X2 should be a good choice as well.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-02 12:06:45
As for FhG I suggest to useas available for instance in Musicmatch Jukebox 7.00.149 wich can be downloaded as version 7.0 from OldApps.com (http://www.oldapps.com/MusicMatch_Jukebox.htm). Highest quality setting in MMJB 7.0 is achieved by Option > Settings > Recorder > Advanced > Processing Level: Very High.

I just tested it in the context of a 96 kbps thread out of curiosity. The outstanding thing with fastenc seems to be that it is a pretty robust codec (my standard bad tonal samples were encoded at a remarkable good quality - much better than I expected, and ff123 once reported about the consistent quality of this codec). The general quality is very good too to my ears (though I wouldn't call it outstanding).

From 25 samples I encoded (various genres of popular music) the average bitrate was 130+/-10 kbps in 16 cases, the lowest bitrate was 95 kbps, the highest was 141 kbps. So taking from that VBR 60% seems to meet our target bitrate pretty well.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-09-02 13:22:03
As for FhG I suggest to use
  • Fastenc VBR 60%, highest quality setting
as available for instance in Musicmatch Jukebox 7.00.149 wich can be downloaded as version 7.0 from OldApps.com (http://www.oldapps.com/MusicMatch_Jukebox.htm). Highest quality setting in MMJB 7.0 is achieved by Option > Settings > Recorder > Advanced > Processing Level: Very High.

I just tested it in the context of a 96 kbps thread out of curiosity. The outstanding thing with fastenc seems to be that it is a pretty robust codec (my standard bad tonal samples were encoded at a remarkable good quality - much better than I expected, and ff123 once reported about the consistent quality of this codec). The general quality is very good too to my ears (though I wouldn't call it outstanding).

From 25 samples I encoded (various genres of popular music) the average bitrate was 130+/-10 kbps in 16 cases, the lowest bitrate was 95 kbps, the highest was 141 kbps. So taking from that VBR 60% seems to meet our target bitrate pretty well.

Aren't the VBR MP3s produced by Fastenc in plain Stereo mode?

EDIT - I just downloaded the MMJB 7 thing and the VBR MP3s are in (surprise, surprise) plain Stereo. I'm not saying this is a bad codec for VBR, it just has this handicap and it would (probably) do much better with JS. (this is just my personal opinion)

J.M.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-02 13:39:29
Aren't the VBR MP3s produced by Fastenc in plain Stereo mode?

Encspot just says stereo and doesn't show stereo usage details. So I'm afraid you are right.
Maybe I was just too impressed by my 96 kbps listening test where a) vbr was the only usable solution because of the 12 kHz lowpass in all the other cases and b) the not-so-good quality of fastenc CBR 96 on my standard bad samples, especially trumpet.

Just tried fastenc CBR 128 (Option > Settings > Recorder > Advanced > Processing Level: Normal) on my standard bad samples, and it's good too.
Because of joint stereo usage I too think this is the better way to go.

Sorry for the confusion.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-02 14:16:55
Should FhG be the only encoder tested in CBR mode (WMP)?

Edit: Except the low anchor of course. By the way, I am leaving tomorrow. Wish me luck since we're going by car and I never drove that long before and have no idea how the streets and other drivers are in Hungary and Romania.
Title: Autumn 2006 Listening Test
Post by: pepoluan on 2006-09-04 14:42:22
Why not? If the FhG somehow kicks the other encoder's a$$es... doesn't it prove the superiority of FhG CBR over <whathaveyou> VBR?
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-09-04 20:07:04
Why not? If the FhG somehow kicks the other encoder's a$$es... doesn't it prove the superiority of FhG CBR over <whathaveyou> VBR?

Of course  But if it doesn't, be sure that several people will deny all validity to a (unfair) comparison between <whathaveyou> and a handicaped Fhg encoder. Unless you can prove first that Fhg performances are really optimal with CBR and not VBR, the inclusion of a CBR encoder in a VBR arena [LAME, HELIX, iTunes] is highly polemical.
And the problem is not only CBR vs VBR, but also 128 kbps vs 135 kbps. For some people such discrepancy in bitrate isn't acceptable at all and would prove a bit more that the whole test is a biased one, etc, etc...
Title: Autumn 2006 Listening Test
Post by: Firon on 2006-09-04 21:33:38
I would prefer multi-format at 96kbps instead of 80kbps.
Title: Autumn 2006 Listening Test
Post by: [JAZ] on 2006-09-04 21:38:16
That reminds me that the surround encoder supporst that CBR mode which is scaled (  i.e. any bitrate ). so.. it shouldn't be a problem to use it in CBR at 135kbps, isn't it?
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-04 22:15:10
... And the problem is not only CBR vs VBR, but also 128 kbps vs 135 kbps. ...

Sure, but I think we have to stick with that. More generally speaking with any encoder we choose a setting which is plausible to be good but we can't be sure of. As for VBR we always have the difficulty that settings are not exactly comparable, but AFAIK nobody seriously complained about it.

For the encoder settings we have to meet decisions. Some choices are rather obvious (Lame -V5), others are not so it is discussed here. And as long as no kind of war comes up here everything is fine. With any decision it can happy that there are people which are not happy with it.

We should use for each encoder that setting which is supposed to be most widely supported in the discussion  here.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-09-05 17:30:11
As for VBR we always have the difficulty that settings are not exactly comparable, but AFAIK nobody seriously complained about it.

Ask Roberto...
...or read old debates about collective listening tests.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-23 00:12:32
I am back folks, but am very tired right now after 21 hours of driving. If you ever want to commit suicide, get a car and drive to Romania - the roads there are perfectly suitable for this task.

Will get back to you Sunday...
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-24 17:07:50
OK, so what do we do with FhG guys? We still have to decide which encoder to use and with what settings.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-24 18:41:16
OK, so what do we do with FhG guys? We still have to decide which encoder to use and with what settings.

I suggest to use Fastenc CBR 128.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-09-24 19:11:21

OK, so what do we do with FhG guys? We still have to decide which encoder to use and with what settings.

I suggest to use Fastenc CBR 128.

... from WMP
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-24 21:58:15
... from WMP

I agree, it would be more useful/practical. MP3 encoder shipped with the new WMP is supposedly more widespread (as WMP is an essential modern Windows component).
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-09-24 22:04:32
OK, so what do we do with FhG guys? We still have to decide which encoder to use and with what settings.

I would say that the answer should depend on the choice of all other competitors. Will you force all MP3 encoders to use CBR or will you take the risk of using CBR with Fhg only?
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-24 22:35:16
I would say that the answer should depend on the choice of all other competitors. Will you force all MP3 encoders to use CBR or will you take the risk of using CBR with Fhg only?

What is the risk? Endless polemics?

Isn't it acceptable to pick a popular, widely (freely) available option to conclude about overall performance? What are other variants that are "popular, widely available"? Nero isn't free, MMJB isn't popular, mp3sEncoder isn't popular (and probably in experimental stage), Audition+MP3 filter aren't free and so on...
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-09-24 22:52:49
Quote
Isn't it acceptable to pick a popular, widely (freely) available option to conclude about overall performance?

It is of course. The persitant problem is simply that the choice of Fhg competitor would be ruled by a different criterion than all other competitors (choice based on popularity instead of supposed quality). BTW, which is the most popular mode for LAME: CBR or VBR? and for iTunes?

If we decide that LAME, HELIX, iTUNES' encoders should be tested with the best available setting (which is known or supposed to be VBR) without any consideration for the popularity (real or supposed) then it would be difficult to make a pertinent exception for the sole FHG's competitor.

An answer to the problem would be CBR for all. But the point of such test would clearly be Ubuesque (deciding which encoder sounds the best with the worst encoding mode    ?)
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-25 05:37:19
Edit: I forgot that the MusicMatch JB supports JS VBR (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47313&view=findpost&p=421557) sometimes  .
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-25 05:52:23

OK, so what do we do with FhG guys? We still have to decide which encoder to use and with what settings.

I would say that the answer should depend on the choice of all other competitors. Will you force all MP3 encoders to use CBR or will you take the risk of using CBR with Fhg only?


All other encoders will use VBR (except low anchor).
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-25 08:13:57
....
If we decide that LAME, HELIX, iTUNES' encoders should be tested with the best available setting (which is known or supposed to be VBR) without any consideration for the popularity (real or supposed) then it would be difficult to make a pertinent exception for the sole FHG's competitor.

An answer to the problem would be CBR for all. But the point of such test would clearly be Ubuesque (deciding which encoder sounds the best with the worst encoding mode    ?)

I think the answer to the problem is by thinking in terms of settings and not giving special attention to vbr vs. cbr. With FhG CBR 128 vs. VBR it's also ls-stereo vs. joint stereo and knowbody knows (and it will depend on the samples) whether to give more weight to the vbr vs. cbr thing or to the ls-stereo vs. joint stereo. And even that is only part of the story as long as we don't know all about FhG's vbr and cbr behavior at ~ 130 kbps.

So it's the old thing we also have for Helix vbr settings: without a pre-test we don't know what settings are best. The best we can do is use those settings which are most widely considered best during the discussion.

We should keep our minds on settings - not concentrate on isolated though important features.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-25 16:23:06
The best we can do is use those settings which are most widely considered best during the discussion.


Which would be..?
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-25 17:10:15
Umm... WMP Eleven, 128?
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-25 18:32:25
If I see it correctly that's fastenc cbr 128 from WMP.
As the WMP proposal came from jmartis it would be best if he told us which version he has in mind.
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-09-25 18:56:50
If I see it correctly that's fastenc cbr 128 from WMP.
As the WMP proposal came from jmartis it would be best if he told us which version he has in mind.

I believe both WMP 10 and 11 use FHG Fastenc v3.3.x.44 (as ACM codec)
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-09-26 05:31:26
I believe both WMP 10 and 11 use FHG Fastenc v3.3.x.44 (as ACM codec)

WMP 11 beta installer includes version 3.4.0.0.

Edit. The latest version of l3codecp.acm from WMP11 (beta 2) can be found here (http://www.mdgx.com/wmp.htm#L3CMP11).
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-09-26 08:40:13
I believe both WMP 10 and 11 use FHG Fastenc v3.3.x.44 (as ACM codec)

WMP 11 beta installer includes version 3.4.0.0.

Edit. The latest version of l3codecp.acm from WMP11 (beta 2) can be found here (http://www.mdgx.com/wmp.htm#L3CMP11).

so I stay corrected now
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-27 19:12:12
Guys, I see no point discussing this further because we won't get any clear result. I will stick to 128 kbps CBR using Windows Media Encoder. Now let's discuss samples - do you think only new samples should be used? If not, which old samples do you recommend?
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-09-29 19:41:37
Guys, I see no point discussing this further because we won't get any clear result. I will stick to 128 kbps CBR using Windows Media Encoder. Now let's discuss samples - do you think only new samples should be used? If not, which old samples do you recommend?

What Windows Media Encoder do you mean? I know WME9 but this encodes only to wma (or have I overlooked something?) Guess you mean WMP10 or WMP11?

ADDED:
As for the samples for simplicity (and comparision) can't we just take those from the last 128 kbps listening test or take them as a basis for discussion. And add a series of problem samples?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-09-29 20:59:23
WMP 11.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-04 07:34:19
Come on guys...
Title: Autumn 2006 Listening Test
Post by: dbAmp on 2006-10-04 08:52:19
I almost hate to interject at this point... as you seem to all know a whole lot more about this than I do... but doesn't iTunes use a FhG MP3 encoder?
Title: Autumn 2006 Listening Test
Post by: stephanV on 2006-10-04 11:33:01
@Sebasian:

I don't see any objection using the samples from the last 128 kbps test you did.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-04 13:30:18
Yeah, but it would be wise to use some killer samples too. I would like to know which ones you think should be used and which of the old samples you think should be excluded this time.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-04 17:15:18
Yeah, but it would be wise to use some killer samples too. I would like to know which ones you think should be used and which of the old samples you think should be excluded this time.

I don't see any reason to exclude some samples.


My suggestion.

One hypothesis: using the same samples to feed two similar tests is necessary (but not sufficient) for cross-comparison purpose. So why not using the same 12 samples used in the past during the last MP3@128 listening test? We could use this 12 samples, and add 6 other ones. That would make 18 samples. And as appendix to the final result, the conducer could publish the plots corresponding to the 12 first samples and put in regard the results obtained in 2004. It's maybe a way (good or bad?) to show the progress done in two years and that would be the second teaching of this test (the first one is of course the relative quality of all competitors).

What do you think? Am I clear?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-04 20:11:00
Don't you think 18 samples are too much?
Title: Autumn 2006 Listening Test
Post by: adamjk on 2006-10-04 21:04:33
This thread is like very old opera, almost 2 months discussions and no conclusion, congratulations for author!
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-04 21:10:41
Don't you think 18 samples are too much?

In my opinion the optimal number of samples depends on the overall motivation. Because I really can't evaluate this level I absolutely can't answer to your question. If people are motivated, 18 samples should be fine ; if they aren't, even 12 would be too much.
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-10-04 21:27:53
Considering that it was hard to have enough contributions to conclude the low bitrate AAC test, I think that 18 samples on mid bitrate might be too much.

While we have some people that will be motivated to provide results, unfortunately too many people are interested by the results but think that they don't need to contribute, and are simply waiting for other people to take the test.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-04 21:38:15
With so many samples, I think that I will have a much much higher number of results for samples 1 to 5 compared to 15 to 18.

This thread is like very old opera, almost 2 months discussions and no conclusion, congratulations for author!


Discussion about encoders is over and we now moved to discussing samples. I am sorry that I had to leave to Romania and clarify some things, but "real life" is more important to me. I would really like to accelerate the whole pre-discussion thing so I can take care of the preparations and get the test started.
Title: Autumn 2006 Listening Test
Post by: gameplaya15143 on 2006-10-04 23:42:48
I think all of the potential samples should included in the running.  Then randomly select what samples will be used.  It's the only way to be fair.

Picking and choosing samples carefully is not fair, you would be influencing the results.
Title: Autumn 2006 Listening Test
Post by: haregoo on 2006-10-05 00:47:32
Considering that it was hard to have enough contributions to conclude the low bitrate AAC test, I think that 18 samples on mid bitrate might be too much.

While we have some people that will be motivated to provide results, unfortunately too many people are interested by the results but think that they don't need to contribute, and are simply waiting for other people to take the test.

Low bitrate help tester to discern lossy from reference. But it was pretty difficult to tell each contender in AAC 48kbps test. I'm not surprised if this can result in smaller number of tester.

Last 128kbps multiformat test had more tester. 18 samples is good for the test which people are interested in.
Title: Autumn 2006 Listening Test
Post by: uart on 2006-10-12 09:35:50
Is there any news on when these tests are likely to proceed?
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-12 10:12:18
As for the problem samples I suggest we choose from each of the problem categories (pre-echo, ringing, warbling, ... , as well as distortions that don't match well with one of the preceding categories) two or three samples and choose one sample out of each category by chance as gameplaya suggested.

As for a pre-echo sample I suggest castanets.
As for the not-clearly classifiable distortions I suggest harp40_1 (in case somebody can classify it: put it in that category).
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-12 13:00:44
Is there any news on when these tests are likely to proceed?


Depends on the feedback that I get. I also have holidays in two weeks, so from my point of view, we can start anytime, but I need some hints regarding recommended samples.
Title: Autumn 2006 Listening Test
Post by: level on 2006-10-13 22:11:36
Thanks. "seemed to work" - well, they do if they have the right syntax, but do people agree that they sound well?

Why don't you verify this by yourself?

And do you guys think I should use some cryptic command line arguments or should I stick to the simple -V setting (and -X2 to write a Xing header with TOC)?

How could you decide or taking a decision without previous testing?
the "some cryptic command line arguments" (as you called them) were the
result of a very SERIOUS, DEDICATED, HARD and LONG test which I performed many months ago, but that the people here didn't check by LAZINESS.

Obviously, this is a very irresponsible position deprecate the results from this hard test without testing
nor verification; right Sebastian?

Going to the topic again:
How do you know that [-V60 -X2] is better or not than [-V60 -X2
-HF2 -SBT450 -TX0] for example? Obviously, you cannot know it; much less, recommend it or suggest it for a public test.

At least, for high bitrate test; they IF do a lot better difference there, and more specifically, TX" switch, whose default value in Helix is not the adequate.

BTW, I don't have idea if in low bitrate the "some cryptic command line arguments" do a good result there; and now I don't have the patience nor the motivation for performing ABX tests for obvious reasons, but
if you think to use Helix in this public listening test AT LEAST would have to take more seriously my test as an initial reference; because it was performed according to the rules from HA.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-13 22:19:41
level, I didn't mean to offend anyone. Problem is that I cannot test myself because the result would have no meaning (one person's opinion) especially since I have a pretty bad hearing (left ear is totally deaf). The only solution would be running a pre-test, but then we have the problem that actually all encoders should go through a pre-test to decide the best settings. Such a pre-test requires a lot of work and I have no idea if enough people are willing to spend the time - probably not.

And with cryptic I really mean cryptic since you cannot tell anything by reading "-HF2 -SBT450 -TX0" - it's not like "--vbr-new" or "--preset standard" which is more or less English. Anyways, this isn't supposed to have any meaning since we are not comparing how good the names of the switches sound),
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-13 23:58:04
The problem with HELIX is that no reference/trusted commandline exists. A bit like LAME before the --alt-preset era. You can either rely on default settings (which are maybe well-tuned, or maybe not) or rely on people's own setting. For the latter there's one risk: to find (and therefore use for test) extravagant tunings, inferior to the default one.

You can also think about suggesting a pre-test for HELIX but I already (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47313&view=findpost&p=421121) exposed the risk of being caught in a trap. A pre-test for HELIX? Then one not pre-tests for all other competitors like Fraunhofer or iTunes! And indded there's no reason to put a huge effort on finding the best available setting for one competitor and to refuse it at the same time for all other ones. Just think about Fhg possible VBR benefits...

It's up to you, but I wouldn't take the risk to use a personal tuning of HELIX - unless a developer gives you the green light for this. It's not that I don't trust level's commandline but there's no kind of validation for this.
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-10-14 08:51:19
My choice would be to let encoders developpers provide settings when it's possible.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-14 09:08:18
I know Gabriel, but the Helix developers (or at least the two persons I contacted) have no idea either.
Title: Autumn 2006 Listening Test
Post by: music_man_mpc on 2006-10-16 15:09:10
Sorry to enter the debate so late guys but this has to be said:

You all know that FHg @ 128 CBR is going to get majorly pwned right?  Personally I don't see the point in testing something that has two large disadvantages over the rest of the field.  It's going to come in last, almost for sure.*

I also think its important to remember that trying too hard to aim our tests at "Joe Average" is rather pointless.  "Joe Average" doesn't even rip his own CDs, has never heard of FHg, LAME, et al and doesn't care to.  Even if his his friend "Very Slightly Geeky Joe" makes up the majority of the CD ripping population, they are not the best target audience as, in my experience, they are happy with the results they are already getting and don't know/care if there digital audio isn't as good as the full blown "Audio Geek" that populates HA.org.

I'm not sure if anyone else looked at the freeDB statistics page (http://www.freedb.org/en/statistics__client_requests.15.html) brought up earlier which proves there is a significant percentage of the population using Nero to rip there CDs.  It should also be noted that this group is a far better one to target as they have for some reason decided to break with the convention, and user friendless, found in WMP and iTunes.  This group may actually be interested in our results, the "average" WMP/iTunes ripper will undoubtedly ignore/not even get a chance to see the results of this test.

As for samples I have a suite that I can upload if anyone would be interested in picking through them.

*Yes I might be proved wrong here . . . in fact I would like to be but I highly doubt that CBR@128 encoding is going to hold up very well against a field of VBR@135+.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-16 15:57:34
For the samples, sure, go ahead an post. I am more than happy to see it.

As for the FhG thingy - I already sugested Nero but the majority wanted to use WMP. Problem is that there are so many FhG implementations out there. Nearly every commercial product uses FhG, including WMP, MMJB, Nero, WinOnCD (at least the versions I tried), etc. The problem we had with CBR over VBR is that VBR does not use Joint Stereo coding (or it does, but uses Simple Stereo all the time rather than switching to M/S Stereo sometime).
Title: Autumn 2006 Listening Test
Post by: music_man_mpc on 2006-10-16 17:20:03
eep!  there seems to be only 2.99mb available in the uploads section . . my suite is 43megs in wavpack -hx.

On the FhG topic I thought someone said that Nero used JS for VBR, but no time to test now I have to run out to work.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-10-16 17:29:10
The problem we had with CBR over VBR is that VBR does not use Joint Stereo coding (or it does, but uses Simple Stereo all the time rather than switching to M/S Stereo sometime).

Alex B found (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47313&view=findpost&p=421557) an implementation with MS&SS joint stereo frames in VBR mode.

I'm not sure if anyone else looked at the freeDB statistics page (http://www.freedb.org/en/statistics__client_requests.15.html) brought up earlier which proves there is a significant percentage of the population using Nero to rip there CDs.  It should also be noted that this group is a far better one to target as they have for some reason decided to break with the convention, and user friendless, found in WMP and iTunes.

However, I'm not sure if freedb statistics provide the number of WMP users.
Title: Autumn 2006 Listening Test
Post by: kennedyb4 on 2006-10-16 17:54:19
It took a while to read this thread!

I would like to re-raise the idea of Lame VBR vs ABR being included in the test. I have not seen a direct comparison of the two at 128 and it seems that this is a fundamental question that could be put to rest at last.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-16 18:34:44
It took a while to read this thread!

I would like to re-raise the idea of Lame VBR vs ABR being included in the test. I have not seen a direct comparison of the two at 128 and it seems that this is a fundamental question that could be put to rest at last.


Sorry, this is not possible because it would mean having too many encoders.

I am going to download MMJB and encode some tracks.
Title: Autumn 2006 Listening Test
Post by: level on 2006-10-16 19:18:09
It's up to you, but I wouldn't take the risk to use a personal tuning of HELIX - unless a developer gives you the green light for this.

You seem to have forgotten that the developers of Helix (specifically Real networks) offered the Helix encoder as open source with the intention that the people will experiment and why not, to improve it.
This is very easy to confirm reading very carefully the Helix thread. The Real networks team did not put objection in this, and there were discovered switches in the encoder; and even were created new ones; in other words; the Real team gave GREEN LIGHT from the beginning. In this point, that you are saying is a contradiction.

It's not that I don't trust level's commandline but there's no kind of validation for this.

That it's only and only your personal opinion based on conjectures. Even the moment, you don't have presented high bitrate tests with my 'tuned' setting that support or doesn't support your affirmation.
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-10-16 19:40:20
I say let's stick with Level's command line. Why not? He has tried it out and it is most likely not worse that the standard V settings. No biggie, not the end of the world if it turned out worse in the end. Nobody else has come up with any other suggestion.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-16 20:32:12
You seem to have forgotten that the developers of Helix (specifically Real networks) offered the Helix encoder as open source with the intention that the people will experiment and why not, to improve it.

Same for LAME. And it's not a reason to test the first commandline found on the web. Experience is telling us that several people are able to provide a 'tuned' commandline but most often these personal tunings are usually considered as a bad thing by developers themselves. LAME history is full of misconception and aberration.

Quote
the Real team gave GREEN LIGHT from the beginning. In this point, that you are saying is a contradiction.

Green light to what? To improve or ruin the quality? To refine or to handicap the encoder before a listening test?
The question is not about the righftulness of tuning HELIX encoder - of course it's welcome! - but to use for an '''official''' listening test a commandline which isn't validated by any publication. Your tunings are maybe excellent (and I sincerely hope it) but what we need is a bit more than a simple claim.

Quote

It's not that I don't trust level's commandline but there's no kind of validation for this.

That it's only and only your personal opinion based on conjectures. Even the moment, you don't have presented high bitrate tests with my 'tuned' setting that support or doesn't support your affirmation.

As far as I know you never provided any detail for any bitrate. And this isn't a conjecture: it's simply a fact. Testing an encoder is a very big task (and suggesting alternative commandline a big responsability). I hope that you did with 20 or 30 various samples at least and I would be very glad to see your detailed results very soon. Cheers
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-16 20:47:03
I say let's stick with Level's command line. Why not? He has tried it out and it is most likely not worse that the standard V settings. No biggie, not the end of the world if it turned out worse in the end. Nobody else has come up with any other suggestion.

OK, and we should give credit to the first person giving is personal tuning over the default one just because nobody is making an alternative suggestion? So what about LAME 3.91 -V5 -q0 -X5 --lowpass 18200 --athlower 12 --athaa-sensitivity -2 --allshort --replaygain-accurate? If nobody will bother or take time to test it and to prove me that I'm wrong shouldn't we use it?

Again, and to be clear: I'm not against Level, or Level's suggestion, or any kind of tuning. It's simply that I'm against compensating our ignorance of HELIX's encoder by using without any kind of verification a tuned commandline which maybe:
- work with few samples only
- correspond to Level's own preference
- ruin the quality in several cases.

It's a listening test and we can't act as sorcerer's apprentice.
Title: Autumn 2006 Listening Test
Post by: DigitalDictator on 2006-10-16 21:15:51
Your deep analysis is dead on as always, Guru. I'm just starting to feel that a lot of people are losing stamina with this endless debate about encoders and settings.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-16 21:33:26
If you want know my most intimate and irrational wish on this subject it would be the inclusion of Level's tuning. But if I militate for the opposite it's because:
- we don't know if HELIX default settings are bad
- we don't know if HELIX tuned commandline suggested by Level is better, or worse
- if we test HELIX with a tuned commandline and if HELIX results aren't great, Sebastian have to take the full responsability of this failure but...
- if we test HELIX with default setting and if HELIX results aren't great AND EVEN IF A BETTER COMMANDLINE WAS AVAILABLE then the responsability would be imputable to Helix's developer's only


You can play with fire and set a test with competitors corresponding to some feelings on your own instead of sticking with the default and safer commandline. But unless your feelings were truly excellent you can't publish the final result and let your own fantasy misleadingly appear as the official representative of the tested encoder. So either HELIX will end on top or the whole test could legitimately be refuted.
One question any conducer, or tester, should keep in mind is: how could I defend my choice AFTER the results. So ask you this question. What was the basis for -V60 -X2 -HF2 -SBT450 -TX0? What would you answer in 6 or 12 months to a new member questionning this choice? We used it because someone tested it and was happy with it? It's a bit short, don't you think...
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-16 21:56:59
The special thing with HELIX is that the developers were asked if I understood this correctly, and they couldn't give a hint. This sounds like they don't have a very good opinion on their defaults. It sounded like that already in the HELIX thread.

From the HELIX thread I learnt that level did extensive tests though he didn't report in detail about it. AFAIK that's the only results we've got. The bigger problem to me is that level tested at a considerably higher bitrate, so we don't know whether or not his parameters are good at 130 kbps.

Anyway we're in a pretty bad situation similar to the FhG usage question.
But as we can't debate endlessly (and obviously don't get at a better decision basis) I suggest again just to use what the majority of members here think is more appropriate.
This brings responsibility quite a bit away from Sebastian and takes it to the community.

As for now I see 3 members having given their opinion: level, guruboolez and DigitalDictator.

Any more opinions?
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-16 22:24:21
Level's methodology is nicely explained here (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=35531&view=findpost&p=337766).

He apparently used 2 samples and 2 only to develop his personal commandline. His tuning is based on 2 samples (electric guitar for both). Then he used 25 samples to check the validity on the commandline. The conclusion is:

Quote
For my ears [-V120 -X2 -HF2 -SBT450 -TX0] was in ABX total transparent with all of these 25 songs of different music.

What we know is: this commandline is transparent (but at 204 kbps is it really a surprise?) but what we we don't know is if it's better than the default setting. As a consequence the most important part of the test, what should be tested first and what should interest us in this debate, is totally missing.

Moreover, Level's tunings used for -V120 basis are exactly the same than the one suggested for -V65. Is it a problem? Take a look on other encoders like LAME, Vorbis, MPC... code and check if the "inner" settings are reproduced from one VBR setting to another (like -V2 ot -V5, -quality 6 to --quality 4, ...). The answer is clearly negative. Most 'switchs' are changed, or tuned, according to the preset. How could -HF2 -SBT450 -TX0 be optimal for both 130 kbps (-V65) and 200 kbps (-120)? For me it's suspect.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-17 06:25:46
I just installed MMJB (or Yahoo! Music Jukebox what it's called now) and I am afraid that I cannot use it. The High settings create files with an average bitrate of 96 kbps and a sampling rate of 32 KHz, the Ultra High mode creates files at 192 kbps with a sampling rate of 44.1 KHz. Unfortunately, there is no setting in the middle.
Title: Autumn 2006 Listening Test
Post by: stephanV on 2006-10-17 07:37:20
Perhaps guidelines for choosing encoder settings could be:

1. Use developers recommendations.
2. If none are given, try to find detailed tests from users and see if there is some consensus on what is good.
3. In lack of that, use the latest encoder with default settings.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-10-17 07:52:04
1. Use developers recommendations.
2. If none are given, try to find detailed tests from users and see if there is some consensus on what is good.
3. In lack of that, use the latest encoder with default settings.

Aha, then for FhG we would use mp3 surround encoder from all4mp3.com:
- no recommendations
- no comparison of various available implementations
- the encoder from all4mp3.com uses the latest FhG mp3 encoder libraries (as they claimed in the email response (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47313&view=findpost&p=425692)).
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-17 08:40:45
We can go on like this endlessly.

It's good to learn about critical issues posted here (like the concern about level's setting especially at 130 kbps), and this goes into the opinion of members taking part in this discussion.

But finally we have to choose something, and IMO that should be for any encoder that setting finding the best support here no matter whether or not we're always on solid ground.

We should keep in mind this isn't a scientific test with the ultimate meaning for the rest of the world. If we would want that we should do pre-tests or let various encoder settings take part in the test (but we're through with this I think).

Moreover I think we shouldn't try taking these things so extremely serious. Any listening test is important but can only provide for a limited experience due to the limited number of samples. Please don't understand me wrong: this doesn't degrade these tests but relativates their meaning a bit.

Let's try to get a bit quicker to the real test.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-10-17 09:26:19
Choosing the latest FhG encoder (mp3sEncoder) may actually help the development if we will find regressions, bugs or problem samples. And it is also possible that sooner or later every new application will implement their latest mp3 libraries.

Also, there is growing number of mp3surround-enabled popular software:
- DivX Player 6.3.1
- Steinberg Cubase 4
- Magix Music Maker.

Edit: Nero.com site also contains (http://www.nero.com/deu/EULA.html) "mp3 Surround audio coding technology licensed from Fraunhofer IIS, Agere Systems and Thomson."
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-10-17 12:57:33
I have already said my opinion about FhG. The VBR implementations look like a can of worms. MMJB VBR might be the best option if VBR is preferred. MMJB has at least some popularity and it is available for Joe Average to use. A limited pretest would be good for making sure that WMP CBR 128 does not happen to be clearly better than MMJB VBR. Perhaps Sebastian could select and make available a few (let's say five) MP3 problem samples. The active members in this thread could then post their personal ABX results. Just to make sure.

For Helix I would actually recommend using Real Player 10.5. I just tried it (what an awkward experience) and it seems to have only two 128 kbps MP3 options. CBR and VBR. Once again the 128 kbps VBR bitrates seem to be within the test target (~ about 135 kbps).

One would like to assume that Real Networks has tweaked their flagship program to produce best possible MP3 quality. This may not be practically true, but it would be acceptable and better reasoning for the used Helix setting than I have seen in this thread so far.

If Gogo is still on the list I would like to remind you of my earlier comment:
I have had an impression that Gogo should be used with ABR at a bitrate like ~130 kbps. It is based on LAME 3.88 and VBR wasn't very matured at that time. Even LAME 3.9x ABR has been used at lower bitrates for example by Guruboolez until the most recent versions. I mentioned earlier that I have used command lines like this: "-a -b 192 -m j -q 2". For this test the command line could be like "-a -b 135 -m j" and the quality option 0, 1 or 2. (I have used -q 2 without any testing just because LAME 3.88 -h was mapped with the -q 2 option and it produced a good balance between the quality and speed).
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-17 13:31:53
Alex, I see no point testing an FhG implementation from a software that is several years old. I already tested the current version of MMJB and no setting comes close to 130 kbps. Using MMJB 8 or whatever is not an option.

The Helix suggestion seems reasonable to me.

What do the others think about Gogo?
Title: Autumn 2006 Listening Test
Post by: jmartis on 2006-10-17 14:50:08
Just a thought: the "MP3 surround command line encoder" from FHG supports CBR setting with any value (it switches between adjacent bitrates to acieve the set bitrate) so it could be used to produce 135kbps (or any other bitrate) mp3 streams.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-10-17 14:56:34
Alex, I see no point testing an FhG implementation from a software that is several years old. I already tested the current version of MMJB and no setting comes close to 130 kbps. Using MMJB 8 or whatever is not an option.

I didn't suggest to use MMJB 8.x. I assumed that the current version has not changed in this sense.

Perhaps it would be good to just use the latest WMP and the options MS has decided to make available. After all they should know what is best for their MP3 encoder.

The test idea would be then like what the big guys offer vs. LAME & Gogo at about 130 kbps. The big guys are naturally MS, Apple and Real. (I don't know how big share Real has anymore, but they were one of the first and Real Player is still very well-known.)
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-17 16:21:15
I support Alex B's suggestion for the gogo setting. Sounds reasonable to me.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-17 16:40:03
Just a thought: the "MP3 surround command line encoder" from FHG supports CBR setting with any value (it switches between adjacent bitrates to acieve the set bitrate) so it could be used to produce 135kbps (or any other bitrate) mp3 streams.


Between using some weird encoding mode that is not really better than CBR but not as good as VBR, I prefer to go with "real" CBR from WMP.

Before you guys start to tell me that my decision sucks because I did not test if it's better than CBR - it can't be simply because it alternates between 128 and 160 kbps frames in such a way that the end file has 135 kbps - there is no "intelligent" function there that allocates more bits to complex parts and less bits to simple parts.

Now that I know how to use Helix (by using Real - I hope you were right and there really are only the two options CBR and VBR ), how to use Gogo? I already see two people agreeing that ABR is better and the arguments given are convincing.

Gabriel, were you recommending ABR over VBR in the LAME 3.88 days?
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-10-18 08:42:57

Just a thought: the "MP3 surround command line encoder" from FHG supports CBR setting with any value (it switches between adjacent bitrates to acieve the set bitrate) so it could be used to produce 135kbps (or any other bitrate) mp3 streams.


Between using some weird encoding mode that is not really better than CBR but not as good as VBR, I prefer to go with "real" CBR from WMP.


Note: it's unusual, but not weird. This is a form of freeformat that is described in the standard. FhG encoders are using it since ages in order to produce 20kbps CBR streams. (but it's still quite unusual)

Gabriel, were you recommending ABR over VBR in the LAME 3.88 days?

Fur such bitrates, definitively ABR. Note: it's likely that abr will undersize its target bitrate.
Title: Autumn 2006 Listening Test
Post by: Egor on 2006-10-18 08:53:32
Between using some weird encoding mode that is not really better than CBR but not as good as VBR, I prefer to go with "real" CBR from WMP.

Note: it's unusual, but not weird. This is a form of freeformat that is described in the standard. FhG encoders are using it since ages in order to produce 20kbps CBR streams. (but it's still quite unusual)

And mp3sEncoder can produce solid 128k streams as well.
Title: Autumn 2006 Listening Test
Post by: sTisTi on 2006-10-18 15:42:39
Do you really consider testing GoGo? If it's based on Lame 3.88, wouldn't it be a waste of time, because it was included in earlier listening tests where it already proved to be inferior to Lame and hasn't changed since? If you really want to include another competitor (I wouldn't do it), a Lame ABR setting would be much more useful because it could may be turn up problem samples where ABR is still better than VBR in current Lame and thus provide valuable tuning help for VBR.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-18 16:10:12
I am testing Gogo because it's pretty fast and I want to see how it competes against LAME and other fast encoders like iTunes, Helix and FhG.

Discussion about encoder decision is really closed now. All additional codec requests or questions regarding why codec A was used over B are going to be ignored. Current list of encoders is:

LAME 3.97: -V 5 --vbr-new

iTunes 7.0.1.8: 128 kbps, VBR, highest quality, automatic sampling rate, automatic channel mode, "Stereo (joint)" stereo mode, intelligent, filter frequencies below 10 Hz

RealPlayer 10.5: 128 kbps, VBR

Gogo 3.13: have to check which ABR settings are suitable. Should I manually specify JS coding? What about the quality - should I even use that switch?

FhG: Windows Media Player 11, 128 kbps CBR

Low Anchor: l3enc 1.0, -br 128000. Should I foce JS coding here, too?
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-27 20:00:16
Bleh guys, come on... Holidays are here and I really want to set up the test this week.

For the samples, here is purposal regarding old ones:

DaFunk
gone
mybloodrusts
polonaise
riteofspring
Scars
Waiting

These are the samples that performed pretty bad in the last MP3 test - 7 files. If 12 samples should be tested, this makes room for 5 problem samples. Which ones to use?

As for the encoder setting. I think I will force JS with everything since I am also forcing it with iTunes.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-27 20:36:24
Suggested samples for common problems:

PRE-ECHO: eig.wv (submitted by Mo0zOoH (http://www.hydrogenaudio.org/forums/index.php?showtopic=49601))

MICRO-IMPULSE: E50.wv (http://gurusamples.free.fr/samples/E50_PERIOD_ORCHESTRAL_E_trombone_strings.wv) or Mahler.wv (http://www.rarewares.org/test_samples/Mahler.wv) (both are from mine)

LOW-VOLUME PERFORMANCE: Debussy.wv (http://www.rarewares.org/test_samples/Debussy.wv) (LAME VBR performed poorly in the past with -V5; better performance since 3.97 beta 2 and even better ones with 3.98 alpha)
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-27 22:26:51
eig.wv is a good proposal. I withdraw my castanets proposal in favor of eig.

So my proposal left is harp40_1 which I'd love to see participating.
Title: Autumn 2006 Listening Test
Post by: Firon on 2006-10-27 22:52:28
I agree with eig.wv for pre-echo, it's even worse than castanets for pre-echo.

What is micro impulse? I don't know what I'm listening for.
Title: Autumn 2006 Listening Test
Post by: LoFiYo on 2006-10-28 18:39:39
Quote
Gogo 3.13: have to check which ABR settings are suitable. Should I manually specify JS coding? What about the quality - should I even use that switch?

Here is what I think we should use for Gogo (it is my personal opinion):

At one point, maybe still now to a degree, I was obsessed with using GOGO instead of LAME. For lower bitrates, GOGO's VBR is generally aweful. It's not even that great at the highest setting -v 0 (unless you add "-vb 192 320" or something). But Gogo still has a slight chance to be competitive among the newer encoders in either the CBR or ABR mode. And one thing I would like to point out is that at ~128kbps, -q 2 isn't really ideal. Either -q 6 (default) or -q 4 produces better resulsts around that bitrate from my experience. - q 6 (rather than -q 2) is actually a nice quality setting across the board at any bitrate (except for some killer sample cases). I do not belive that gogo's "-q 2" is automatically equal to LAME 3.88's "-h" and that therefore it is an ideal balanced setting.

The last time Gogo was tested in one of Roberto's tests, I erroneously suggested adding "-q 0" without much testing, and that was a mistake on my part (now I wonder what would have happened if I had not suggested that!). Since then, I have been testing and using Gogo 3.13, and know more about its behaviour, and would like to suggest a quality setting of "6" (default).

So, my suggesting would be:  gogo INPUT.WAV OUTPUT.MP3 (= no switches specified = CBR128kbps)
or:  gogo -b 136 -a INPUT.WAV OUTPUT.MP3

As far as GOGO is concerned, I think CBR is slightly more stable than ABR qualitywise. But then ABR might be more flexible with some killer samples at the cost of overall stability. You could add "-q 6 -m j" to both settings, but since they are the default around this bitrate, they wouldn't make a difference.

Maybe Mr. Guru or someone with a better set of ears than me will come up with a totally different suggestion and overturn all the above, but up to this point, this is what I think regarding gogo.

I hope this helps a bit.
LoFiYo

edit: typos
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-28 18:47:44
Maybe Mr. Guru or someone with a better set of ears than me will come up with a totally different suggestion (...)

I'm sorry but I never played with gogo's options in order to improve the quality.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-30 21:45:02
OK, omitting the quality setting is a good idea since we don't have any large tests that show which setting is better. I will not force JS since it's used by default - same applies to l3enc as far as I can see.

Any other proposals for samples?

BTW: Did you guys notice that WMP 11 final is out? I think this means testing of new WMA codec can be done soon, so let's hurry up with this test.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-30 23:15:33
BTW: Did you guys notice that WMP 11 final is out? I think this means testing of new WMA codec can be done soon, so let's hurry up with this test.


Anyways, I see that the crowd wants a new listening test and since the 48 kbps multiformat test was delayed until the new WMA codec is open for testing, we have two reasonable possibilities: test MP3 at 128 kbps and include fast encoders like Helix, or test various formats at 80 kbps. (...)


I wonder: now that the 48 kbps multiformat listening test is possible, does it still make sense to perform an 'intermediary' one dedicated to mp3?
As far as I can see, people didn't show that much enthousiasm about the latter. I'm maybe wrong but... the competitors are decided but nobody dared to publish a bitrate table; few samples were suggested; you always have to lauch people,etc...
Moreover the choice of settings often lead to debate (forced JS or not, VBR for iTunes, for GOGOG, for Fhg; dedicated commandline for HELIX and now GOGO...). As far as I can remember I never saw such inextricable situation and lethargic attitude in any previous listening test. It's maybe just a feeling of mine and in no case I'm trying to put the blame on someone.

So I wonder if it wouldn't be better to immediately start a debate about a multiformat test and to postpone the current one. Maybe will we find a better occasion to test different MP3 encoders. But the currents audio developments (aoTuVb5, WMA10 or 9.2) are clearly far from GOGO, Helix and Fhg MP3.

What do you think? Or are you personaly interested by a MP3 challenge?
Title: Autumn 2006 Listening Test
Post by: IgorC on 2006-10-31 00:00:52
I don't want to offend anyone but some people lose interest in any comparison if discussion takes a few months from then.  It's important to get clear info about all codecs, setings, conditions but discussions can be endless.  Just my thoughts as they are 
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-10-31 00:41:12
I wonder: now that the 48 kbps multiformat listening test is possible, does it still make sense to perform an 'intermediary' one dedicated to mp3?
As far as I can see, people didn't show that much enthousiasm about the latter. I'm maybe wrong but... the competitors are decided but nobody dared to publish a bitrate table; few samples were suggested; you always have to lauch people,etc...
Moreover the choice of settings often lead to debate (forced JS or not, VBR for iTunes, for GOGOG, for Fhg; dedicated commandline for HELIX and now GOGO...). As far as I can remember I never saw such inextricable situation and lethargic attitude in any previous listening test. It's maybe just a feeling of mine and in any case I'm trying to blame someone.

But I wonder if it wouldn't be better to immediately start a debate about a multiformat test and to postpone the current one. Maybe will we find a better occasion to test different MP3 encoders. But the currents audio developments (aoTuVb5, WMA10 or 9.2) are clearly far from GOGO, Helix and Fhg MP3.

What do you think? Or are you personaly interested by a MP3 challenge?


I kinda lost interest in faster MP3 encoders @ about 128 kbps after this:

As a quick rehearsal I tried this sample

http://www.hydrogenaudio.org/forums/index....showtopic=47370 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47370)

using the following encoders & settings:

LAME 3.97b2, -V5 --vbr-new
WMP10 / FhG, 128 kbps CBR default settings (I used the acmenc command line interface)
Helix (hmp3enc.exe 23.7.2005), -X2 -U2 -V60 -HF2 -F17000
iTunes 6.0.5.20, 128 kbps, VBR, Quality: Highest, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
iTunes 6.0.5.20, 128 kbps, CBR, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
Gogo 3.13, -a -b 133 -m j -q 2

The first few distorded rock guitar chords sent all other encoders except LAME straight to the Hell.

I ABXed them all 8/8, but LAME was difficult and the others were quick and easy.

In general, LAME was quite good, not annoying at all, FhG was barely usable, the others were bad.

I don't know if the encoder versions and settings I used were optimal, but if the actual test samples behave even partially like this I think we may have a good chance to find a clear winner this time.

Personally, after this rehearsal, I would use only LAME at this bitrate. Even the other samples would be transparent with all encoders a standard rock quitar sound is too important for me.


I have used and will probably continue to use GoGo, Helix or WMP FhG (with Nyaochi's excellent ACMENC) for quickly encoding files for MP3 CD-RW discs, which I use with my car and portable MP3 CD players, but then I use bitrates at about 192 kbps or more. (Which encoder of these I select depends on my mood and the star positions, I guess...).

BTW, besides the iTunes bitrates that I already published I have encoded my usual 25 test files with Real VBR 128 and GoGo -a -b 135 -q 2. Real was 130 kbps and GoGo 133 kbps. I have not updated my bitrate table yet, but I uploaded MrQuestionMan reports here: http://www.hydrogenaudio.org/forums/index....st&p=445461 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47370&view=findpost&p=445461)
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-31 08:50:24
I am not sure what to do either to be honest. The lack of interest showed in the discussion can also mean lack of interest when it comes to testing which then means that I won't have enough results. On the other hand, discussion is pretty much over now, finally, and starting another discussion for another test somehow sucks IMO.
Title: Autumn 2006 Listening Test
Post by: guruboolez on 2006-10-31 09:21:14
OK, no problem  (and sorry)
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-31 09:59:37
I will participate in the test.
With this bitrate range I do not expect another result than AlexB does, but I find it interesting to learn about how big the differences are.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-31 10:40:08
I was thinking of another plan: test ~48 kbps Multiformat now and ~128 kbps MP3 in January after the Christmas and New Year period is over. Having discussed everything regarding the MP3 test already, I don't think it will take much to get it started then.

My main concern is that I will not have enough testers now since not many people are really interested.
Title: Autumn 2006 Listening Test
Post by: Gabriel on 2006-10-31 10:54:50
I am interested in both tests, but I think that I should not participate in the samples selection if I have a contender taking part of the test.
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-10-31 11:30:11
Hmm... probably a new discussion about new WMP pro / aoTuV beta / Nero AAC+ etc would take a couple of months more. Perhaps it would be better to test MP3 @ 128 now if it is going to be tested.

Sebastian,

Would you mind gathering the samples suggested so far in one post with download links if available. Perhaps that would bring up ideas about additional samples if needed.

Besides listening to through them I could also compare GoGo default quality vs. -q 2 with these samples since no one has actually tested this. It would be a pity if the better setting would not be used. Because GoGo is not maintained by a software company (or anyone else as it appears) it is a "tweaker's choice" anyway. Major software companies are supposed to use the best encoding parameters by default.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-31 11:35:29
I think sufficiently many problem samples have been suggested taking into account Sebastian Mares' and
guruboolez' samples and harp40_1 which I suggested (hope I didn't forget a proposal).
Title: Autumn 2006 Listening Test
Post by: Alex B on 2006-10-31 11:59:30
OK, I went through this thread. I found these:

guruboolez's list
Suggested samples for common problems:

PRE-ECHO: eig.wv (submitted by Mo0zOoH (http://www.hydrogenaudio.org/forums/index.php?showtopic=49601)) -- Link: http://www.hydrogenaudio.org/forums/index....ost&id=2619 (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=2619)

MICRO-IMPULSE: E50.wv (http://gurusamples.free.fr/samples/E50_PERIOD_ORCHESTRAL_E_trombone_strings.wv) or Mahler.wv (http://www.rarewares.org/test_samples/Mahler.wv) (both are from mine)

LOW-VOLUME PERFORMANCE: Debussy.wv (http://www.rarewares.org/test_samples/Debussy.wv) (LAME VBR performed poorly in the past with -V5; better performance since 3.97 beta 2 and even better ones with 3.98 alpha)
halb27's list
As for the killer samples I suggest to use
  • harp40_1 -- Link: ftp://ftp.tnt.uni-hannover.de/pub/MPEG/au...am/harp40_1.wav (http://ftp://ftp.tnt.uni-hannover.de/pub/MPEG/audio/sqam/harp40_1.wav)
  • trumpet  not suggested anymore?
  • herding_calls  not suggested anymore?
  • castanets  changed to eig.wv
(among others).
It's all natural music with the special effect that without knowing in advance that these are problem samples one would not easily beleive how big the problems are for encoders and settings that are usually transparent on most music (with the exception of castanets).
Sebastian's list
Bleh guys, come on... Holidays are here and I really want to set up the test this week.

For the samples, here is purposal regarding old ones:

DaFunk -- Link:  http://www.rarewares.org/test_samples/DaFunk.wv (http://www.rarewares.org/test_samples/DaFunk.wv)
gone -- Link:  http://www.rarewares.org/test_samples/gone.wv (http://www.rarewares.org/test_samples/gone.wv)
mybloodrusts -- Link:  http://www.rarewares.org/test_samples/mybloodrusts.wv (http://www.rarewares.org/test_samples/mybloodrusts.wv)
polonaise -- Link:  http://www.rarewares.org/test_samples/Polonaise.wv (http://www.rarewares.org/test_samples/Polonaise.wv)
riteofspring -- Link:  http://www.rarewares.org/test_samples/riteofspring.wv (http://www.rarewares.org/test_samples/riteofspring.wv)
Scars -- Link:  http://www.rarewares.org/test_samples/Scars.wv (http://www.rarewares.org/test_samples/Scars.wv)
Waiting -- Link:  http://www.rarewares.org/test_samples/Waiting.wv (http://www.rarewares.org/test_samples/Waiting.wv)

These are the samples that performed pretty bad in the last MP3 test - 7 files. If 12 samples should be tested, this makes room for 5 problem samples. Which ones to use?

As for the encoder setting. I think I will force JS with everything since I am also forcing it with iTunes.
I'll try to find the missing links too.

Edit: added the links
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-31 12:39:40
Thanks for collecting the samples Alex. I was off to buy some bread, water and milk since stores are closed tomorrow.
Title: Autumn 2006 Listening Test
Post by: halb27 on 2006-10-31 12:47:52
...
  • trumpet  not suggested anymore?
  • herding_calls  not suggested anymore?
    ...
No, we had this in another thread. While herding_calls is a problem to many encoders (and trumpet too but to a much smaller extent) these are problems known in advance to be especially problematic to Lame. So it would not be fair to include them.
Title: Autumn 2006 Listening Test
Post by: Sebastian Mares on 2006-10-31 12:56:07
Last Listening Test in 2006, ~48 kbps Multiformat or ~128 kbps MP3 (http://www.hydrogenaudio.org/forums/index.php?showtopic=49761)