Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: MP3 at 128kbps public listening test (Read 53287 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

MP3 at 128kbps public listening test

Hello.

As I already mentioned here some times, I plan to conduce an MP3 @ 128kbps listening test in January.

The reason for the early discussion thread is that I'll travel for the holidays to a place with difficult internet connection. So, I'd rather have everything sorted out in advance because I can then start the test soon after I return.

So, here is what I am planning:

As usual, the test would run for 11 days, starting on a wednesday.

There would be 5 codecs and an anchor:

-Lame - --alt-preset 128
-FhG Audition Legacy Slow - VBR 60~70
-FhG Audition Current - VBR 60~70
-FhG Audioactive - 128kbps high quality
-iTunes - 128kbps
-Xing 128kbps as anchor

[span style='font-size:8pt;line-height:100%']EDIT: Audition Legacy Slow -> Legacy Fast
Radium -> iTunes[/span]

Of course, these are open to debate and critics.

The sample suite would be the same as the 64kbps test. Again, if someone has some constructive criticism on it, speak out loud.

If things go as I plan, the test starts Jan 14th 2004 and ends Jan 25th.


Also, it's worth mentioning my future plans: in february, I plan to conduce another AAC@128kbps test, that would compare:

-Nero AAC
-Compaact
-Faac
-QuickTime
-NCTU-Faac (if they sort out their situation until then. Otherwise, I'll find something else)
-Anchor

And, in march, another multiformat test:

-AAC winner
-MP3 winner
-Vorbis GT3b2 or 1.0.1
-Musepack
-WMA standard
-Anchor

Please comment and give your ideas.

Best regards;

Roberto.

MP3 at 128kbps public listening test

Reply #1
Roberto, I have a request for your 2nd test.  Include Ogg Vorbis in your AAC test.  Ogg Vorbis's direct competition in terms of format is AAC.  In your last test, Vorbis did not perform quite as well as the best AAC at the time.  Since Vorbis has not improved significantly since then (Garf's tunings notwithstanding, since they're not applicable at 128kbps), the results will not be as interesting.

However, seeing Ogg Vorbis compared to other AAC implementations will be more interesting (at least to me).  For example, to the end user, OggEnc is more like FAAC than Quicktime or Nero, and a comparison between the two should be most interesting to observe.

This is just my opinion though, and you're welcome to conduce your tests however you like.

MP3 at 128kbps public listening test

Reply #2
Quote
-Lame - --alt-preset 128
-FhG Audition Legacy Fast - VBR 60~70
-FhG Audition Current - VBR 60~70
-FhG Audioactive - 128kbps high quality
-Radium - 128kbps
-Xing 128kbps as anchor


Which of this is closest to iTunes? I'd sort of like to know how good its MP3 encoder is since a lot of people still use it.

MP3 at 128kbps public listening test

Reply #3
Well, I see the theoretical sense of a 128kbps-mp3-test regarding the upcoming tests.
But 1. can we gather enough participants and 2.  is 128kbps still a valid "border"?

What will we proove with these tests?  Is the 128kbps-range still that important to make three tests?

Let's talk about the targets first.

 

MP3 at 128kbps public listening test

Reply #4
Quote
Roberto, I have a request for your 2nd test.  Include Ogg Vorbis in your AAC test.  Ogg Vorbis's direct competition in terms of format is AAC.  In your last test, Vorbis did not perform quite as well as the best AAC at the time.  Since Vorbis has not improved significantly since then (Garf's tunings notwithstanding, since they're not applicable at 128kbps), the results will not be as interesting.

However, seeing Ogg Vorbis compared to other AAC implementations will be more interesting (at least to me).  For example, to the end user, OggEnc is more like FAAC than Quicktime or Nero, and a comparison between the two should be most interesting to observe.

This is just my opinion though, and you're welcome to conduce your tests however you like.

Well, I would gladly include vorbis. The problem is, I already have 6 codecs there (IMO, the maximum for a 128kbps test), and even if NCTU dropped, there is already another one in line.

Besides, calling it an AAC test would no longer make sense. (Although I'll probably have to consider another format for anchor anyway)

So, I probably won't include Vorbis in this test. But I would gladly help people interested in conducing such AAC vs. Vorbis test.

Quote
Which of this is closest to iTunes? I'd sort of like to know how good its MP3 encoder is since a lot of people still use it.


Veeery good point. I think it's worthing dropping one of the FhG codecs in favour of SoundJam/iTunes.

Any suggestion? Personally, I would drop Radium.

MP3 at 128kbps public listening test

Reply #5
Quote
Well, I see the theoretical sense of a 128kbps-mp3-test regarding the upcoming tests.
But 1. can we gather enough participants

I hope so

Keep in mind that MP3@128 is still what the majority of people consider CD quality. So, it is of interest, even if only to see what encoder is best at this mythical bitrate.

Quote
and 2.  is 128kbps still a valid "border"?


Anything above this is almost untestable due to near-transparency.
And below this, you start getting what people consider low bitrates (96kbps-)

Quote
What will we proove with these tests?


The best encoders at 128kbps.

You can't really extrapolate to higher bitrates, but since anything above that will flood my mailbox with ranked references, 128 will have to do.

Also, regarding AAC, it'll be a good measure to find out how codecs improved since the latest AAC test.

Quote
Is the 128kbps-range still that important to make three tests?


Do you suggest another range?

Quote
Let's talk about the targets first.


OK. My targets are the non-audiophile users (I.E, most people that don't visit to HA, and even some HA visitors) that are happy with MP3 at 128kbps but are interested in broadening their horizons, trying out the new batch of codecs that is being developed.

And my targets are the HA users that use medium bitrates for portable usage or simply because they admittedly don't have golden ears.

Also, one could say my targets are the format developers, that will probably be interested in finding out the flaws people detected, in order to improve their codecs. (Some won't give a damn, like Microsoft. But some probably will - Ahead, Apple, Xiph...)

So there you have my reasoning

Regards;

Roberto.

MP3 at 128kbps public listening test

Reply #6
What about --alt-preset cbr 128? I for one would like to see that one.

MP3 at 128kbps public listening test

Reply #7
Quote
What about --alt-preset cbr 128? I for one would like to see that one.

No problem, but that would replace Lame ABR.

Opinions? Maybe make the test all CBR?

MP3 at 128kbps public listening test

Reply #8
Quote
-Lame - --alt-preset 128
-FhG Audition Legacy Fast - VBR 60~70
-FhG Audition Current - VBR 60~70
-FhG Audioactive - 128kbps high quality
-Radium - 128kbps
-Xing 128kbps as anchor

I believe that Audioactive and Radium are pretty much the same thing.

What's the difference between FhG Audition Legacy Fast and Audition Current?  It's my (wild-assed) guess that both are based on FastEnc (which is what FhG VBR has been based on in the past), and that the faster one should be dropped, if it's anything like the difference between the codecs in Cool Edit Pro 2.

I agree that the iTunes mp3 encoder should be tested if at all possible (I don't think it's FhG based).

There's also the FhG slow codec (high quality codec within fastencc.exe)

ff123

MP3 at 128kbps public listening test

Reply #9
Quote
OK. My targets are the non-audiophile users (I.E, most people that don't visit to HA, and even some HA visitors) that are happy with MP3 at 128kbps but are interested in broadening their horizons, trying out the new batch of codecs that is being developed.

If you are targeting the 'average' non-HA user, then perhaps you should use some of the codecs *they* typically use.  That way you can show them how much better other codecs are.  If you use codecs they don't use, then they would have nothing to relate your results to their experience.

Such as MusicMatch (with several settings for the processing level), RealOne, iTunes, and so on.

Those three (or 5, if you count the three levels in MusicMatch) among the more common codecs that average people use.

(I'm assuming most people don't buy any of the mp3 codecs for WMP from Microsoft....  Although it would be interesting to find out the quality of those!)


And for the record, I do fall into the category of HA visitors who don't have "golden ears" except for the occasional song.  (Better sound card & better headphones would probably help me a lot...[grin])

MP3 at 128kbps public listening test

Reply #10
Quote
I agree that the iTunes mp3 encoder should be tested if at all possible (I don't think it's FhG based).

I think I remembering reading in the LAME mailing list archives that a developer for that mp3 encoder said that it was originally based on the original ISO, but that it was very very VERY heavily modified.  So much so that it is pretty much a new codec.

MP3 at 128kbps public listening test

Reply #11
Quote
I believe that Audioactive and Radium are pretty much the same thing.

Good to know. I'll drop Radium then, since latest Audioactive version is newer.

Quote
What's the difference between FhG Audition Legacy Fast and Audition Current?  It's my (wild-assed) guess that both are based on FastEnc (which is what FhG VBR has been based on in the past), and that the faster one should be dropped, if it's anything like the difference between the codecs in Cool Edit Pro 2.


I was under the impression that FhG Current was the slow codec. In this case, I'll do current and legacy slow. Thanks for the tip. I'll update my first post now.

Quote
I agree that the iTunes mp3 encoder should be tested if at all possible (I don't think it's FhG based).


Yes, I'm quite sure it is SoundJam.

Quote
There's also the FhG slow codec (high quality codec within fastencc.exe)


Huh? So fastencc isn't the fast codec?

Anyway, fastencc is buggy and AFAIK this has been fixed on later releases of that library. Where can I find it? Audition? Musicmatch?

Quote
I think I remembering reading in the LAME mailing list archives that a developer for that mp3 encoder said that it was originally based on the original ISO, but that it was very very VERY heavily modified. So much so that it is pretty much a new codec.


Yeah, I'm pretty confident it's SoundJam. I'll replace Radium with iTunes then \

MP3 at 128kbps public listening test

Reply #12
Someone tell me what is Anchor??? Is it new AAC encoder or, maybe, new format???

MP3 at 128kbps public listening test

Reply #13
Ordinary people in Japan still prefers Gogo (2, 3)...

MP3 at 128kbps public listening test

Reply #14
Quote
Such as MusicMatch (with several settings for the processing level), RealOne, iTunes, and so on.

Musicmatch = Audition, I would guess (both use latest FhG libs)
Real = Xing :-P
iTunes is already in

And I'll mention these associations at the results page.

Quote
(I'm assuming most people don't buy any of the mp3 codecs for WMP from Microsoft....  Although it would be interesting to find out the quality of those!)


Well, it's not like there's a huge amount of codecs in the market. You have Lame, FhG and dist10. iTunes uses a proprietary one, what else do we have?

I would guess these WMP codecs are all based on FhG.

Quote
And for the record, I do fall into the category of HA visitors who don't have "golden ears" except for the occasional song.  (Better sound card & better headphones would probably help me a lot...[grin])


From the results of my 128kbps multiformat test, transparency in the terms of the ITU (creators of teh ABC/HR test) - higher than 4 points - can easily be reached with modern codecs.

The reason of using 4 as the transparency level is that the ITU takes in consideration that their tests are done in a very controlled environment, equipment is top notch and the listeners are trained and extremely attentive - very far from the everyday situation.

Even considering my tests aren't done with the same level of profissionalism as the ITU ones and we cut some slack - say, .5 point more to reach transparency - one can say MPC reaches it and QuickTime gets pretty close.

(More about the ITU transparency border here and at ITU R.BS 1116-1 document)

Regards;

Roberto.

MP3 at 128kbps public listening test

Reply #15
Quote
Someone tell me what is Anchor??? Is it new AAC encoder or, maybe, new format???

Nono, anchor is a compressed stream encoded at low quality, or maybe with a lowpass. It is there to put the results into perspective and to avoid bad codecs getting too low rankings because there was nothing worse.

Quote
Ordinary people in Japan still prefers Gogo (2, 3)...


Well, ordinary people in the west still use Xing (AudioCatalyst, RealJukebox, RealOne...)

So, it's hard to please all markets

MP3 at 128kbps public listening test

Reply #16
Quote
Quote
(I'm assuming most people don't buy any of the mp3 codecs for WMP from Microsoft....  Although it would be interesting to find out the quality of those!)


Well, it's not like there's a huge amount of codecs in the market. You have Lame, FhG and dist10. iTunes uses a proprietary one, what else do we have?

I would guess these WMP codecs are all based on FhG.

I don't know.  I've never heard anybody say what they are based on or how good they are.

I even got curious once and checked some warez sites just to see if I could find one and look through it with a hex editor (I never install warez... Way too much chance of virus or trojan) and I couldn't even find a copy of any of the three official mp3 codecs on the warez sites!

So I've kind of gotten the feeling that they are pretty rare.

I did read a comment here in HA from one user who had bought one and was very disappointed with it.  But that's about all I've heard.

MP3 at 128kbps public listening test

Reply #17
I think LAME will perform better with ABR  @ 128 so you should use --alt-preset 128 to answer, does highest quality FhG out perform highest quality LAME @ 128?


I would like to see Xing with full weapons, Newest encoder, VBR, JS, etc.

MP3 at 128kbps public listening test

Reply #18
Quote
I would like to see Xing with full weapons, Newest encoder, VBR, JS, etc.

No problem.

I'll have to play around with Xing settings some first, though. For some strange reason, I never played with them much


(And yes, "newest weapon" would mean installing RealOne)
The things I do for you guys...

MP3 at 128kbps public listening test

Reply #19
Quote
Quote
What about --alt-preset cbr 128? I for one would like to see that one.

No problem, but that would replace Lame ABR.

Opinions? Maybe make the test all CBR?

[grimace]

I realize that many newbies do use CBR (I used to) but I think you need to draw the line somewhere, and that's a good place.

Maybe do one common encoder at CBR and VBR, just to give newbies an example of how much better the VBR is.

For a newbie friendly test like this, I don't see any point to ABR.  It should be VBR and one CBR for comparison.

MP3 at 128kbps public listening test

Reply #20
Lame VBR doesn't work well at bitrates smaller than ~160 :/

That's why I chose ABR.

MP3 at 128kbps public listening test

Reply #21
Quote
Well, ordinary people in the west still use Xing (AudioCatalyst, RealJukebox, RealOne...)

So, it's hard to please all markets

Right. 

And Xing and Blade, we already know its quality in previous testing, are still popular here, too.

MP3 at 128kbps public listening test

Reply #22
Quote
(And yes, "newest weapon" would mean installing RealOne)
The things I do for you guys...

The full 320kbps mp3 encoder is still linked on the RealOne page:

http://realone.real.com/

it goes to:

http://realone.real.com/r/url/?10442991173750

Which redirects to:

http://forms.real.com/real/jukebox/320k_re...l?src=r1central

I assume the one that comes with the free RealOne is probably limited below 128k, which I why I mention this...

I assume, of course, that the paid version of RealOne (which probably includes a full mp3 encoder) and the free downloadable version are the same encoder, rather than the possibility of the paid version including a decent encoder.

I mention that because it's still the same old RealJukebox link, which makes me wonder if they've updated it since the days of RJB.

MP3 at 128kbps public listening test

Reply #23
Some of my opinions:

I don't think all should be set to CBR. I do however vote for a mixture of typical defaults, like Xing & FhG @ 128 CBR, which would be what the typical non-HA user would get when using MusicMatch, Real or WMP. BTW, I am almost 100% sure that WMP uses FhG. Anyway, these "standard" encodings of a typical non-HA person would be interesting to compare with encodings done with more advanced codecs and settings, like lame -preset 128 and the FhG high quality settings, so that we get a comparision of how much better quality you would get by using these more advanced codecs/settings.

Personally I would really like to se Gogo in the test as well as a fast and free alternative (I have very old computer...) There are already three FhG codecs in the test, and personally I could drop one of them in favour of Gogo. But I guess the majority may very well be more interested in FhG so... And I guess fastenc is pretty fast as well...

To sum up: What I would like to see is a test that shows the newbie or non-HA person why dropping software like Musicmatch, Real or WMP can be a good idea.

MP3 at 128kbps public listening test

Reply #24
Quote
I even got curious once and checked some warez sites just to see if I could find one and look through it with a hex editor (I never install warez... Way too much chance of virus or trojan) and I couldn't even find a copy of any of the three official mp3 codecs on the warez sites!

So I've kind of gotten the feeling that they are pretty rare.

I am not quite sure I understand you correctly, but if you are talking about the fact that you can't find any Lame, FhG or dist10 on any warez sites, the explanation is quite straightforward. Warez sites contain cracked and illegal software, but both Lame and FhG can be obtained legally for free, so why would anyone publish them on a warez site?