Skip to main content

Topic: Public MP3 Listening Test @ 128 kbps - FINISHED (Read 146219 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • level
  • [*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #150
@/mnt

Please; Could you perform the test again with the same samples?

but now with this command line:

-V122 -X2 -HF2 -SBT500 -TX0 -C0

Copy and paste EXACTLY as this is written.

Thanks.
  • Last Edit: 26 November, 2008, 09:45:16 PM by level

  • Steve999
  • [*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #151
Full results are online now. Hopefully I didn't miss anything.


Thanks so much everyone!  Sebastian, I sympathize with you, after putting forth the effort and conducting a very nice test, there are so many arguments!  I had agonized over whether the switch from Lame 3.97 to Lame 3.98 was worth it, and the answer appears to be yes, quite likely, very much so, for some types of samples.  So I will keep using lame 3.98.  Thanks to all of the lame developers for your good work.

The helix results look very interesting.  I am not much of an expert.  I downloaded the hellix encoder from rarewares, but I don't see a guide on how to set the switches.  I like to try new things just for fun.  I'm odd that way.

SO, if someone could help me, if I want to use helix in the low-to-mid 200 kbps range (which is what I use nowadays since memory and hard disk space are so much cheaper), what would be the standard-issue switch settings?  I assume I could use it with EAC (which is what I use)?  Is there a link to a guide for setting the switches?  I can't find the documentation.  For me a decent encoder at this range is fine.  I'm interested in the speed and trying something new.

And most of all, thank you SO MUCH to everyone who participated in setting up and conducting the testing.
  • Last Edit: 26 November, 2008, 10:01:35 PM by Steve999

  • greynol
  • [*][*][*][*][*]
  • Global Moderator
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #152
I had agonized over whether the switch from Lame 3.97 to Lame 3.98 was worth it, and the answer appears to be yes, quite likely, very much so, for some types of samples.

There really isn't much legitimate to argue about.  The problem is that people seem to want to extrapolate the results for a single sample to the entire genre of that sample; and it extends beyond this particular thread.  Without conducting tests on additional samples and showing correlation such claims simply aren't credible.
13 February 2016: The world was blessed with the passing of a truly vile and wretched person.

Your eyes cannot hear.

  • Alexxander
  • [*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #153
Is this basically the end of mp3 by reaching its maximum level of quality being that all codecs are tied?

Define "end of mp3". Judging from this listening test clearly there's room for quality improvement as some encoders perform clearly better on some samples and worse on others.

  • Alex B
  • [*][*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #154
Judging from this listening test clearly there's room for quality improvement as some encoders perform clearly better on some samples and worse on others.

That is one of the reasons I started the sample specific threads. I hope the threads would help the developers (mainly the LAME developers who are active here) to better understand what kind of problems the testers noticed.

I have added new threads for the samples #3 and #4. More is coming.
http://www.hydrogenaudio.org/forums/index.php?showforum=40

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #155
Is this basically the end of mp3 by reaching its maximum level of quality being that all codecs are tied?
You seem to enjoy throwing inflammatory one-liners into this thread.  Nice avatar BTW.

Thanks so much everyone! Sebastian, I sympathize with you, after putting forth the effort and conducting a very nice test, there are so many arguments!
I suspect that Sebastian is very pleased about the debate that his test has created.
I'm on a horse.

  • halb27
  • [*][*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #156

There is a chance that users can get significant encoder differentiation for his individual needs.

A very small chance. ...

You're right, as you can read in my last post.
BTW I guess I can see what our differences are about: I care about other things than you do. For instance it's not a question to me whether encoder X is better than encoder Y, in the first place because of the difficulty in the meaning of 'better'. I think most members here including you are happy when there would have been a good ordering in overall results with well-separated confidence intervals. While in a formal sense this is meaningful (also to me) this is not the most important thing to me.
If for instance the best encoder in this sense would have had modest scores on samples 3, 4, 7, 10 (those samples with very special importance to me - see my last post) this encoder would not be interesting to me.
In the end it comes to that I like only results on those samples I care about (1-8, 10, 13, with special emphasis on 3, 4, 7, 10), and I'd like to see results which are pretty close to 5 (cause only in this cases there is a strong agreement among all the participants that the outcome of the particular encoder on this sample is good). Sure 'pretty close to 5' is a weak statement, but at 128 kbps I can't expect to give a strong formulation in case I want to get some answer.
And I do want an answer cause the question about saving storage space has arrived at me as I plan to use a Meizu M6 SL DAP (guess I'll get it for Xmas) which unfortunatley has only 8 GB. This test came in quite handy, and I was surprised I could only ABX pretty few encoders on pretty few samples (not mentioning the very low anchor). So my very high quality demands I had so far (and which I could easily have with my 40 GB iRiver H140) are inappropriate. So I take these test results as a starting point for my encoder choice, I have to reconsider those problem samples I care about (I still do, but I will drop some of minor practical significance to me and will be content when an encoder gets at the non-obvious issue level status), and once I have found a good candidate based on this (shouldn't be too hard), I will do intensive listening tests with 'normal' music of my favorite kind.
  • Last Edit: 27 November, 2008, 05:16:13 AM by halb27
lame3995n -Q0.5

Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #157
Thanks so much everyone! Sebastian, I sympathize with you, after putting forth the effort and conducting a very nice test, there are so many arguments!
I suspect that Sebastian is very pleased about the debate that his test has created.


Of course. Otherwise, the test would be pretty much pointless and would give me the feeling that nobody cares.

  • /mnt
  • [*][*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #158
@/mnt

Please; Could you perform the test again with the same samples?

but now with this command line:

-V122 -X2 -HF2 -SBT500 -TX0 -C0

Copy and paste EXACTLY as this is written.

Thanks.

Since am busy today, i can only do a few tests.

Anyway, did a few tracks.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6 beta 3
2008/11/27 10:29:05

File A: C:\Temp\Helix Mp3\V122\Metallica - Ride The Lightning\04. Fade To Black.mp3
File B: C:\Rips\Metallica - Ride The Lightning\04. Fade To Black.flac

10:29:05 : Test started.
10:30:15 : 00/01  100.0%
10:30:29 : 01/02  75.0%
10:30:33 : 02/03  50.0%
10:30:39 : 03/04  31.3%
10:30:45 : 04/05  18.8%
10:30:50 : 05/06  10.9%
10:30:55 : 06/07  6.3%
10:31:00 : 07/08  3.5%
10:31:05 : 08/09  2.0%
10:31:10 : 09/10  1.1%
10:31:14 : 10/11  0.6%
10:31:22 : 11/12  0.3%
10:31:49 : 12/13  0.2%
10:31:54 : 13/14  0.1%
10:32:06 : 14/15  0.0%
10:32:08 : Test finished.

 ----------
Total: 14/15 (0.0%)

Warbling is almost gone at 3:55, but i think i spotted a preecho though after the drum snare. But alot better.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6 beta 3
2008/11/27 10:32:57

File A: C:\Rips\Metallica - Ride The Lightning\05. Trapped Under Ice.flac
File B: C:\Temp\Helix Mp3\V122\Metallica - Ride The Lightning\05. Trapped Under Ice.mp3

10:32:57 : Test started.
10:33:17 : 01/01  50.0%
10:33:24 : 02/02  25.0%
10:33:31 : 03/03  12.5%
10:33:50 : 04/04  6.3%
10:34:01 : 05/05  3.1%
10:34:07 : 06/06  1.6%
10:34:22 : 07/07  0.8%
10:34:31 : 08/08  0.4%
10:34:39 : 09/09  0.2%
10:34:47 : 10/10  0.1%
10:34:55 : 11/11  0.0%
10:34:56 : Test finished.

 ----------
Total: 11/11 (0.0%)

Gutiar warbling from the start is still there, but alot better.

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6 beta 3
2008/11/27 10:36:14

File A: C:\Rips\Fear Factory - Digimortal\05. Linchpin.flac
File B: C:\Temp\Helix Mp3\V122\Fear Factory - Digimortal\05. Linchpin.mp3

10:36:14 : Test started.
10:36:30 : 01/01  50.0%
10:36:37 : 02/02  25.0%
10:36:48 : 03/03  12.5%
10:36:58 : 04/04  6.3%
10:37:05 : 05/05  3.1%
10:37:12 : 06/06  1.6%
10:37:20 : 07/07  0.8%
10:37:27 : 08/08  0.4%
10:37:32 : 09/09  0.2%
10:37:40 : 10/10  0.1%
10:37:48 : 11/11  0.0%
10:37:56 : 12/12  0.0%
10:37:57 : Test finished.

 ----------
Total: 12/12 (0.0%)

Hardly any improvement with sample 11, warbling and smearing.
"I never thought I'd see this much candy in one mission!"

  • sld
  • [*][*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #159
Maybe LAME 3.98.2 is far into the realm of diminishing returns at this stage (for 128 kbps), when taking into account the tradeoff in encoding speed.

Does LAME have to trade its quality to reach Helix's speed? Does Helix have to trade its speed for LAME's quality (consistency and less metal artifacting)?

Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #160
Quote
-V122 -X2 -HF2 -SBT500 -TX0 -C0

where did you get that command line from? I've been asking about this before, we should explore the existing switches to come up with the best possible. It's the best we can do, since tuning Helix doesn't seem like an option.
  • Last Edit: 27 November, 2008, 07:44:42 AM by DigitalDictator
//From the barren lands of the Northsmen

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #161
Two things to consider is that this where all problem samples and that people where paying attention as much as they can to hear errors.

I think in normal conditions, people would not be so picky when listening.

Like, when i'm listening to music and working, i don't pay not half the attention to the music.

Same if i'm in a meeting with friends, and we have background music.

I think in "reality" , encoders sound better than on tests like this one.

  • Polar
  • [*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #162
The people interested, here are the samples (don't know how long they will be available since my account expires on December 1st):

http://rapidshare.com/files/167638538/Sample01.zip
http://rapidshare.com/files/167638513/Sample02.zip
http://rapidshare.com/files/167638551/Sample03.zip
http://rapidshare.com/files/167638514/Sample04.zip
http://rapidshare.com/files/167638550/Sample05.zip
http://rapidshare.com/files/167638524/Sample06.zip
http://rapidshare.com/files/167638545/Sample07.zip
http://rapidshare.com/files/167638522/Sample08.zip
http://rapidshare.com/files/167638544/Sample09.zip
http://rapidshare.com/files/167638527/Sample10.zip
http://rapidshare.com/files/167638543/Sample11.zip
http://rapidshare.com/files/167638554/Sample12.zip
http://rapidshare.com/files/167638529/Sample13.zip
http://rapidshare.com/files/167638525/Sample14.zip

Or an all-in-one ZIP from kwanbis:

http://www.megaupload.com/de/?d=13B7NWEP
They'll also still be available at
http://listeningtest.vanquickel.be/Sample01.zip
http://listeningtest.vanquickel.be/Sample02.zip
http://listeningtest.vanquickel.be/Sample03.zip
http://listeningtest.vanquickel.be/Sample04.zip
http://listeningtest.vanquickel.be/Sample05.zip
http://listeningtest.vanquickel.be/Sample06.zip
http://listeningtest.vanquickel.be/Sample07.zip
http://listeningtest.vanquickel.be/Sample08.zip
http://listeningtest.vanquickel.be/Sample09.zip
http://listeningtest.vanquickel.be/Sample10.zip
http://listeningtest.vanquickel.be/Sample11.zip
http://listeningtest.vanquickel.be/Sample12.zip
http://listeningtest.vanquickel.be/Sample13.zip
http://listeningtest.vanquickel.be/Sample14.zip

To throw in my own 2 perhaps simplistic cents, all contenders yielding a 4.5ish tie, I personally see little interest in endless debates about which encoder comes out best from whichever point of view, as all have once again proven to have bumped into the boundaries of the practically testable.  One of the few, maybe the only, conclusion to draw imho, is that except for killer samples, 128k on any modern codec is no longer interesting to test at this scale.

  • Neasden
  • [*][*][*]
  • Banned
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #163
You seem to enjoy throwing inflammatory one-liners into this thread.  Nice avatar BTW.


Did you find my one-liner inflammatory? I don't think it is. I think perhaps it was a naive question that was interpreted as sarcasm (which is quite used by other members, often unremarked) but I really meant in this way: if there were more room for MP3 improvements, because of the known limitations bounded to the format.

  • TechVsLife
  • [*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #164
Does this latest test suggest that the recommended settings for LAME be changed (or any other part of the ha wiki)?  And one specific point: is -v2 still the recommended minimum setting for transparency for most listeners in good listening environments etc., or should that be lower now? 



Thanks for any comments (and thanks to Sebastian Mares for conducting the test).

  • gerwen
  • [*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #165
Two things to consider is that this where all problem samples and that people where paying attention as much as they can to hear errors.

I think in normal conditions, people would not be so picky when listening.

Like, when i'm listening to music and working, i don't pay not half the attention to the music.

Same if i'm in a meeting with friends, and we have background music.

I think in "reality" , encoders sound better than on tests like this one.


True enough, but do you really want your encoder to let you down when you do decide to pay specific attention? 

Imo, pick your encoder and bitrate on very critical listening, and tough samples.  At least then you can be reasonably confident that you won't hear artifacts when you do crank it up.

  • Synthetic Soul
  • [*][*][*][*][*]
  • Global Moderator
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #166
  You seem to enjoy throwing inflammatory one-liners into this thread.  Nice avatar BTW.
Did you find my one-liner inflammatory? I don't think it is. I think perhaps it was a naive question that was interpreted as sarcasm (which is quite used by other members, often unremarked) but I really meant in this way: if there were more room for MP3 improvements, because of the known limitations bounded to the format.
Yes, I do see the suggestion that this is the "end of mp3" as inflammatory.  If you say that is not your intention then so be it.  FYI, I have compiled a few more of your statements from this thread as background for my remark, as way of explanation:

Does that make Helix the new recommended MP3 encoder, or has it to be LAME because it's open source?
Edit: Both are open source.
They are all techincally tied, but Helix outperformed all of them. Also, the encoding speed compared to LAME is absurd faster. Could these two arguments qualify Helix for the new recommended MP3 encoder? (LAME being the second recommended)
* Facts
...
Helix performed a bit better than LAME in this test.
I have nothing to add, like I am not saying Helix is BETTER than LAME, I didn't say that... but the numbers are there, and I am gonna stick with the numbers. You can't tell against the numbers.
Is this  basically the end of mp3 by reaching its maximum level of quality being  that all codecs are tied?
I'll say no more on the subject now.
I'm on a horse.

  • Destroid
  • [*][*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #167
Interesting results. It appears 128kbps really is enough for general use MP3 encoding after all. I actually stick to lossless, but I was satisfied with CBR 128kbps Helix on the old P3-733 system because it's advantageous speed and acceptable quality.

If anything, these results should help rid those ridiculous threads about customized encoding setting (you know, the ones with "-V 0 -m s -b320 -B320" et cetera).
"Something bothering you, Mister Spock?"

  • greynol
  • [*][*][*][*][*]
  • Global Moderator
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #168
Does this latest test suggest that the recommended settings for LAME be changed (or any other part of the ha wiki)?
No.

is -v2 still the recommended minimum setting for transparency for most listeners in good listening environments etc., or should that be lower now?
-V2 is not recommended as the minimum setting for transparency.  If you're talking about the wiki which is intended only to be a general guideline the minimum setting is -V3.  This is not to say that -V4 might also deliver transparency to a large number of people over a large number of tracks, let alone -V5.  Consider that -V5.7 was used in this test for Lame 3.98
13 February 2016: The world was blessed with the passing of a truly vile and wretched person.

Your eyes cannot hear.

  • TechVsLife
  • [*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #169
Thanks. I must have been mistakenly thinking of -v2 as the minimum standard for transparency from some even older recommendation. I'll consider -v3 as a conservative and safe min. standard level for transparency for most (the vast majority?) of listeners under even very good conditions (i.e. quiet, but not abx testing). I understand this is only a very general guideline, and listeners vary, etc.

The reason I mention this test as possibly changing the wiki, is that if the perceived quality of the mp3 encoders at 128kps is now higher than it used to be over a couple of years ago (e.g. getting closer to "5"), then that might shift the recommendations a bit. The graph on the wiki of LAME -v settings against both resulting quality and filesize is very useful; I'm assuming that's still consistent with the latest results.

Scroll down a little from here for the graph:
http://wiki.hydrogenaudio.org/index.php?ti...ate.29_settings


-V2 is not recommended as the minimum setting for transparency. If you're talking about the wiki which is intended only to be a general guideline the minimum setting is -V3. This is not to say that -V4 might also deliver transparency to a large number of people over a large number of tracks, let alone -V5. Consider that -V5.7 was used in this test for Lame 3.98
  • Last Edit: 27 November, 2008, 02:17:29 PM by TechVsLife

  • Neasden
  • [*][*][*]
  • Banned
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #170
Quote
Yes, I do see the suggestion that this is the "end of mp3" as inflammatory.


Sorry if I offended anyone with those one-liners.
Exiting the thread.

  • jimmy69
  • [*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #171
Has anyone else noticed that the itunes encoder has actually performed very well from looking at the results.  I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes, only to find that it performed just as well as LAME 3.97, which was considered to be a very good encoder for a long time.  On top of that Apple have now updated itunes with the fix version of there mp3 encoder.  From these results I think it should be ok to say that people who use itunes and would prefer the ease of use in using the built in mp3 encoder, go for it.

  • halb27
  • [*][*][*][*][*]
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #172
... I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes ...

Yes, I've never liked tendencies like these on HA (I've never seen strong reason for this), and I like the outcome of this test so everybody can pick his favorite also for non-quality related reasons.
lame3995n -Q0.5

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #173
I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes,

Probably based on the last mp3 listening test where it SUCKED.

It is really good that now it does not.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Public MP3 Listening Test @ 128 kbps - FINISHED
Reply #174
I mean all this time members f this forum have strongly recommended against using the mp3 encoder in itunes,

Probably based on the last mp3 listening test where it SUCKED.

It is really good that now it does not.

It is really good that Sebastian have dualcore computer, and Alex B doesn't