Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Quality ranking (MPC, OGG & MP3) (Read 6671 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Quality ranking (MPC, OGG & MP3)

Hi,

First let me say thanks to those who help me out on the EQ issue a few days ago, so I can move on to continue my quest for a better alternative to MP3. I'm trying to find a sweet spot in MPC or OGG that sound great with reasonable bitrates. Ideally, I should judge with my own ears. But my audio equipment is extremely poor, and yet I want my encoded music to last through a couple of equipment upgrades in the future, so I need to tap into the collective wisdom of people here.

Below is a table of bitrate distribution for the test tracks I used.
Code: [Select]
        Aria      chorus    missa     Otello2   sym     average

             brass     flute     Otello1   piano     trio  



MPC s   166  141  149  171  149  158  153  157  145  159  154.8

MPC x   191  178  173  197  172  184  179  180  172  186  181.2

MPC i   253  260  230  253  234  236  235  240  237  253  243.1

MPC x+  265  241  243  278  243  263  254  252  237  260  253.6



OGG 5   153  141  148  159  149  151  154  157  155  155  152.2

OGG 6   181  168  182  191  185  184  189  184  192  189  184.5

OGG 7   213  194  221  223  223  226  231  215  229  223  219.8

OGG 8   231  213  242  239  242  249  253  236  246  242  239.3

OGG 9   296  259  316  312  313  327  330  295  316  308  307.2

OGG 10  357  294  387  379  379  407  406  349  378  367  370.3



MP3 s   186  144  177  189  178  193  191  188  180  177  180.3

MP3 x   239  189  243  247  233  245  239  211  234  228  230.8

MP3 i   320  320  320  320  320  320  320  320  320  320  320

(s for standard, x for extreme/xtreme, i for insane, OGG number for -q value, and MPC x+ for the often recommended "--xtreme --tmn 32 --nmt 16".)

From reading the archive I've learned that in terms of sound quality, in general, MPC > OGG > MP3, when the bitrate is comparable. So the following should be easy to establish:

MPC s > OGG 5
MPC x > OGG 6 > MP3 s
MPC x+ > MPC i > OGG 8 > MP3 x
OGG 10 > OGG 9 > MP3 i

Now, the hard part:

How high should we rank MPC? We've already got a big debate in the "Virtual Listener" thread about MPCi vs. MP3 i, and most people seem to believe MPC i is still better, despite its far lower bitrate. If that's the case, we'll get:

MPC x+ > MPC i > MP3 i

But how about

1. MPC x+ and MPC i vs. OGG 9 or 10? Does MPC still has the edge here, despite the very high bitrates of OGG 9 & 10. (This is largely academic to me, since I'm not going to go for OGG 9 or 10 anyway, given the bitrates.)

2. OGG 7 (220k) v. MP3 x (231k), which is better? I'm leaning toward OGG here. How do you think?

3. OGG 8 (239k) v. MP3 i, and OGG 5 (152k) v. MP3 s (180k) which is better? I guess the bitrate gap is too large to overcome here. Is it?

4. MPC x (181k) v. OGG 7 (220k) or 8 (239k), is MPC that good? How about MPC s (155k) v. OGG 6 (185k).


Tentatively I rank them in the following order

OGG 10 > OGG 9 > MPC x+ > MPC i > MP3 i > OGG 8 > MPC x > OGG 7 > MP3 x > OGG 6 > MPC s > MP3 s > OGG 5

How do you think?
tw101

Quality ranking (MPC, OGG & MP3)

Reply #1 –
I'd rank Ogg 8 better then MP3 i

Quality ranking (MPC, OGG & MP3)

Reply #3 –
I'd say that there is no absolute total ordering of all the different codecs.

By their very nature, MPC and Vorbis will each have types of sound which they perform best/worst with. Even if you want to tie them down to a particular listener, a particular sample, and a particular version of encoder, a straight line ranking (a 'total ordering') may not exist.

It may be interesting to try and find some reference samples, and correllate users' ratings of those samples to the codecs which they prefer. By analysing this data, this could then be used to suggest to new users which of the various codecs their ears are most suited to.

For example, by listening to the various samples on the artifact training page (I'm afraid I've lost the link at the moment), I've determined that I have very poor sensitivity to pre-echo (as evinced by the 'castanets' clip in particular). Given that the only theoretical failing of Vorbis's encoding is a slight tendancy toward pre-echo, I can confidently say that Vorbis will be perceptually very good *for me*. If you are very sensitive to pre-echo, then Vorbis may not be the codec for you. But it's all down to individual choice - you can't make blanket statements, unless the quality differences are huge -- and they rarely are huge even at 128kbps.

Quality ranking (MPC, OGG & MP3)

Reply #4 –
Quote
It may be interesting to try and find some reference samples, and correllate users' ratings of those samples to the codecs which they prefer. By analysing this data, this could then be used to suggest to new users which of the various codecs their ears are most suited to.
For example, by listening to the various samples on the artifact training page (I'm afraid I've lost the link at the moment), I've determined that I have very poor sensitivity to pre-echo (as evinced by the 'castanets' clip in particular). Given that the only theoretical failing of Vorbis's encoding is a slight tendancy toward pre-echo, I can confidently say that Vorbis will be perceptually very good *for me*. If you are very sensitive to pre-echo, then Vorbis may not be the codec for you. But it's all down to individual choice - you can't make blanket statements, unless the quality differences are huge -- and they rarely are huge even at 128kbps.
You hit the nail right on the head there Jon. I tried to explain that to JohnV and ff123 in the VL thread but I'm just not as eloquent as you are.

Quality ranking (MPC, OGG & MP3)

Reply #5 –
I just completed testing the wayitis package that JohnV made available on hydrogenaudio yesterday. Here are my results:

164 (wma8) - 2.8, noise between 1-2s, distorted sound around 8s, piano sounds a little brittle
290 (xing) - 2.2, poorly defined attacks, watery, ringing noises in right channel
310 (aac) - 3.3, some pumping, a little loss of definition, not particularly annoying but something just isn't right
429 (ogg) - 3.8, some minor pumping, nothing horrible
549 (lame) - 3.1, somewhat like 310 but with more pumping, some watery sounds between 16-19s
772 (mpc) - 3.8, similar to 429 but with a little more pumping and somewhat more natural attacks

I didn't know which was which before I rated them, but I know what they are now. None are transparent, all have artifacts, but at least 2 are rather listenable.

Quality ranking (MPC, OGG & MP3)

Reply #6 –
Quote
You hit the nail right on the head there Jon. I tried to explain that to JohnV and ff123 in the VL thread but I'm just not as eloquent as you are.


Perhaps you missed the part where I agreed that preferences vary from listener to listener and that people should choose the codec they prefer?

ff123

Quality ranking (MPC, OGG & MP3)

Reply #7 –
Quote
Originally posted by tw101
OGG 10 > OGG 9 > MPC x+ > MPC i > MP3 i > OGG 8 > MPC x > OGG 7 > MP3 x > OGG 6 > MPC s > MP3 s > OGG 5

How do you think?

Wow, I thought I was particular, but you take the prize.

You can definitely overanalyze this. If encoder A --xyz is deemed "better" than encoder B --abc, does that mean this will hold a few months later as improvements are made to each encoder? Things do change. Ogg is getting better. LAME is much better with the alt-presets. MPC hasn't changed as much because it was already at a very high standard.

Honestly, I think the only two choices are Ogg and MPC. I'm talking about quality here, not compatability with portable devices. Both of these formats are newer and established from the beginning to place quality first or very close to first. I personally use MPC because I prefer sub-band encoders to transform encoders. I think I'm particularly sensitive to pre-echo artifiacts and musical transients, so MPC serves my ears better than Ogg. That's not to say that Ogg is littered with pre-echo problems, but it isn't as good as MPC in this regard.

But as I said before, things change. If Ogg ever gets wavelets and we find out that they really are the silver bullet, then maybe Ogg will take the cake. But as it stands now, for the very best quality MPC is generally regarded the premier choice.

You should probably perform some ABX tests to see where your tolerance for artifacts lies. Some people can live with Ogg -q4 and MPC --radio. Others need more bits. ABX tests aren't the final word however (open to debate) so I prefer to err on the side of caution and go for a setting that is little heavier on the bits than perhaps necessary. I've been weaning myself off of the "--xtreme --nmt 16 --tmn 32" habit since that may be obsessiveness taken too far. I'm currently testing out "--insane --minSMR 0" which should produce files in the 200-220kbps range. I don't want to "cut back" much more because this is within the LAME's aps bitrange range, so I wouldn't consider this MPC setting "insane" at all.

Quality ranking (MPC, OGG & MP3)

Reply #8 –
Ogg has fluttering/roughness in some samples which can be heard quite easily at 64 kbit/s, but is sometimes still there at very high quality settings.  So I don't think pre-echo is the only problem Ogg needs to solve.

ff123

Quality ranking (MPC, OGG & MP3)

Reply #9 –
Quote
Perhaps you missed the part where I agreed that preferences vary from listener to listener and that people should choose the codec they prefer?
No, I didn't miss that. But the next statement you made is more to the point here:
"It's only when they want to make a general statement about quality that group preferences need to be considered as well."
How can a special group of listeners, who are sensitive to specific artifacts, listening to specific problem samples, make a general statement about quality?

Quality ranking (MPC, OGG & MP3)

Reply #10 –
Quote
No, I didn't miss that. But the next statement you made is more to thepoint here:

"It's only when they want to make a general statement about quality that group preferences need to be considered as well."

How can a special group of listeners, who are sensitive to specific artifacts, listening to specific problem samples, make a general statement about quality?


You're speculating again that the group of people who have similar preferences are likely biased towards certain types of artifacting.

If anything, the biggest problem with a group listening test such as the ones I set up is not the listeners, but the samples chosen for the test.  I'd love to run a test with at least a dozen different samples chosen from a wide variety of music styles.  It would be even better if they were selected at random, like EarGuy does.

But even if I could ignore logistics, and organize a huge listening test with lots of samples, then I'd likely run into the problem that most samples are too "easy" to hear problems with, especially for high bitrate tests.  Look how hard it was for people to say that there were audible problems with fossiles.wav and rawhide.wav even at 128.

Regarding general preferences vs. the individual listener: yes, there is always the possibility that there will be a person like the last listener of wayitis.wav, who will have widely divergent preferences from most other people (he should choose xing over mpc his encoder of choice at 128).

But if one wants to make a general statement, for example, "mpc averaging 128 is better than xing at 128," (which I believe to be true, btw) then one had better back up that claim with some data from a bunch of people who statistically agree about the ranking. 

ff123

Quality ranking (MPC, OGG & MP3)

Reply #11 –
Quote
Ogg has fluttering/roughness in some samples which can be heard quite easily at 64 kbit/s, but is sometimes still there at very high quality settings.  So I don't think pre-echo is the only problem Ogg needs to solve.
Not to mention what ogg did to that sine wave JohnV posted. Yikes!

Quality ranking (MPC, OGG & MP3)

Reply #12 –
Quote
Originally posted by ff123
Ogg has fluttering/roughness in some samples which can be heard quite easily at 64 kbit/s, but is sometimes still there at very high quality settings.  So I don't think pre-echo is the only problem Ogg needs to solve.
ff123
Definitely true. In "Ogg Killer track" -thread, I've listed only non-pre-echo Vorbis problem cases.
http://www.hydrogenaudio.org/forums/showth...s=&threadid=726
Juha Laaksonheimo

Quality ranking (MPC, OGG & MP3)

Reply #13 –
Quote
So I don't think pre-echo is the only problem Ogg needs to solve.

I agree, the current encoder isn't perfect (although, given the amount of development man-years spent on it, it's an amazing achievement). However, I was talking about the *theoretical* limitations of the format. If you're very sensitive to pre-echo, then Vorbis may not be the codec for you, due to inherent design features. Luckily, I'm not very sensitive to pre-echo, so I won't need to wait for the wavelets

Quality ranking (MPC, OGG & MP3)

Reply #14 –
Quote
Originally posted by Jon Ingram
If you're very sensitive to pre-echo, then Vorbis may not be the codec for you, due to inherent design features.


Not necessarily. Anyone ever heard preecho with oggenc GT2@350kbps ? (question, not a challenge...)

--
GCP

Quality ranking (MPC, OGG & MP3)

Reply #15 –
Quote
Originally posted by Garf
Not necessarily. Anyone ever heard preecho with oggenc GT2@350kbps ? (question, not a challenge...)

-- 
GCP
Yes.
http://sivut.koti.soon.fi/julaak/castanets_si02.flac
Actually pretty easy, try it. My abx ABX 16/16.

[edit]I screwed up a little, was using rc2tune, not gt2. Now using gt2.. After one practise abx round , I scored 8/9. Ok, gt2-350 is not very easy to abx, but I think it's quite possible.
Juha Laaksonheimo

Quality ranking (MPC, OGG & MP3)

Reply #16 –
Quote
I just completed testing the wayitis package that JohnV made available on hydrogenaudio yesterday. Here are my results: 

164 (wma8) - 2.8, noise between 1-2s, distorted sound around 8s, piano sounds a little brittle 
290 (xing) - 2.2, poorly defined attacks, watery, ringing noises in right channel 
310 (aac) - 3.3, some pumping, a little loss of definition, not particularly annoying but something just isn't right 
429 (ogg) - 3.8, some minor pumping, nothing horrible 
549 (lame) - 3.1, somewhat like 310 but with more pumping, some watery sounds between 16-19s 
772 (mpc) - 3.8, similar to 429 but with a little more pumping and somewhat more natural attacks 

I didn't know which was which before I rated them, but I know what they are now. None are transparent, all have artifacts, but at least 2 are rather listenable.


Your ratings yield a correlation of 0.92 with the overall listening panel, which is very high (meaning that your opinion agreed with the general opinion).  It will be interesting to see what VL's correlation is.

ff123

Quality ranking (MPC, OGG & MP3)

Reply #17 –
Quote
Originally posted by Garf
Not necessarily. Anyone ever heard preecho with oggenc GT2@350kbps ? (question, not a challenge...)


I haven't tried this recently with RC3, but before that was released it was still possible to hear pre-echo with GT2@350kbps on death2.  I believe I actually mentioned this to you before come to think of it, but wasn't able to come to the irc channel for quite awhile after so possible tuning was never performed.

Quality ranking (MPC, OGG & MP3)

Reply #18 –
Quote
Originally posted by Dibrom


I haven't tried this recently with RC3, but before that was released it was still possible to hear pre-echo with GT2@350kbps on death2.  I believe I actually mentioned this to you before come to think of it, but wasn't able to come to the irc channel for quite awhile after so possible tuning was never performed.


Ah, yes, I more or less remember this now. Was it an obvious problem? I'm asking because I know it will still be possible to ABX the gt2 mode on some real hards clips, but it shouldn't have any obvious problems on any.

I think these problems can be virtually eliminated by further tuning i.e., there's no reason why vorbis can't be good with preecho without wavelets.

--
GCP

Quality ranking (MPC, OGG & MP3)

Reply #20 –
Quote
Originally posted by Jon Ingram
I'd say that there is no absolute total ordering of all the different codecs.


I've never meant to make any absolute quality ranking. If I made you think that way, I apologize. I'm just trying to get some sort of general idea about how things stand, according to some very experienced people.

Quote
By their very nature, MPC and Vorbis will each have types of sound which they perform best/worst with.


Pardon my ignorance, but this is new to me. Could you tell me then what types of sound you think MPC and Vorbis perform best/worst with? Or could you please clarify what you mean by that?

Quote
But it's all down to individual choice - you can't make blanket statements, unless the quality differences are huge -- and they rarely are huge even at 128kbps.


While I agree it ultimately comes down to individual choice, IMHO some sort of generalization is inevitable, even valuable, because that's the only way information or knowledge can be effectively propagated.

Come to think of it, we're doing it every day. We say in general, MPC is better than OGG & MP3, given comparable bitrate. Even those presets are trying to convey some sort of generalized idea (like "standard", "extreme", "insane").

I trade classical music with friends (unlike most other genres, many classical recordings can be legally traded, for the copyrights on the works have expired). I'm in charge of finding an alternative to MP3 for us. (Sound quality isn't the biggest issue, gapless recording is. Music like opera often requires continuous playing.) I'll make as many references as possible to online sources for furthur information, but I need to provide generalized suggestions as the starting point of furthur discussion.

With MP3, some of us are satisfied with LAME aps, some use apx, and some insist on api, so I need to let them have an idea about how alternative formats are in terms of quality. Rest assured I'm not going to present my finding as conclusive, absolute, or anything of the sort.
tw101

Quality ranking (MPC, OGG & MP3)

Reply #21 –
Quote
Originally posted by mithrandir

Wow, I thought I was particular, but you take the prize.




Quote
You can definitely overanalyze this. If encoder A --xyz is deemed "better" than encoder B --abc, does that mean this will hold a few months later as improvements are made to each encoder? Things do change. Ogg is getting better. LAME is much better with the alt-presets. MPC hasn't changed as much because it was already at a very high standard.


I know things change over time. But relatively speaking, it seems to be quite stable now. Ogg has hit RC3, and RC4 is said to be concentrated on improving low-bitrate quality, which doesn't concern me.

Quote
Honestly, I think the only two choices are Ogg and MPC. I'm talking about quality here, not compatability with portable devices.


If I didn't make myself clear (I thought I did ), my other post should have. I'm looking for an alternative to MP3, and I've narrow down my target to MPC and OGG. I'm still putting MP3 in the ranking only because I need some sort of reference. After all MP3's quality is something people are more familiar with.

Quote
I personally use MPC because I prefer sub-band encoders to transform encoders. I think I'm particularly sensitive to pre-echo artifiacts and musical transients, so MPC serves my ears better than Ogg.


Could you (or others) provide more information on this? I've seen similar statements several times, but don't really understand. Could you give me some links so I can read more about it. My basic question is: if sub-band encoding is so good, why bother developping transform encoder at all. What's the strength of transform encoding? (Sorry for sound like newbie again. I'm newbie.)

Quote
You should probably perform some ABX tests to see where your tolerance for artifacts lies.


I'm indeed test/train-ing myself with test samples on ff123's website. But it takes a lot of time, which is a luxuary to me. And my equipment is so poor even original CDs sound crappy on my system.  (Several people on another thread have said equipment isn't important in hearing artifact, but I still have doubt. I'll ask there.)

In short, it'll take quite some time before I can bring myself up to a degree I can be more or less confident/competent in judging sound quality from different codecs. Before that, I need some quick (though over-simplified) answer here.
tw101

Quality ranking (MPC, OGG & MP3)

Reply #22 –
Quote
Originally posted by tw101
Could you (or others) provide more information on this? I've seen similar statements several times, but don't really understand. Could you give me some links so I can read more about it. My basic question is: if sub-band encoding is so good, why bother developping transform encoder at all. What's the strength of transform encoding? (Sorry for sound like newbie again. I'm newbie.)


Transform coders are more efficient than subband coders because transform coefficients have lower entropy. Therefore transform coders will easily be better than subband coders at low bitrates. At higher bitrates, the difference in efficiency is smaller, and that is where subbands are better because they don't have the problems with preecho.

The theoretical best way to do things would be a hybrid transform-wavelet codec, but it'll probably be at least a year or two before we start seeing codecs with this capability.

Quality ranking (MPC, OGG & MP3)

Reply #23 –
Hi, Tangent:

Thanks for your explanation. Though I don't understand every word of it (it's gotten too technical for me), I got the general indea.
tw101

Quality ranking (MPC, OGG & MP3)

Reply #24 –
Quote
Originally posted by Garf
Ah, yes, I more or less remember this now. Was it an obvious problem? I'm asking because I know it will still be possible to ABX the gt2 mode on some real hards clips, but it shouldn't have any obvious problems on any.


Hrmm.. I wouldn't really call it obvious probably, no.  It was for the most part on the edge of perception.

Quote
I think these problems can be virtually eliminated by further tuning i.e., there's no reason why vorbis can't be good with preecho without wavelets.


I agree with this, at least at bitrates this high.  I do believe Vorbis can continue to get even better in this regard at lower bitrates too.  When I'm talking about wavelets though, what I have in mind is near perfect pre-echo control at around 160kbps or so.  This, I'm not so sure is easy/possible with the current setup.  I could be wrong though