Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Why is MPC perceived to be the best? (Read 38001 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Why is MPC perceived to be the best?

Reply #25
Quote
I do not think you are going to find a single exact point for each of these codecs where all samples become transparent. The best you could hope for is a tight range that generally results in transparency. Any of the various problem samples currently at hand show that issues at quality X might or might not be fixed with quality Y.

Correct.  We would discover a different transparency threshold for each format, for each sample, by each tester.  Then build a table of results, showing the average transparency threshold for each format based on the averaged results from all testers on all tested samples.

Why is MPC perceived to be the best?

Reply #26
For narrowing down quality settings use binary search, e.g.:

q5 pass
q4 fail
q4.5 fail
q4.75 pass
q4.625 pass

ad nauseam. At some point it will be narrow enough to warrant a stop.

To trim down the total number of tests, shrink your goal. If you seek to verify that MPC is superior to all codecs, don't try to calculate the "transparency threshold" for each codec, just locate a bitrate that is transparent under MPC and not under the codec in question.

Finally, if you do separate tests by codec, you will open yourself up to placebo. If people strongly believe that a particular setup is transparent, they may relax and not look hard for differences. The solution to this is to provide, each time, a set of flacs without identifying what the codecs are, and reveal that only after the test.

For example, here's a plan:

Round 1:

MPC -q5 and Vorbis, LAME, WMA, AAC, whatever, bitrate-matched to MPC

Suppose ABX reveals that MPC, Vorbis and AAC pass, LAME and WMA fail => these two are dropped from future tests.

Round 2: MPC -q4 and -q4.5; two each of Vorbis and AAC bitrate-matched to the two MPC's

Suppose everything at MPC -q4 bitrate fails, Vorbis fails, MPC -q4.5 and AAC counterpart pass => drop Vorbis.

Round 3: MPC -q4.125, -q4.25, -q4.325 with AAC counterparts.

=> should settle it.

Why is MPC perceived to be the best?

Reply #27
Quote
Quote
Some of these codecs have an 'advertised' setting that is supposed to be transparent, coincidentally averaging around 160-200kbps. Perhaps holding them to this would be appropriate here?
True, but I would not base this test on how the codecs are marketed.  They can call whatever they like "transparent" or "CD quality".  But ABX results don't lie. 

My point was perhaps missed. You will need to start somewhere, preferably close to the threshold of transparency to save time. By starting from a particular codec's 'advertised' transparent setting, you immediately put to test the veracity of this claim. By 'advertised' I mean using official encoder settings, like with Nero's AAC Transparent, as well as common knowledge settings like Vorbis Q6 and MPC Q5. Later tests would merely isolate the general location of this threshold. This would be a fair starting point for all involved. If an 'advertised' transparent setting is used, no claims of bias could be lodged. Transparent is transparent is transparent; if someone can hear the difference, then this claim is false. If this happens to result in some codecs utilizing an overly high bitrate, then so be it. The next test (below) will show if this was truly necessary.
Quote
Quote
A pass or failure would determine an appropriate direction (lower or higher) for a second test to determine operational tolerance.

That's exactly what I had in mind.

This is the merely the second step of the above process, to follow the above initial step. Neither tests are intended to stand solely by themselves, although claims of transparency could be proven or disproven for the settings used in the first step.

As to issues of encoder version variance: establish a cutoff point (such as today) where only the major or stable versions of official encoders that were generally available as of that date will be used. Valid or not, the assumption can be made that later encoder releases should, if nothing else, be no worse than prior versions. Maintain the test as a snapshot of the world of audio compression on this, a normal and unimportant day.

Talk of 3 or more settings used per codec is already making my ears ache and my mind wander. Keeping the test groups to a small, friendly number would open your test up to a larger and more active, audience. This would be good for maintaining relevance. Anything else would merely be a test of those few indvididuals willing to stick it out, and they will almost be guaranteed to break the curve (you know who you are).

Why is MPC perceived to be the best?

Reply #28
Do we ask why god is omnipotent?  He just is....same with MPC. 

Why is MPC perceived to be the best?

Reply #29
Quote
Do we ask why god is omnipotent?  He just is....same with MPC. 

ABX please

Why is MPC perceived to be the best?

Reply #30
The god part or the MPC part?

Why is MPC perceived to be the best?

Reply #31
Quote
Quote
Do we ask why god is omnipotent?  He just is....same with MPC. 

ABX please

....When worlds collide.......

recommended  medicine:
"Phillip K. Dick: Valis"
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
        - Oceania Association of Autonomous Astronauts

Why is MPC perceived to be the best?

Reply #32
Quote
For narrowing down quality settings use binary search, e.g.:

q5 pass
q4 fail
q4.5 fail
q4.75 pass
q4.625 pass

ad nauseam. At some point it will be narrow enough to warrant a stop.

To trim down the total number of tests, shrink your goal. If you seek to verify that MPC is superior to all codecs, don't try to calculate the "transparency threshold" for each codec, just locate a bitrate that is transparent under MPC and not under the codec in question.

Finally, if you do separate tests by codec, you will open yourself up to placebo. If people strongly believe that a particular setup is transparent, they may relax and not look hard for differences. The solution to this is to provide, each time, a set of flacs without identifying what the codecs are, and reveal that only after the test.

For example, here's a plan:

Round 1:

MPC -q5 and Vorbis, LAME, WMA, AAC, whatever, bitrate-matched to MPC

Suppose ABX reveals that MPC, Vorbis and AAC pass, LAME and WMA fail => these two are dropped from future tests.

Round 2: MPC -q4 and -q4.5; two each of Vorbis and AAC bitrate-matched to the two MPC's

Suppose everything at MPC -q4 bitrate fails, Vorbis fails, MPC -q4.5 and AAC counterpart pass => drop Vorbis.

Round 3: MPC -q4.125, -q4.25, -q4.325 with AAC counterparts.

=> should settle it.

Excellent system Doctor.  I bet this would be a much simpler and more efficient method than my brute force technique.  However we need to guess at what --quality values are going to be used ahead of time so we can find out the corresponding command lines for the other codecs.  Once we have some idea what --quality values will *probably* be tested, we should try to find their nominal bitrates.  To do this I propose that several of us encode a whole bunch of random tracks at a specific --quality value and then post the average bitrate we got  (you can calculate with this) along with the number of tracks encoded (try not to include tracks of silence or tracks with long periods of silence).  Then we can get the mean by multiplying those two numbers together and adding all the products, as well as all the track numbers together, then dividing total products by total tracks.

Example:

Person 1:  A = [Number of tracks]; B = [Average bitrate]
Person 2:  C = [Number of tracks]; D = [Average bitrate]
Person 3:  E = [Number of tracks]; F = [Average bitrate]

Total Average = (A×B+C×D+E×F)/(A+C+E)

I already tested 394 tracks at --quality 4.1 and got an average bitrate of 128kb/s (oddly enough).  Don't follow my lead, please as I'm not sure if we will be using --quality 4.1 yet.  I did this before Doctor's post.
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

Why is MPC perceived to be the best?

Reply #33
Quote
Excellent system Doctor.  I bet this would be a much simpler and more efficient method than my brute force technique.

However if we did the test this way we would not be able to get statistical information to the effect of how many people find the sample transparent at --quality x.x.  The following idea would be very difficult to achieve:

Quote
Correct. We would discover a different transparency threshold for each format, for each sample, by each tester. Then build a table of results, showing the average transparency threshold for each format based on the averaged results from all testers on all tested samples.
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

Why is MPC perceived to be the best?

Reply #34
Quote
However if we did the test this way we would not be able to get statistical information to the effect of how many people find the sample transparent at --quality x.x.

That's what I meant by "shrinking the goal" - I strongly suspect people will be dropping out of a 120-sample barrage. ;-)

Also, if another codec raises interest we can later take these results and continue in a different direction.

Regarding matching bitrates, we might simply take the average bitrate of the samples we are testing. However, this may be a stupid idea because these samples - the killer ones - are probably interacting with the codec's bitrate management in peculiar ways.

So, we probably should take a "representative" set of recordings and rely on that.

Why is MPC perceived to be the best?

Reply #35
Quote
Regarding matching bitrates, we might simply take the average bitrate of the samples we are testing. However, this may be a stupid idea because these samples - the killer ones - are probably interacting with the codec's bitrate management in peculiar ways.

Exactly!  Thats why MPC had an average bitrate, over all samples it was 140+kbit/s, on the last (128kbit/s) multiformat listening test.  We need to do rigorous testing ahead of time to make sure we have comparable bitrates for all the codecs to be tested.  I think we should start working toward this right away.  Again, first thing that must be decided is the likely --quality values we will use during the test (I know we can't be sure what we will and won't use but I don't mind testing more than I have too).  For example:

--quality 4 - 5.5 (just to be sure) in increments of 0.1 or
--quality 4 - 5.5 in increments of 0.2
etc . . . 

-Tyler
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

Why is MPC perceived to be the best?

Reply #36
Wouldnt a problem sample set make the most sense? My understanding is that the subband encoding has fewer problem samples. Otherwise there really isnt much point as other codecs, like AAC, also perform very well.

But as I said above MPC has other strengths that AAC doesnt, and you dont need to ABX to hear gapless payback or enjoy the benefits of a better tagging system...

Why is MPC perceived to be the best?

Reply #37
Quote
Wouldnt a problem sample set make the most sense? My understanding is that the subband encoding has fewer problem samples. Otherwise there really isnt much point as other codecs, like AAC, also perform very well.

No, no, no!  The point of this test is to find out where (the approximate --quality value and average bitrate) MPC becomes transparent for most people, on most samples and how that compares to other leading codecs (AAC, Ogg, Mp3, etc.).  So a test suite of all problem samples would completely defeat the purpose. . . . . I don't mean to sound too harsh.
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

Why is MPC perceived to be the best?

Reply #38
Well, the time frame for this test should really be May-June.  That'll give Roberto time to conduct the tests already on his schedule, then we can begin serious discussions about this test some time in April.

Everybody's got really good ideas here.  We'll have plenty of time to talk more about them in the official pre-test thread, when the time's right.

As complex as this test would be, there has to be a way to get it all done.  I've also gotten some really good insights from Roberto.  When the time's right, we can all determine the best approach for this test, and then go from there with sample selection and determining encoder versions and settings.

To sum up.....deviating from how I titled this thread, I don't want the test to be "MPC-centric", but rather a determination of average transparency threshold for each of the formats mentioned, all following the exact same testing methodology.  We shouldn't try going into too much more detail than we have already.  The only thing for sure at this point is that the test would have to be divided into "phases" somehow, to make it manageable for everyone who participates.  I don't want people keeling over from "ABX-overdose" and blaming me for it. 

Thanks to everyone here for your input.  And thanks to Roberto for sharing his vast knowledge on this subject, and offering his assistance in what's going to be a pretty big endeavor.  I've hinted at it previously, but now I'm officially announcing my intention to conduct the test, in the time frame mentioned above.



Edit:  Adjusted the planned time frame for the test.

Why is MPC perceived to be the best?

Reply #39
Quote
Quote
Regarding matching bitrates, we might simply take the average bitrate of the samples we are testing. However, this may be a stupid idea because these samples - the killer ones - are probably interacting with the codec's bitrate management in peculiar ways.

Exactly!  Thats why MPC had an average bitrate, over all samples it was 140+kbit/s, on the last (128kbit/s) multiformat listening test.  We need to do rigorous testing ahead of time to make sure we have comparable bitrates for all the codecs to be tested.  I think we should start working toward this right away.  Again, first thing that must be decided is the likely --quality values we will use during the test (I know we can't be sure what we will and won't use but I don't mind testing more than I have too)...

Not to step on ScorLibran's excellent summation, but I felt that this might be a valid point to mull over for the actual test. Doctor appears to have the same reservations.

Matching the bitrates for the test samples should not be a concern, and the resulting bitrates at transparency are merely a formality. This test is about determining the general threshold (setting) of transparency for all of the involved codecs. Some codecs might yield complete transparency with an average below 200kbps, while others are never fully transparent (it could happen...) even above 300kbps. Or vice-versa.

Going to this point, my understanding of why Musepack was the best was because when it did fail, it generally was not catastrophic. A tendency for low bitrates just happened to be an added plus. As long as the bitrate was below lossless, transparency was of primary importance.

Why is MPC perceived to be the best?

Reply #40
Whoa, I haven't posted here in 6 months! I hope people still take what I have to say seriously.

This thread started out as a simple question: is MPC really the best lossy format for achieving transparency at a reasonable bitrate? Then, the follow-up question: if not, then what format is?

I think answering the first question is more important than answering, for example, "What are the exact quality settings for each encoder so that, on problem samples, 80% of the population represented by the test group will be unable to distinguish between the encoded and original sample?"

A test for transparency must be done differently than a test for quality. For low-bitrate quality tests, we are examining the encoder's ability to intelligently throw out information that is less critical for the reproduction of the musical sample. An encoder that is incapable of producing a transparent encoding can still win a low-bitrate contest, if it degrades more pleasantly than its competitors. For transparency tests, the encoder can't underestimate the audibility of any kind of distortion. Though I've never tuned an audio encoder myself, from what I've read here by codec developers, the tuning needs to be done somewhat differently. Think about the Vorbis discussions that claim Monty is working on improving low bitrates, but not high bitrates. Consider that the tuning done for the alt-presets provided no benefit for lower bitrates in Lame. Hopefully, someone who has done codec tuning will quickly and mercilessly correct me if my assumption is wrong.

Also consider the behavior of a codec when it fails on a problem sample: increasing the bitrate rarely helps. Practically any sample that defeats --alt-preset standard will beat --alt-preset extreme. Garf has expressed his findings that any sample besting MPC --standard will also fail with --xtreme. Problem samples for Vorbis may be non-transparent up through --quality 9. A problem sample can be overcome by telling the encoder to dump buckets of bits on it, but that's beside the point, because the codec has already failed to provide transparency at a quality setting designed to be transparent.

If you want to find out which encoder achieves transparency 90% of the time at the lowest bitrate, you can probably extrapolate from Roberto's 128kbps tests. However, Musepack isn't considered "the best" because it hits the 90% marker at a lower bitrate than its competitors. Musepack gets the crown for being closest to the lossy encoder's ultimate goal of 100% transparency at a reasonable bitrate.

Let me provide a contrived example. Format ABC gets 90% transparency at 120kbps average, XYZ at 150kbps. However, XYZ gets 99% transparency at 180kbps, whereas ABC needs 300kbps. This would be the case if ABC degraded more gracefully than XYZ, but XYZ was tuned to handle extreme test-sample cases more thoroughly than ABC. In this situation, XYZ would be championed as the best encoder when transparency is the goal.

Here's my point: if we're testing for transparency, the exact bitrate or quality setting of the encoder is unimportant, because when an encoder fails, adding 20kbps or 40kbps often won't solve the problem. We can simply pick one setting from each encoder that gives the encoder enough "breathing room" to not run out of bits, and then see which encoder can get over the most problem-sample hurdles. This would mean Musepack "standard", AAC "transparent", and Vorbis "quality 5" (or something along those lines). Perhaps even "xtreme", "extreme", and "quality 6", so that we're really testing the encoder's ability to overcome the worst-case scenario, rather than its ability trim bits as close to the wire as possible.

Now, all of this rambling does nothing to address the very difficult issue of picking samples for a fair multi-codec transparency test...

Why is MPC perceived to be the best?

Reply #41
Quote
Here's my point: if we're testing for transparency, the exact bitrate or quality setting of the encoder is unimportant, because when an encoder fails, adding 20kbps or 40kbps often won't solve the problem.


I don't believe the point of this test is the consideration of "problem" samples because of the exact point you're mentioning: problem samples being problems at all bitrates.

Why is MPC perceived to be the best?

Reply #42
Quote
Quote
Wouldnt a problem sample set make the most sense? My understanding is that the subband encoding has fewer problem samples. Otherwise there really isnt much point as other codecs, like AAC, also perform very well.

No, no, no!  The point of this test is to find out where (the approximate --quality value and average bitrate) MPC becomes transparent for most people, on most samples and how that compares to other leading codecs (AAC, Ogg, Mp3, etc.).  So a test suite of all problem samples would completely defeat the purpose. . . . . I don't mean to sound too harsh.

I disagree. Just because one codec is transparent at a bitrate of 10-20 lower then another doesnt make it a superior codec as the size difference really isnt that significant. However if one codec handles problem samples alot better then it, IMHO is a better codec.

Why is MPC perceived to be the best?

Reply #43
Quote
However if one codec handles problem samples alot better then it, IMHO is a better codec.


Once again, I don't think the point of this test is to find the "best" codec.  The test is for the determination of the bitrate/transparency threshold for today's most popular codecs.  ScorLibran's title of the thread doesn't really support the reason for the test anymore(IMO).

 

Why is MPC perceived to be the best?

Reply #44
Quote
Quote
However if one codec handles problem samples alot better then it, IMHO is a better codec.


Once again, I don't think the point of this test is to find the "best" codec.  The test is for the determination of the bitrate/transparency threshold for today's most popular codecs.  ScorLibran's title of the thread doesn't really support the reason for the test anymore(IMO).

Well, Im not sure what the point of the test is then. The original question was what is the best codec (or why is MPC considered the best)? Somehow that got twisted into a listening test to see at what bitrate codecs become transparent. Most ppl code with a little extra head room anyway so is this even an important question? The big contenders here are MPC and AAC (and vobis I guess). We all know that they can achieve transparency at a realatively reasonable bitrate, so is it meaningful to say that one does it at ~180, one at 150 and one at 200 (random #s)? IMHO a test would do better to assume that the modern codecs do transparency realavily well at a reasonable bit rate, but which one performs the best on problem samples, so that you know your music library has the best fidelity you can get with lossy encoding.

Why is MPC perceived to be the best?

Reply #45
Quote
IMHO a test would do better to assume that the modern codecs do transparency realavily well at a reasonable bit rate, but which one performs the best on problem samples, so that you know your music library has the best fidelity you can get with lossy encoding.

I agree. I'd much rather know which lossy encoder is least likely to fail, given enough headroom to transparently encode normal music. Even if it requires 20-30kbps more on average, I would still use it. If a test could find the exact threshold for transparency on typical audio samples for a typical listener, I would still encode at an arbitrarily higher bitrate to account for somewhat more difficult audio samples, possible sound hardware upgrades, and my steadily-improving ability to hear encoding artifacts. Most people would probably do the same, unless they plan to re-encode often. For this reason, I think a "bitrate for transparency" statistic would not be particularly useful.

Why is MPC perceived to be the best?

Reply #46
Hm, I feel a flamefest waiting to happen. Before emotions get out of hand, another chapter from software engineering.

ScorLibran, by virtue of volunteering to organize and run the test, decides how exactly the test is executed. He is the dictator of the test. We can discuss details, but he is free to accept or reject our ideas.

On the other hand, the test is worthless without several dozen participants. So, he better be a benevolent dictator. If he sets unreachable or dubious goals, people will refuse to participate, making the test less conclusive.

So, let's work out something we can calmly agree on.

Why is MPC perceived to be the best?

Reply #47
Concerning the goal of the test. It is obviously interesting to know where true transparency is for each codec. But going from "is MPC the best" to "what settings are adequate for each codec" is like saying "I need a house, so let me build a castle". It is called feature creep. We have a budget: patience of testers. We run over the budget, we get no results.

I proposed a scheme that can tell whether MPC is transparent at a lower bitrate than everything else in ~18 codec/quality setups. If the testing is performed on 5 samples, that's 90 samples to test, in three phases of 30. I'd like to see a scheme that can find the transparency setting for each codec without using more than a hundred of samples.

Why is MPC perceived to be the best?

Reply #48
I would not consider this thread flammable, let alone engulfed in flames. However, it would seem that some of the more central points for this test have yet to be hammered out to the satisfaction of all posters.

What is the real question being asked here? How can the test be conducted to most efficiently answer this question?

After some thought, I am even now more interested in a test to see what people think of how different codecs handle all know problem samples. Testing each codec at two settings:
  • 'base transparency' setting (the 'standard')
  • 'bitrate not an issue' setting (the 'extreme', taking it up a notch)
Give each encoder a crack at handling all known problem samples for all codecs. The best would have the least number of accurately ABX'ed samples, averaged out over the number of samples that are known to affect that codec. I think this could be fair, as even codecs with greater numbers of problem samples might still handle them well enough for the 'masses' to not notice. Would this be a valid form of handicapping?

I realize that the above is not really an original proposal, though the inner workings might be. This might best follow the test for establishing 'transparent' settings for all involved, though it would be no less relevant as even now individuals are using the 'advertised' transparent settings as such, proven or not.

Why is MPC perceived to be the best?

Reply #49
Before spending hours discussing and thinking about this, try a little pre-test... (I originally typed reality check instead of pi]pre-test[/I]!)

Pick an audio sample. It probably doesn't matter what it is, as long as it's not a known "codec killer", but isn't too simple either.

Decide whether to go for "standard" or "extreme" settings - for this pre-test, try standard I think.

Encode this sample using, say, MusePack standard, --alt-preset standard, ogg -q5, AAC transparent etc.

Take these four (or more) samples, as and people to ABX them against the original.

Look at the results. Consider each negative ABX result as a "5.0" grade, and each positive ABX result as a "4.5" grade. Do a statistical analysis. Are the results significant?

If not, I propose you need to think carefully about your "full" test.


You can set up this pre-test in an hour or so - do the encoding, and post the original and decoded FLACs, or make a .bat file to do the job on the users machine if that's easier.

You may decide to use more samples, but this is a pre-test: don't use too many! I think try one, look at the result, try another, look at the results, maybe try a third, look at all the results, and then decide on what the real test should be.


Just my advice - but you're doing the work - I'm just typing waffle!

Cheers,
David.