Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: A Call for Codecs Releases (Read 11020 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

A Call for Codecs Releases

Hi,
I will soon start a listening test sequel to my old listening test. Actually I am sorting files on my HDD to prepare the test, so if the Nero & aotuv dev have something unreleased in their bag, it is the right time to release it. I prefer to ask publicy (I will send pm linked to this topic to the dev) before I start because this kind of test might seems simple ... but it requires between 3 days to a week of work (Finding samples/editing samples/encoding samples/listening to the samples/posting the results/answering remarks...) & it is very boring/tiring, so I don't run them every week ... I don't even run one every year. In fact there is much more works in the background than just the published results.

I intend to do it my way & run the same kind of test as my last one, carefully pre-selected very short edited samples with 8 runs if it's a full success & up to 12 if I start failing, but because of what I have learned from my previous test, I will not use the same target bitrates. It is likely that I will focus on a restricted area centered around the "transparency point", so I will not test 96-128-192-256Kbps, but something like 112-128-160-192.

With modern codecs, unless the encoder is bugged, 192Kbps is already transparent to me, so testing 256Kbps is completely useless IMHO.
Also no codec was fully transparent at 128Kbps for me in my last test, so, as I will not use non-transparent bitrate for my own use, I consider that testing 96Kbps is not really usefull (At last not for me, I know the aotuv & celt dev might be interested in low bitrates for streaming), I think I will use vorbis at 96kbps to identify new killer samples, but I don't think I will publish results for it.

I have some hesitation to test 128-144-176-192Kbps instead of 112-128-160-192Kbps, but even if I suspect 144-176Kbps to be interesting to test, those are unusual bitrates so right now I dunno. Testing 112Kbps would be a compromise between the dev interest & mine, a close-to-96Kbps slot for low bitrate, & then 160Kbps would be the only slot left for the 144-176Kbps zone.

The problem is that on the one hand I already know 112Kbps will not be transparent, so it is not really more usefull for me than 96Kbps & on the other hand I expect the transparency point of Nero AAC (which I use with video) to be close to 160Kbps, so once I will have tested 160Kbps I know I will be interested in 144-176Kbps, hence my small dilemma.

Anyway I think the dev will prefer 112-128-160-192 in order to have an hint of the performance in low bitrate (correct me if I am wrong), & also 128-144-176-192 will be harder (read longer & more tiring) for me than testing 112-128-160-192 as the bitrate jumps between the steps are smaller.

As of now I intend to test:
Nero AAC V1.5.4.0
aoTuV exp-bs1
Celt V0.11.0

I don't intend to test Lame (obsolete), Musepack (dead) & Apple AAC (no CLI), don't even ask. Other than a handfull of ABXers & devs I don't really care about the opinion of others anymore. I only publish my result in order to be sure that I didn't make any obvious misstake.

My goals are
1: see what does CELT really worth ? (specially against aotuv)
Answering the question, is CELT better than vorbis ?
2: seek the transparency point for Nero AAC which I expect to be between 128 & 192Kbps
Answering the question, is 160Kbps fully transparent for me ?

I intend to re-test on the 5 samples that I used last time (The "Rush" sample will be left out as it was an aotuv specific bug) as I am already trained on those & don't have to re-do the search & edit samples pre-listening work on those samples which is a gain of time for me.

I also intend to enlarge my sample corpus but to do so I must do a pre-listening work on a database I collected of around 70 possible killer samples & that would  probably be the longest part of the test. (Much longer than the listening test itself because listening to 30 secs samples in which you don't know what to listen to is WAY much longer than listening to edited 2-3 sec samples in which you know exactly what you search)

... so I have two options:
Start testing right now on my old 5 samples & publish something quickly for CELT developers.
Start selecting new samples in order to give time to Nero & aotuv dev so that they can release a new version & then publish later without caring much for CELT.

(As actually I care more for Nero AAC than CELT, I admit I would prefer to have a new Nero release to rise my interest)

So before I start if both Nero AAC & aotuv dev could tell me how close or how far they are from a new release, it would be nice thks. (I suspect the answer is far for both    but it costs me nothing to ask...)

A Call for Codecs Releases

Reply #1 –
Here is what I have in my bag of samples, this is actually a complete mess & right now I have no clue what I can or cannot ABX in it. Needless to say that testing & sorting this alone will take a lot of time & anyway I only intend to select a handfull in it, maybe doubling the number of samples would be a good start, so 10 samples overall with (5 old/5 new) would be a reasonnable target.

Code: [Select]
# Generated on 2011-02-11 16:17:37 by Directory Lister v0.7.2

------------------------------------------------------------------------------------------------------------------------------------------
c:\07- Samples For Listening Test\03- Potential DCT Killer Samples (Lossless)\ (5)                      120 422 002 2011-02-11 16:16 -----
------------------------------------------------------------------------------------------------------------------------------------------
<01- Possible Killer Samples>                                                                                 <DIR> 2011-02-09 14:09 -----
<Low Bitrate AAC Test Samples>                                                                                <DIR> 2010-07-31 00:49 -----
<Vorbis Samples>                                                                                              <DIR> 2010-07-31 00:49 -----
00- Bibilolo.flac                                                                                         1 846 621 2009-08-10 20:48 -a---
00- Ministry (Artefact+Context) Lossless.flac                                                               337 182 2009-04-02 03:27 -a---
00- Ministry2.flac                                                                                        1 969 385 2009-04-04 04:55 -a---
00- Pierre Henry (Artefact+Context) Lossless.flac                                                           775 311 2009-03-27 04:28 -a---
01- Samples Informations.txt                                                                                    265 2009-04-20 10:44 -a---
------------------------------------------------------------------------------------------------------------------------------------------
c:\07- Samples For Listening Test\03- Potential DCT Killer Samples (Lossless)\01- Possible Killer Sa     98 727 704 2011-02-09 14:09 -----
------------------------------------------------------------------------------------------------------------------------------------------
06 Mandylion Shrt.flac                                                                                    3 014 004 2011-02-09 14:05 -a---
2nd Vent Clip.flac                                                                                        2 803 025 2011-02-09 14:05 -a---
Amnesia48.flac                                                                                              729 502 2011-02-09 14:05 -a---
Aquatisme.flac                                                                                            2 087 442 2011-02-09 14:05 -a---
Berlin Drug.flac                                                                                          1 259 662 2011-02-09 14:05 -a---
Boing Boom Tschak.flac                                                                                    3 077 973 2011-02-09 14:05 -a---
Closer To God Edit.flac                                                                                   1 185 246 2011-02-09 14:05 -a---
Convulsion Edit.flac                                                                                        953 321 2011-02-09 14:05 -a---
Cranberries Zombie Partial2.flac                                                                          2 085 380 2011-02-09 14:05 -a---
Creuza.flac                                                                                               3 098 930 2011-02-09 14:06 -a---
Ecstasy.flac                                                                                              4 264 611 2011-02-09 14:06 -a---
Erase Replace.flac                                                                                        1 285 878 2011-02-09 14:06 -a---
Fatboy 30sec.flac                                                                                         3 115 727 2011-02-09 14:06 -a---
Fear factory - 2001 - Digimortal - 05 - Linchpin.flac                                                     1 808 930 2011-02-09 14:06 -a---
Fighter Beat.flac                                                                                         2 155 307 2011-02-09 14:06 -a---
Girl.flac                                                                                                 1 568 332 2011-02-09 14:06 -a---
Grip Inc. - Nemesis - Descending Darkness.flac                                                            2 116 112 2011-02-09 14:06 -a---
Harp40 1.flac                                                                                             1 317 007 2011-02-09 14:06 -a---
Herbie Hancock.flac                                                                                       1 254 830 2011-02-09 14:06 -a---
Herding Calls.flac                                                                                          758 721 2011-02-09 14:06 -a---
Hexonxonx.flac                                                                                            2 370 742 2011-02-09 14:06 -a---
Human Disease.flac                                                                                        3 461 281 2011-02-09 14:06 -a---
Keys 1644ds.flac                                                                                            128 731 2011-02-09 14:06 -a---
Kool & The Gang - Ladies Night.flac                                                                       2 348 550 2011-02-09 14:06 -a---
Lead-Voice.flac                                                                                           1 464 245 2011-02-09 14:06 -a---
Là Ou Je Suis Née - Camille (Le Sac Des Filles).flac                                                        973 575 2011-02-09 14:06 -a---
Memories Excerpt.flac                                                                                     2 496 261 2011-02-09 14:06 -a---
Mmasq.flac                                                                                                1 125 109 2011-02-09 14:06 -a---
Musique Non Stop.flac                                                                                     2 999 391 2011-02-09 14:06 -a---
Nin Quake.flac                                                                                            1 877 987 2011-02-09 14:06 -a---
Part I - Keith Jarrett (The Köln Concert).flac                                                            1 014 852 2011-02-09 14:06 -a---
Ptp Robocop.flac                                                                                          3 144 768 2011-02-09 14:06 -a---
Rammstein - 2009 - Liebe Ist Für Alle Da - Roter Sand.flac                                                2 564 859 2011-02-09 14:06 -a---
Richard Dawkins - The God Delusion (CD1-Track02).flac                                                     1 158 851 2011-02-09 14:06 -a---
Spill The Blood.flac                                                                                      2 792 201 2011-02-09 14:06 -a---
Sweet Old World - Emmylou Harris (Wrecking Ball).flac                                                     1 088 184 2011-02-09 14:06 -a---
Ta Douleur - Camille (Le Fil).flac                                                                          289 225 2011-02-09 14:06 -a---
Taco - Puttin On The Ritz.flac                                                                            3 672 325 2011-02-09 14:06 -a---
Techno Pop.flac                                                                                           2 938 455 2011-02-09 14:06 -a---
The Missing.flac                                                                                          3 516 516 2011-02-09 14:06 -a---
The Product.flac                                                                                          4 427 578 2011-02-09 14:06 -a---
The Robots 2009 Remaster 0 00 0 30.flac                                                                   1 676 467 2011-02-09 14:06 -a---
The Robots.flac                                                                                           3 126 825 2011-02-09 14:06 -a---
Tiesto - Do You Feel Me.flac                                                                              2 529 190 2011-02-09 14:06 -a---
Tout Le Monde - Carla Bruni (Quelqu'un M'a Dit).flac                                                        571 860 2011-02-09 14:06 -a---
Trumpet Myprince (May Be Lossy TAG II - 6.1. VBR Panic (Trumpet My Prince).flac                           1 634 174 2011-02-09 14:06 -a---
Trumpet.flac                                                                                                995 160 2011-02-09 14:06 -a---
Waiting.flac                                                                                              2 400 402 2011-02-09 14:06 -a---
------------------------------------------------------------------------------------------------------------------------------------------
c:\07- Samples For Listening Test\03- Potential DCT Killer Samples (Lossless)\Low Bitrate AAC Test S      4 175 932 2010-07-31 00:49 -----
------------------------------------------------------------------------------------------------------------------------------------------
Birthday_Dethday__Edit_.flac                                                                              1 131 228 2009-05-06 07:32 -a---
Convulsion__Edit_.flac                                                                                      944 368 2009-05-06 07:33 -a---
Descending_Darkness.flac                                                                                  2 100 336 2009-05-06 08:52 -a---
------------------------------------------------------------------------------------------------------------------------------------------
c:\07- Samples For Listening Test\03- Potential DCT Killer Samples (Lossless)\Vorbis Samples\ (19)       12 589 602 2010-07-31 00:49 -----
------------------------------------------------------------------------------------------------------------------------------------------
01- applaud.flac                                                                                          1 115 520 2009-04-19 17:59 -a---
02- spahm.flac                                                                                            2 497 003 2009-04-19 17:59 -a---
03- testsignal3.flac                                                                                         24 312 2009-04-20 06:33 -a---
04- Mahler.flac                                                                                           2 594 536 2009-03-15 00:19 -a---
05- amnesia.flac                                                                                            655 145 2009-04-20 12:50 -a---

A Call for Codecs Releases

Reply #2 –
I think I will left out the Kraftwerk sample because like the Rush sample it is too vorbis specific, unlike the Rush sample the Kraftwerk sample also affects other codecs but it only affects them at 96Kbps, so if it is a killer sample for vorbis, it is not a killer sample for Nero AAC. Unlike my previous test which was targetting Vorbis, I plan to target Nero AAC this time, due to x264 my personnal interest has switched from vorbis to aac, between my previous test & the one I am preparing.

Also for Nero, unlike my previous test, I think that I will use non-"rounded" (I mean 0.05 step) quality setting in order to respect the exponential increase:
192-160=32Kbps vs. 190-160=30Kbps
128-112=16Kbps vs. 130-115=15Kbps

So for Nero I think I will use Q0.41 instead of Q0.40 & Q0.53 instead of Q.55:
Q0.38 (~115Kbps) / Q0.41 (~130Kbps)/ Q0.47 (~160Kbps) / Q0.53 (~190Kbps)

I have build a blank table for the results, now I will test the CELT encoder to see how it works (today), then I will search for 6 new samples (tomorow & maybe for several days) that I can ABX at vorbis 128Kbps minimum, for this I will ABX at vorbis 96Kbps, which should be easy, then I will rise the bitrate to vorbis 128Kbps & see if I still hear something & finally see if Nero AAC 128Kbps is affected which would mean for me that the sample is really a bad ass.

So far I plan to include:
Abfahrt Hinwil
Autechre (Almost 100% sure)
Castanets
Harlem (One or maybe two applauds-like samples is 100% sure)

plus 6 new samples that remains to be selected/edited. I am only at the very beginning & I already spend an hour chatting with myself & sorting folders/files. I need courage ...

A Call for Codecs Releases

Reply #3 –
Sauvage,
It will be interesting to see what are your findings.

Quote
seek the transparency point for Nero AAC which I expect to be between 128 & 192Kbps

Some samples from your list were already reported to not be transparent at 192 kbps. But I don't intervene  because it is your personal test and want to see it as independent as possible.

just my 0.02$

A Call for Codecs Releases

Reply #4 –
Quote from: sauvage78 on
I don't intend to test (...) Apple AAC (no CLI)
There's both qtaacenc and iTunesEncode; find the latter on RareWares.
By letting out Apple's AAC codec, I hope you're not ignoring findings from older personal listening tests published here at HA, IgorC's amongst others, which showed performance on par with, if not better than Nero, at least in your bitrate range of interest.

A Call for Codecs Releases

Reply #5 –
Why aoTuV exp-bs1 unstead of aoTuV [20091012]?

A Call for Codecs Releases

Reply #6 –
I just got pm answers to my call from Aoyumi & menno, nero will not have a new version anytime soon but Aoyumi will probably have a new version ready within 2 weeks (He hopes so, don't blame him if he is late). So I will postpone the test itself untill there is a new aotuv version. Anyway this is not much a problem for me as I was planning to spend around a week testing samples in order to fill my 6 empty slots, so in fact it only postpones the test itself by something like a week. It is more a problem for jmvalin in case he waits for results to freeze CELT bitstream, but it's not really my problem.

menno suggested that I could test 2 pass ABR/CBR if I am interested by audio for video, but it's a long time I don't target a final size to burn my rip on a media (CD/DVD/BR), I use --crf with x264, so I will only test VBR.

IgorC: I already know that I am not the "best" ABXer around, it happened several times that I couldn't ABX a sample posted by /mnt. Also I think I cannot ABX above 18Khz. That's just doesn't matter much to me, I don't run the test to prove anything to others. I run the test for myself, if others are fools enough to think that I would deliver "holy words" without testing themselves, it is their own problem. I consider that my tests are only an information among others. What is important for me (other than the fact that I made no misstake) is that others can re-run the same test if they want to know how valid is my test compared their own earing. I think a bullheaded average listener posting a methodic full test is more usefull in the end than a lazy golden eared ABXer posting a few scattered ABX logs (specially if he only ABX already known samples). I don't care if I have gold ears (I don't), I only care that I don't have an earing below average.
Also not being transparent at 192Kbps doesn't mean that it sounds awfull, sometimes I can even successfully ABX without even knowing what I am listening to. Usually it happens when I am very near to the transparency point. In fact I think I should stop speaking of transparency point in general, & start speaking of MY personnal transparency point. So far at 192Kbps if was able to ABX, it was either a very subtile difference or an obvious encoder bug which was affecting all bitrates no matter how high, that's why I, more or less, consider 192Kbps transparent, specially on Nero AAC & will not test higher.

Polar: I haven't tested Itunes AAC recently, I haven't even tested it at all since my last big test which was more than a year ago, but last time I tested it, the kind of software you posted were requiring that you would have Itunes installed so that it could access the dll (or that you had at last a copy of the dll in the right folder). This is a problem for me, I don't want to have anything to do with fat graphical software commercially oriented toward newbies, I don't want to install it at all, even temporary. I don't want to waste HDD space & put crap key in my registry just for Itunes AAC. Maybe nowadays Itunes AAC is better then Nero AAC, I have no clue. Even if it was the case, without a CLI usable in F2K & Eac3to, I think the quality gain would not be worth the usability loss. Even if, in all honesty, I personnaly really dislike their products (Macs , Ipods ...), I have nothing against Apples itself, I think the same about Nero GUI than I think about Itunes GUI, still I use Nero AAC CLI. It's not just about a CLI, it's about a CLI without dependency. I may be wrong here so correct me if things have changed.

Edit:
Also, 10 samples x 3 codecs x 4 bitrates=120 ABX, adding a 4th codecs would make the test too long IMHO, I am not an ABXing machine. Adding Itunes would mean lowering the number of tested samples. I already had hesitation between 8 or 10 samples to start with, I only set 10 in order that if ever I get bored I can do 8, which would still have some meanings. 6 like my old test is a little weak IMHO, specially when 2 of those were vorbis specific. Actually 10 samples is just an arbitary number, it will depends on my pre-test listening round, all I know is that I will not go below 8 & not above 10. But even with 8 samples it doesn't leave any room for Itunes AAC, sorry. More than an exact number of samples, I actually plan to run around 100 ABxing, which is also an arbitrary number. It is important for me to keep the test reasonnable, because I know by experience that I can lose interest if it is too long.

lvqcl: I will use the next aotuv version now, so it's solved.

I will start ABXing candidates for the test this evening.

A Call for Codecs Releases

Reply #7 –
Quote from: sauvage78 on
I don't intend to test Lame (obsolete)

Really? LAME is obsolete? I thought it is one of the most worked, tuned, codecs available. LAME is sooo good, that it can keep up with any of the "modern" codecs, meaning non MP3 ones.

A Call for Codecs Releases

Reply #8 –
Quote from: sauvage78 on
the kind of software you posted were requiring that you would have Itunes installed so that it could access the dll (or that you had at last a copy of the dll in the right folder). This is a problem for me, I don't want to have anything to do with fat graphical software commercially oriented toward newbies, I don't want to install it at all, even temporary.


installing quicktime should be enough; no need to install itunes.  (However, i haven't tested quicktime only, since I do also have itunes installed, though I never use it.)
http://www.apple.com/quicktime/download/

(quicktime does install a media player etc., so might be too big for you.)

A Call for Codecs Releases

Reply #9 –
Quote from: sauvage78 on
I just got pm answers to my call from Aoyumi & menno, nero will not have a new version anytime soon but Aoyumi will probably have a new version ready within 2 weeks (He hopes so, don't blame him if he is late). So I will postpone the test itself untill there is a new aotuv version. Anyway this is not much a problem for me as I was planning to spend around a week testing samples in order to fill my 6 empty slots, so in fact it only postpones the test itself by something like a week. It is more a problem for jmvalin in case he waits for results to freeze CELT bitstream, but it's not really my problem.


On the CELT side, we haven't changed anything since 0.11 that would be visible to normal users. Also, while we're certainly not going to wait for your test results, if they show a problem with CELT *and* if they are available early enough, then it's still possible to make a last minute change to the bit-stream (assuming it's not something that can be fixed in the encoder). But two weeks probably puts us beyond that point, which means your results will most likely be useless from the PoV of making CELT better for your use cases.

A Call for Codecs Releases

Reply #10 –
Quote from: kwanbis on
Quote from: sauvage78 on
I don't intend to test Lame (obsolete)

Really? LAME is obsolete? I thought it is one of the most worked, tuned, codecs available. LAME is sooo good, that it can keep up with any of the "modern" codecs, meaning non MP3 ones.

Really? Maybe for "average" audio content, but for a lot of codec-critical recordings, my blind tests tell me that AAC and the upcoming unified speech and audio coding are better at basically any bit rate below 160 kb or so (regardless of AAC encoder).

Chris
If I don't reply to your reply, it means I agree with you.

A Call for Codecs Releases

Reply #11 –
As an user of a "dead" format I feel amazingly comfortable
But I agree, there is no need to test over and over again something that was already proven.

A Call for Codecs Releases

Reply #12 –
Quote from: C.R.Helmrich on
Really? Maybe for "average" audio content, but for a lot of codec-critical recordings, my blind tests tell me that AAC and the upcoming unified speech and audio coding are better at basically any bit rate below 160 kb or so (regardless of AAC encoder).

Really? It is obvious that any encoder can have problem with specific samples, ANY encoder. That said, care to show your ABX and explain what "non average audio" you listen to that shows AAC better at 160 than lame?

A Call for Codecs Releases

Reply #13 –
Quote from: kwanbis on
Quote from: C.R.Helmrich on
Really? Maybe for "average" audio content, but for a lot of codec-critical recordings, my blind tests tell me that AAC and the upcoming unified speech and audio coding are better at basically any bit rate below 160 kb or so (regardless of AAC encoder).

Really? It is obvious that any encoder can have problem with specific samples, ANY encoder. That said, care to show your ABX and explain what "non average audio" you listen to that shows AAC better at 160 than lame?

Really?
Do you guys have to bicker about audio formats in someone else's ABX thread?

Apparently LAME is obsolete to sauvage, but that doesn't mean LAME itself is unusable.
Personal opinions are personal opinions; don't get too riled up about it.

A Call for Codecs Releases

Reply #14 –
Quote from: kwanbis on
Quote from: C.R.Helmrich on
Really? Maybe for "average" audio content, but for a lot of codec-critical recordings, my blind tests tell me that AAC and the upcoming unified speech and audio coding are better at basically any bit rate below 160 kb or so (regardless of AAC encoder).

Really? It is obvious that any encoder can have problem with specific samples, ANY encoder. That said, care to show your ABX and explain what "non average audio" you listen to that shows AAC better at 160 than lame?


It's actually an interesting question.  As a codec developer I immediately scoffed at kwanbis' assertion that LAME keeps up.  It's quite true that LAME is amazingly well tuned, but the MP3 format is simply handicapped in many significant ways that more modern codecs are not.  And its performance at some bitrates certainly shows those handicaps.

But those warts are only super-apparent at low bitrates. The higher bitrates are harder to talk about, because listening tests in them are difficult, and at least around here that causes people to focus on small collections of killer samples.    I think this make be a mistake in the common HA methodology, but perhaps an uncorrectable one.  The issue I see with most of the killer samples commonly used is that they are transient-heavy transform-codec-killers.  For VBR codecs, what you're really doing when you're testing these samples is testing the codecs' block switching and rate control.

This is fine and important and essential for the encoder to get right, but it's not necessarily representative of anything else.  One outcome that we should _expect_ from this methodology is that it will conclude that Vorbis and AAC are much closer to MP3 in performance than they actually are— the reason for this is because these codecs get a lot of their improvement over MP3 by using longer transforms, which must be avoided on these samples.  Yet improvement that Vorbis and AAC have is real and should be applicable across bitrates (as it's a true increase in coding efficiency for many samples), but it's not applicable in the case of the transient heavy samples, unlike some of the fully parametric coding improvements in things like HE-AACv2 which really are low rate only things. (On the other hand, I would also expect this testing to overstate the improvement of CELT vs Vorbis— because the CELT bitstream is _much_ better for transients without compromising its overall performance)

It might be that the theoretical advantages of the Vorbis, AAC, and (especially) CELT over LAME MP3 are not yet actual advantages at high rates because the encoders are not well enough tuned, but since these formats are arguably functional supersets of MP3 with many technical improvements,  it seems inevitable to me that they will show benefits given equivalent tuning effort for high rates.

Given the difficulty of testing at these rates, I don't have listening test results for you, but I do have some objective data.    The PEAQ quality assessment methodology gives pretty wonky results at supra-transparent rates, since that is largely outside of its tuning, but has a pretty decent high resolution masking model in it.  We can use that to get a dumb idea of what other formats are possibly giving us over MP3.

Here is a run on some stereo audio (the 'comp' collection that we often use in the public CELT demos), with the latest VBR lame (the highest point on lame is CBR 320, since the VBR didn't quite make it up that high), CELT, and aoTuV B5.7. I created this graph as part of the high rate testing I'm doing with CELT because I'm very concerned about the risk of the high rate performance not being good enough simply because I can't effectively perform listening tests at these rates. Each line is about 500 measurements. (this is the amplitude of the noise relative to the estimated masking; more negative is better. Due to leakage in the filterbanks it bottoms out at around -35dB or so)


Now I don't mean to imply that this maps directly to perceived quality. This metric is blind to pre-echo, for example, so the codecs could get a good result in spite of getting transients completely wrong...

As this shows, at high rates the formats aren't _that_ different (the story is different at lower rates, as MP3 falls apart completely very quickly). But I think it does a reasonable job of arguing that newer formats absolutely do offer at least the _potential_ benefit of a material quality improvements over MP3 at least for some kinds of audio.

Cheers,

A Call for Codecs Releases

Reply #15 –
kwanbis:
As far as I recall, the Ministry - Show Me Your Spine & Abfahrt Hinwil samples are good examples where lame fails at 192 VBR ... if you look at my old test you will have the illusion that lame mp3 competes well, that's because the test was built to hurt vorbis, samples where selected because they were bad on vorbis (not lame) ... I think I already stated that this was a weakness of my test, there were not enougth samples to reflect the weakness of lame mp3 in my old test (specially not enougth known bad samples for lame). First lame mp3 (like musepack) doesn't compete at all at 128Kbps & below against AAC, but even if the test gives the illlusion that it can resist vorbis at 192Kbps & above, in the end it is IMHO not the case. The difference between lame mp3 & other modern codecs is that lame is very likely to be affected by a larger number of killer samples. This is what I would call the codec resiliency or the robustness of the codec. If both codec A & B sounds the same on average music for an average listener but codec B is only affected by 50 killers samples when codec A is affected by 100 killer samples for an advanced user, then IMHO codec B is better no matter if the two codecs are tied on average music for the average listener. Now you can argue that there is a larger number of samples affecting lame mp3 than other codecs because so many users have been using it for so long & this is probably right. The robustness of a codec compared to the robustness of another codec is a very difficult thing to prove & I have no absolute proof of this. Still AAC exist for a reason, it wasn't created just to add DRM to mpeg audio. When you push lame mp3 to it's limits the technical merits of aac, as small as you may think they are, appear & that's where aac beats mp3.

Anyway, like sh1leshk4 my opinion is just my opinion. I was already "annoyed" in my previous test by codec fanboys, this is why I clearly stated my opinion about testing other codecs: to avoid other people interfering with their own interest.

-----------------------------------------------

As for the test, I have begun ABXing samples, I first started ABXing at aotuv 96Kbps, but I quickly changed my methodology & started using directly Nero AAC VBR Q0.40 (~125Kbps). Also I have changed the file/folder organization on my HDD, I have made 1 folder by sample & all folder sorted by sample length. (All samples within 1 folder was a mess as I begun adding plenty of .txt for ABX logs & personnal comments.)
Doing so I can discard more samples directly in the first pass as I admit I target AAC (which I think, based on my previous test, is more resilient than aotuv, so the basic idea is that if it's hard for nero aac it will likely be hard for aotuv [could be revealed false in the end, but it's the expected behavior & easier for me]). Also testing shorter samples first is faster, as I don't have to search where to listen within 30sec unknown samples.

Overall I have 56 Samples 18Min07Sec (I may add 4 more) & I tested 12 samples, I can ABX 8/8, 6 samples, among those 3 are real candidates, I cannot ABX at all 5 samples. 1 is 10/12 without knowing what I listen to (=discarded).

Candidates so far are:
06Sec - Amnesia (44Khz) doublon with (48Khz) both sounds the same
08Sec - Applaud
08Sec - Dimmu Borgir - Enthrone Darkness Triumphant - Midnight Masquerade

Possible are:
01Sec - Keys 1644ds
01Sec - Testsignal3

As I only tested the shorter samples I only did the easiest part so far, I expect to have trouble finding artefact within longer samples (...unless it's right at the start).
Things are always moving too slow I guess, but they are moving, which is the most important IMHO.

Edit: Lots of typos.

A Call for Codecs Releases

Reply #16 –
Quote from: sauvage78 on
I don't want to have anything to do with fat graphical software commercially oriented toward newbies, I don't want to install it at all, even temporary. I don't want to waste HDD space & put crap key in my registry just for Itunes AAC. Maybe nowadays Itunes AAC is better then Nero AAC, I have no clue. Even if it was the case, without a CLI usable in F2K & Eac3to, I think the quality gain would not be worth the usability loss. Even if, in all honesty, I personnaly really dislike their products (Macs , Ipods ...), I have nothing against Apples itself, I think the same about Nero GUI than I think about Itunes GUI, still I use Nero AAC CLI. It's not just about a CLI, it's about a CLI without dependency.
Since you had already displayed your keen interest in assessing audio samples objectively, albeit through empirical comparative methods, I would advise you to step down from hardly rational arguments such as above. Your blind listening test won't even begin to hint at whether the sample you're hearing originated from a pure sang CLI encoder or a GUI-with-a-workaround. If you're really into objectivism, why should you even care?

As for the gratuitous labeling of both LAME and Musepack, come on people, I'm sure those Norwegian pixies have better taste than that.

A Call for Codecs Releases

Reply #17 –
kwanbis:
I just realized that Abfahrt Hinwil was in my previous test... , in fact if I recall well it was a late addition to the test to reflect how bad lame mp3 can be. My previous test has shown that if you search you can find samples that hurts vorbis no matter the bitrate. I specially added Abfahrt Hinwil to illustrate that lame mp3 can be put to shame if your goal is to make it look bad. Just like vorbis, lame mp3 can be affected by some samples no matter the bitrate. You can argue that what is true for vorbis & lame mp3 is true for nero aac & you would be right this is likely true. But again the number of samples that affects nero aac no matter the bitrate are fewer & way much harder to ABX than Abfahrt Hinwil or Show Me Your Spine on lame 192Kbps VBR. Finding these samples is the reason why I target nero aac this time, which might reassure you in the fact that I don't blindly think that any lossy codec would be perfect or even better than the others just because it would be newer ... in fact I am more searching the least worst lossy codec. I am a lossless user at heart, not a nero aac fanboy. I only use nero aac because I am an x264 fanboy (is there any alternative in the video world? seriously) & I want an aac codec to use with it.

Polar:
You seem to not have understund, so I will repeat myself : the reason why I don't want to use Apple AAC has nothing to do with audio quality. It is a practical reason: no f2K & eac3 support= no ABXing. Endpoint. You want lame/musepack/itunes ABX results ... just do it yourself.
It might sound rude, but this is just pragmatism. I have no time to lose for codecs that I will not use in the end. I am sorry, but I do encourage you to test by yourself.
I am cold because in my previous test I surrender to the insistence of people wanting me to test musepack & apple aac, I tested them & it brought me nothing personnaly, so this time I don't care. I will accept constructive comments & critics but not requests.

I would prefer spending my time ABXing than discussing non-tested codecs, thks.

A Call for Codecs Releases

Reply #18 –
Today's update:

Here is how I sorted my database: (I post it so that you can understand what I am doing & where I am)

Code: [Select]
c:\08- Potential DCT Killer Samples (Lossless)\01- Possible Killer Samples (58 Samples 18Min28Sec)\     130 991 265 2011-02-16 00:35 -----
------------------------------------------------------------------------------------------------------------------------------------------
<01Sec - Keys 1644ds (Bad)>                                                                                   <DIR> 2011-02-14 19:27 -----
<01Sec - Testsignal3 (Bad)>                                                                                   <DIR> 2011-02-14 19:28 -----
<03Sec - Camille - 2005 - Le Fil - 02 - Ta Douleur>                                                           <DIR> 2011-02-14 19:28 -----
<05Sec - Carla Bruni - Quelqu'un M'a Dit - Tout Le Monde>                                                     <DIR> 2011-02-14 19:28 -----
<06Sec - Amnesia (44Khz) (Bad)>                                                                               <DIR> 2011-02-14 19:38 -----
<06Sec - Amnesia (48khz) (Bad)>                                                                               <DIR> 2011-02-14 19:54 -----
<08Sec - Applaud (Bad)>                                                                                       <DIR> 2011-02-14 20:00 -----
<08Sec - Dimmu Borgir - Enthrone Darkness Triumphant - Midnight Masquerade (Bad)>                             <DIR> 2011-02-14 20:12 -----
<10Sec - Birthday Dethday>                                                                                    <DIR> 2011-02-14 20:35 -----
<10Sec - Convulsion>                                                                                          <DIR> 2011-02-14 21:03 -----
<10Sec - Emmylou Harris - 1995 - Wrecking Ball - 08 - Sweet Old World>                                        <DIR> 2011-02-14 21:29 -----
<10Sec - Herbie Hancock>                                                                                      <DIR> 2011-02-14 21:15 -----
<10Sec - Trumpet>                                                                                             <DIR> 2011-02-15 18:48 -----
<10Sec - User Friendly>                                                                                       <DIR> 2011-02-15 18:57 -----
<11Sec - Camille - 2002 - Le Sac Des Filles - 11 - Là Ou Je Suis Née>                                         <DIR> 2011-02-15 19:12 -----
<11Sec - Canadian Idiot>                                                                                      <DIR> 2011-02-15 19:30 -----
<11Sec - Foo Fighters - 2007 - Echoes, Silence, Patience & Grace - 03 - Erase-Replace>                        <DIR> 2011-02-15 20:28 -----
<11Sec - Herding Calls>                                                                                       <DIR> 2011-02-15 20:35 -----
<13Sec - Berlin Drug (Bad)>                                                                                   <DIR> 2011-02-15 23:40 -----
<14Sec - Part I - Keith Jarrett (The Köln Concert)>                                                           <DIR> 2011-02-16 00:10 -----
<14Sec - Trumpet Myprince (Tag II - 6.1. Vbr Panic (Trumpet My Prince)>                                       <DIR> 2011-02-15 23:57 -----
<15Sec - Closer To God>                                                                                       <DIR> 2011-02-16 00:13 -----
<15Sec - Fear Factory - 2001 - Digimortal - 05 - Linchpin (Bad)>                                              <DIR> 2011-02-16 00:18 -----
<15Sec - Nin Quake (Done)>                                                                                    <DIR> 2011-02-16 00:31 -----
-----------------------------------------------------------------------------------------------------------------------
<16Sec - Cranberries - Zombie>                                                                                <DIR> 2011-02-14 17:28 -----
<16Sec - Girl>                                                                                                <DIR> 2011-02-14 17:28 -----
<16Sec - Harp 40.1>                                                                                           <DIR> 2011-02-14 17:28 -----
<17Sec - Richard Dawkins - The God Delusion (CD1-Track02)>                                                    <DIR> 2011-02-14 17:28 -----
<18Sec - Fighter Beat>                                                                                        <DIR> 2011-02-14 17:28 -----
<18Sec - Grip Inc. - Nemesis - Descending Darkness>                                                           <DIR> 2011-02-14 17:28 -----
<20Sec - Green Day - Waiting>                                                                                 <DIR> 2011-02-14 17:28 -----
<20Sec - Hexonxonx>                                                                                           <DIR> 2011-02-14 17:28 -----
<20Sec - Ministry2>                                                                                           <DIR> 2011-02-14 17:28 -----
<23Sec - Kool & The Gang - Ladies Night>                                                                      <DIR> 2011-02-14 17:28 -----
<23Sec - Memories>                                                                                            <DIR> 2011-02-14 17:28 -----
<23Sec - Rammstein - 2009 - Liebe Ist Für Alle Da - Roter Sand>                                               <DIR> 2011-02-14 17:28 -----
<25Sec - 2nd Vent Clip>                                                                                       <DIR> 2011-02-14 17:28 -----
<25Sec - Lead-Voice>                                                                                          <DIR> 2011-02-14 17:28 -----
<25Sec - Spahm>                                                                                               <DIR> 2011-02-14 17:28 -----
<26Sec - Bibilolo>                                                                                            <DIR> 2011-02-14 17:28 -----
<29Sec - 06 Mandylion>                                                                                        <DIR> 2011-02-14 17:28 -----
<29Sec - Fatboy>                                                                                              <DIR> 2011-02-14 17:28 -----
<29Sec - Musique Non Stop>                                                                                    <DIR> 2011-02-14 17:28 -----
<29Sec - The Missing>                                                                                         <DIR> 2011-02-14 17:28 -----
<30Sec - Aquatisme>                                                                                           <DIR> 2011-02-14 17:28 -----
<30Sec - Boing Boom Tschak>                                                                                   <DIR> 2011-02-14 17:28 -----
<30Sec - Creuza>                                                                                              <DIR> 2011-02-14 17:28 -----
<30Sec - Human Disease>                                                                                       <DIR> 2011-02-14 17:28 -----
<30Sec - Mahler>                                                                                              <DIR> 2011-02-14 17:28 -----
<30Sec - Ministry - Ptp Robocop>                                                                              <DIR> 2011-02-14 17:28 -----
<30Sec - Spill The Blood>                                                                                     <DIR> 2011-02-14 17:28 -----
<30Sec - Taco - Puttin On The Ritz>                                                                           <DIR> 2011-02-14 17:28 -----
<30Sec - Techno Pop>                                                                                          <DIR> 2011-02-14 17:28 -----
<30Sec - The Robots>                                                                                          <DIR> 2011-02-14 17:28 -----
<30Sec - The Robots (2009 Remaster)>                                                                          <DIR> 2011-02-14 17:28 -----
<30Sec - Tiesto - Do You Feel Me>                                                                             <DIR> 2011-02-14 17:28 -----
<31Sec - Ecstasy>                                                                                             <DIR> 2011-02-14 17:28 -----
<43Sec - The Product>                                                                                         <DIR> 2011-02-14 17:28 -----

I have tested 24 samples so far, so it's done up <15Sec - Nin Quake (Done)> included.
Today I tested 12 samples, among those I can only ABX 2 samples which are:
Candidate:
15Sec - Fear Factory - 2001 - Digimortal - 05 - Linchpin ==> 8/8
Possible:
Berlin Drug ==>11/12

I will post my ABX log & maybe a recapitulative table with comments of what I hear when all will have been tested.
Those I can ABX are marked as (Bad), if you can ABX one of the failed sample at Nero AAC VBR Q0.40 (~125Kbps), telling me what you hear & when, would be usefull.
(The sample I failed may be for other codecs than Nero AAC, I search back the forum when I can.)

A Call for Codecs Releases

Reply #19 –
If it's any help for you, well, here it is.

Nero -q0.42 present problem with following samples:
fatboy - transient issue. (very bad artifacts on artificial speech)
creuza - tonality issue (adds some extra sine waves, especially in left channel)
since always - stereo direction (stereo degraded to a kind of mono)
waiting - artifacts on voice of the singer.

P.S. And maybe sample of classic music as well. Instruments are the cellos. Beautiful piece of music. Many codecs sound like crap on it . I concentrate on some particular instrument and then big picture (all instruments).
www.mediafire.com/?e8ldh4urm3ymnn7

These samples were intensively reported on many previous public tests (as really bad).
 

A Call for Codecs Releases

Reply #20 –
Cant wait I'm excited about the new version of aotuv ogg. Good luck too you're abx testing.