128kbps Extension Test - OPEN

Topic: 128kbps Extension Test - OPEN (Read 59668 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

128kbps Extension Test - OPEN

Reply #175 – 2003-07-29 20:21:57

Quote

A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

WMAPro in VBR variated too much, and a setting couldn't be choosen. That's why the test uses the ABR mode (2-pass).

128kbps Extension Test - OPEN

Reply #176 – 2003-07-29 20:29:35

Quote

A question which hasn't been answered yet: what about WMAPro 2 pass? By using 2 pass you are forcing a vbr codec to a certain bitrate; tell it "to make the most out of 128k" which is unfair because you are artificially limiting the codec.

WMA9Pro 2-pass VBR is a difficult case. The normal WMA9Pro VBR had wild bitrate swings from album to album, and we had to rule out using that in this comparison because the average bitrate across different albums just wasn't stable enough.

So we figured 2-pass VBR would be the next best thing. However, in the ideal world, the 2-pass would be calculated by encoding an entire album, or at the very least, an entire song, and then the sample would be sliced out afterwards.

Note that there is at least one critical difference between WMA9Pro 2-pass and limiting the bitrate on the other VBR codecs -- Microsoft actually provides a 2-pass mode, and the other codecs don't. So 2-pass is a mode that listeners might actually choose to use, whereas limiting the bitrates on the other VBR codecs would not be how they would normally be used.

ff123

128kbps Extension Test - OPEN

Reply #177 – 2003-07-29 20:44:13

Just wanted to inform everyone here that the test pages have been moved to Paul's server for practicity reasons.

The new url is:
http://audio.ciara.us/test/

RareWares will stay here at HA, only the Listening Tests subsection will go.

Sorry for any inconvenience.

Regards;

Roberto.

128kbps Extension Test - OPEN

Reply #178 – 2003-07-29 21:02:05

Quote

So what does this tell you about the quality of the motor?

Not much, but we're shopping for a whole car, not just a motor. Who cares how efficient your motor is if you loose all that efficiency in a crappy transmission.

Quote

The vbr boxer reacts and pulls out a heavy shield that can deal with even the most forcefull blows, but the cbr boxer simply collapses. Again, what does this tell you?

It tells us that our glassjaw CBR boxer should find a new line of work before he gets himself killed.

In this analogy, if defense represents adaptability of bitrate, what does "ring generalship" stand for? Which codec should be accused of ear biting?

128kbps Extension Test - OPEN

Reply #179 – 2003-07-29 21:42:22

@rjamorim

Thank you for your great work with that test!

Regards, fileman.

128kbps Extension Test - OPEN

Reply #180 – 2003-07-31 14:02:06

I will just say one more thing before I wave my white flag and/or get banned for being too stubborn, whichever comes first Please, spend some time on my little rant, one last time.

Another 'hypothetical situation' for you:
Suppose we have a test between 2 encoders, both VBR and only 2 samples.

Now we run the encoders with 'quality settings such that they will average at 128kbit, or to put it differently, such that in the end the sum of all filesizes add up to exactly the same: 327680 bytes. Both encoders have used the same amount of bytes to 'play with'.

Now. The results: *drumroll*
Encoder A:

- Bachpsichord, encoded at 192kbit average.
- Hardrock, encoded at 64kbit average[/li][/list]Encoder B:

- Bachpsichord, encoded at 128kbit average.
- Hardrock, encoded at 128kbit average.[/li][/list]

Now we run the listening tests, and suppose it turns out that overall, the Encoder A and Encoder B versions of both Bachpsichord and hardrock tracks get the same score. (they don't sound the same, they just sound equally 'pleasing' when you consider overall score). Or if you wish, more skewed: Say that Encoder A gets a slightly higher score than B on bachpsichord, but a slightly lower score than B on hardrock and the average score is the same.

And now my point is:

The result: "well, that's that, Encoder A and Encoder B are equal for 128kbit." is what would be the conclusion of this test like everybody supports here. And I agree! It is Fair.

But.

If you weigh the averages, you can say MORE. There is more information in the results! You can then say:
- Encoder A probably has an inferior coding method for bachpsichord, and needs more bits to reach its target quality.
- Encoder A probably has a superior coding method for hardrock and needed fewer bits to reach its target quality

And this is useful information, wouldn't you say? Isnt this what this test is all about? Finding out as much as we can from listening tests we do, trying to compare the various files? Wouldn't it be nice to be able to say "Well, for Classical music, mpc scored highest, while for hardrock ogg vorbis and AAC are the preferred choices? (just making something up here).

Plus, of course my example is highly stylized. With a lot more encoders, a lot more samples, filesizes and so on, it is not inconceivable that interesting extra results can come to the light. Again, I'm not saying this weighing scheme is rock solid, it, like the test itself, gives some extra data to think about. I really don't see why you would want to ignore this.

As a last comparison, if you are choosing which car to buy, and a test gives Car A and Car B *exactly* the same score and same price, then you don't just pick one blindfolded. No. You go and check what exact features both cars have that you consider important[/i]. One car will go faster while the other is safer. A USEFUL car test won't just say "Car A and B are equal", no, a useful test will say "Car A and B are equal, Car A is faster and more dangerous, car B is slower and safer". Pick what you prefer.

128kbps Extension Test - OPEN

Reply #181 – 2003-07-31 14:20:13

Quote

This reminds me of a benchmark I recently heard about comparing a Pentium 4s to a G5. They disabled hyperthreading on the P4 to make the test "fair". That's like saying if one CPU had an extra fast cache, it should be disabled to make the test "fair". Nonsense of course.

This is sort of off-topic (except that it shows how difficult benchmarking and testing can be) but:

They disabled hyperthreading on the P4 because the P4 got *worse* scores when it was turned on. What you think was done to make things 'fairer' for the G5 was actually done to benefit the P4.

Weird, huh?

128kbps Extension Test - OPEN

Reply #182 – 2003-07-31 14:42:34

Quote

- Encoder A probably has an inferior coding method for bachpsichord, and needs more bits to reach its target quality.
- Encoder A probably has a superior coding method for hardrock and needed fewer bits to reach its target quality

No, it might mean that encoder A has a superior psychoacoustic model, which detected, that first sample will be hard to encode and used less bits for easier second track. It is right if the same codec fails at bachpsichord using less bitrate.

Do you think that 64kbps MP3 is superior to 128kbps AAC, because the latter uses more bits?

Anyway, we're testing QUALITY, not QUALITY/SPACE tradeoff. Go figure.
In the second case, 48kbps MP3 might be better than 64kbps one,
because quality loss is substancially smaller than filesize loss.

128kbps Extension Test - OPEN

Reply #183 – 2003-07-31 15:32:26

Quote

Wouldn't it be nice to be able to say "Well, for Classical music, mpc scored highest, while for hardrock ogg vorbis and AAC are the preferred choices? (just making something up here).

Yes, conclusions are fine to me.
But you forget one thing. Roberto's test had a precise purpose : building an approximative hierarchy between many audio format, at ~130 kbps, and for general purpose. You can't conclude anything reliable and precise on musical genre, with only 12 samples. If you wan't to know which codec is the best for harpsichord, encode at least three different harpsichord sample (harpsichord didn't have a big dynamic range, but with recording conditions, manufacturing, and music content, results may change). For piano, you should take at least 6 samples : low/mid/high volume, noise or not, precise recording or more distant one... For orchestra, damn, it will be more difficult : strings, wind, full-orchestra, solo instruments, recording conditions, etc... are many variations of a same reality. And you to continue with string quartets, solo recorder, trumpets, wind ensemble, voice of course, organ... And I'm only talking here about "classical" genre.

In Roberto's test, there are two classical samples : macabre (orchestra) and bachpsichord. It doesn't make sense to conclude that mp4 is the must accurate encoder for piano and voice, just because mp4 was the best for macabre.wav. There's more sense to conclude than sample X, winner of the bachpsichord test, should be a privileged choice for solo harpsichord discs (nice : it's <0,01% of the whole discs of this planet), simply because another harpsichord recording should differ too much. For orchestra, I wouldn't conclude anything on macabre.wav results. Of course, blade failed horribly, and had 99,99% chance to be a bad choice on other orchestra parts. But the winner on these 20 seconds, how many chances to keep the first place on a long clarinet solo ? On a tutti ? A string introduction ?

Roberto's test conclusion will give us less answers than lead. If an encoder will fail on electronic sample (#5), and be the best on metal (#10), we can't conclude that this encoder is good for metal and useless for electro. We just have to investigate further to be sure, by testing other similar but different samples. It's just a preliminar hypothesis, that need to be proved by user experience, or by isolated tests.

(Note : for harpsichord, my choice is made. I know for a good time what encoder is the best for my ears. I'm just waiting the end of the test before any comments).

128kbps Extension Test - OPEN

Reply #184 – 2003-08-03 15:52:34

rjamorim,
i am still listening to some samples, so dont hurry to close the test

btw. does anybody know from where the macabre clip was taken (i mean from which composer, etc...)

128kbps Extension Test - OPEN

Reply #185 – 2003-08-03 16:05:17

Hehe. Well, the test will be closed tonight at midnight brazilian time (4AM of Monday GMT)

So, please send in your results until then

128kbps Extension Test - OPEN

Reply #186 – 2003-08-04 02:00:52

[span style='font-size:14pt;line-height:100%']TEST CLOSED![/span]

No more mails accepted from this moment on.

Expect results in a few hours.

128kbps Extension Test - OPEN

Reply #187 – 2003-09-10 04:18:46

Quote

Here is the sample batch - you need to make an executable from it!

And how do I do that?

I tried a tool called bat2exe, it didn't work (batch execution stops at the middle)
http://nlsn.free.fr/batch-down/Bat2Exe.ZIP
That tool actually creates .com files.

Can somebody help me out with a good bat to exe tool?

128kbps Extension Test - OPEN

Reply #188 – 2003-09-10 06:25:49

Just tried this one:
http://www.bdargo.com/info/batcomp.htm

Compiles the batch files well and these .exe from .bat work perfectly here at home. But they generated GPFs on ff123's PC, and output some very weird errors (could not find C:\bdtmp)

Notice