Skip to main content

Topic: Pre-Test discussion (Read 32103 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • rjamorim
  • [*][*][*][*][*]
Pre-Test discussion
Reply #75
Quote
I agree in principle, but why QT in one and Nero in the other?    I must be missing something.

Because I'm facing lots of pressure from people wanting a listening test with Nero AAC since it didn't do so well. See it as a revenge attempt  (And no, I'm not talking about Ivan)

Besides, if I don't fuck up, the results of both tests would be comparable among themselves. You wouldn't be able to draw definitive conclusions (that would require same set of samples, same listener group, same listening conditions for each listener...), but you could at least have an idea.

Quote
Do we really need an anchor?


IMO we do, as I said before, to put things into perspective.

Low bitrate tests are even worse, they need two anchors. :B


@ff123: Have you found out if Blade is suitable for anchor?

Regards;

Roberto.


Edit: Yes, the mass-test would be the first.
  • Last Edit: 19 July, 2003, 11:03:08 AM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Pre-Test discussion
Reply #76
Quote
@ff123: Have you found out if Blade is suitable for anchor?

I ran through some comparisons, and here's how I would find things:

Code: [Select]
                   10kHz                   blade
41_30sec           easy                    very easy
atrain             moderate                moderate
beautyslept        easy                    easy
blackwater         moderate to difficult   easy
death2             moderate to difficult   very easy
flooressence       moderate to difficult   very easy
layla              moderate                easy
lifeshatters       moderate                very easy


I think blade would make a better anchor than 10 kHz, even though some samples may be harder to find problems in than others.

Note:  Probably some people would find 10 kHz easy to detect on all samples.  Some people might find it very hard to tell the difference on some.  Maybe somebody else could ring in and try out a few?

ff123

  • rpop
  • [*][*][*][*][*]
  • Global Moderator
Pre-Test discussion
Reply #77
So the masses test is the one scheduled to start Wednesday? How much later will the HA test start?
Happiness - The agreeable sensation of contemplating the misery of others.

  • rjamorim
  • [*][*][*][*][*]
Pre-Test discussion
Reply #78
Quote
So the masses test is the one scheduled to start Wednesday?

Yes. Unless someone comes with a very good reason not to split the test in two.

(BTW, big thanks to Dibrom for pointing out the issue of the target audience)

Quote
How much later will the HA test start?


Much later. Before that, I plan to test codecs at 64kbps, and then do some vocodec tests with the help from JM Valin. Expect the HA test to happen somewhere around late this year, early next year (maybe).

With the extension test, it's enough 128kbps tests for now. :B

Regards;

Roberto.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Pre-Test discussion
Reply #79
Roberto,

Before you start the test, could you upload a test archive, so that I can doublecheck volume level and time alignment?

ff123

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Pre-Test discussion
Reply #80
Quote
The Masses test would compare QuickTime, Lame, Sony┬┤s Atrac3, WMA STD and Vorbis

The HA test would compare Nero AAC, Lame, Musepack, WMA PRO and Vorbis.

Just an apprehension : do you really consider QuickTime (and his codec) as a solution for the masses ? I know there's iTune [span style='font-size:8pt;line-height:100%'](really same encoder : not sure)[/span], but it's for the next 6 months limited to MAC world (not really a mass-system). On the other side, QuickTime doesn't accept CDRipping or even batch encoding. And I prefer forget mass-tagging issues here... Mass or experts, QT isn't a useful tool. Last but not least, QT isn't intuitive for most people (save as... dosn't work : you need to export, and of course to know it).

Ahead encoder is so simple to use in comparison. And so many people have Nero installed on their computer. And popularity will certainly increase with the HE-AAC encoder, and its impressive quality at ~64 kbps.


If you really want to put both encoders on two different tests, I find more coherent to keep Nero VBR for the mass-test, and QT CBR for the HA one.
  • Last Edit: 19 July, 2003, 06:38:56 PM by guruboolez

  • bond
  • [*][*][*][*][*]
Pre-Test discussion
Reply #81
1) plz dont forget that the tests are about sound quality!!!!
and not about usability of the encoding tool or something else (which has nothing to do with the purpose mentioned above)...

2) codecs which were judged worse than another in a fair, scientific test (like quicktime - nero) are out at the tested bitrate (no need for a revenge until there are serious changes on the encoder side)

3) first of all everybody should ask himself:
"hey, what's the sense of a codec comparison @128kbps?"

-> people want to know which codec gives the best quality output at that bitrate

-> only the best codecs should be compared
-> quicktime, lame, vorbis, wma9 pro
(if you want to add additionally atrac3 or musepack or any other codec isnt really important to reach the goal of the comparison)

sorry to hear that rjamorim plans this (imho most) important comparison very late this year (or even next year) 
  • Last Edit: 19 July, 2003, 06:55:28 PM by bond
I know, that I know nothing (Socrates)

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Pre-Test discussion
Reply #82
Quote
plz dont forget that the tests are about sound quality!!!!

and not about usability or something else (which has nothing to do with the purpose mentioned above)...

Indeed. But in my opinion, the difference between mass and 'expert' presuppose some ergonomy consideration.
Maybe the idea wasn't well explained :
- a PC only test (= HA test)
- a portable-use test (= mass test)

Even here, I have some reticences. I can't imagine someone filling an iPod with QT installed on his PC. Just try to encode one album only : it's a real pain.
A listening test should be useful, especially when practical purpose is in mind (call it portable-test or mass-test, it doesn't change anything).

When Doom9 performed nice codec comparison, he doesn't maximise perceptual quality with impossible avisyth script, and he's not using H264 encoder family - better theorical quality, but so slooooow... He have quality in mind, but doesn't forget to be in harmony with his readers behaviour and possibility
  • Last Edit: 19 July, 2003, 06:52:30 PM by guruboolez

  • bond
  • [*][*][*][*][*]
Pre-Test discussion
Reply #83
ups crosspost

added point 2 and 3 in my last post...


ok, if you call it "portable-test" than the purpose of the test changed from "quality only" to "usability and quality", in this case i would add, like proposed from guruboolez, nero to the "mass-portable test" instead of quicktime...


am i the only one who thinks that the "ha-(only the best)-test" is more important and should be done first?
  • Last Edit: 19 July, 2003, 07:06:35 PM by bond
I know, that I know nothing (Socrates)

  • rjamorim
  • [*][*][*][*][*]
Pre-Test discussion
Reply #84
Hello.

@Guruboolez: I understand your concern about QT not being really usable. But it has the best quality, it won the AAC test, and I'm not willing to fathom the criticism I expect will come from all sides about "WTF, QuickTime won the AAC test but you used Nero on the extension test though? Are you working for Ahead now, you bastard?".

You see, doom9 is a good example, but what is tying my hands now is the AAC test result.

@Bond: IMO, the HA test can't be conduced now. Even though it might be more important, still it would be wasting all the attention the AAC@128kbps test gathered.I plan to keet the HA test more "low profile", and not announce it everywhere else, at most at CD-rw.org

And no, this is a quality test. Even though usability is cool, it's an utterly subjective concept, and I want to keep these tests objective.

And, of course, I can be convinced to hurry up the HA test a little.


@ff123: I'll contact you as soon as I have something ready (probably this tuesday, when I return home)
Have you found out the Blade offset when decoded by Lame?

Thank-you all.

Regards;

Roberto.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • bond
  • [*][*][*][*][*]
Pre-Test discussion
Reply #85
Quote
"WTF, QuickTime won the AAC test but you used Nero on the extension test though? Are you working for Ahead now, you bastard?"



Quote
IMO, the HA test can't be conduced now. Even though it might be more important, still it would be wasting all the attention the AAC@128kbps test gathered.I plan to keet the HA test more "low profile", and not announce it everywhere else, at most at CD-rw.org

i dont think that it will waste attention
-> the only "big" difference between the two tests is wma9 pro and std
-> the most important purpose for using an audio codec is storing music clips on the hd (portables are much less important imho compared to storing on hd (especially @128) and i dont want to know how many people from the masses reencode their stuff to lower bitrates anyways before using it in a portable)...

-> so if wma9 std will be replaced by the pro version, this will not lead to a loss in attention imo even the dumbest user will think: "hey, of course the pro version is better than the standard version and rjamorim also used the pro version, so there can be no doubt that its better, lets drop wma9 std @128"

-> i dont think that it will be a fault to let the pro version out on the masses (and the wma9 std codec will be important in the 64kbps test anyways (especially for portables))

Quote
And, of course, I can be convinced to hurry up the HA test a little.

would be really great
  • Last Edit: 19 July, 2003, 07:49:22 PM by bond
I know, that I know nothing (Socrates)

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Pre-Test discussion
Reply #86
Quote
@Guruboolez: I understand your concern about QT not being really usable. But it has the best quality, it won the AAC test, and I'm not willing to fathom the criticism I expect will come from all sides about "WTF, QuickTime won the AAC test but you used Nero on the extension test though? Are you working for Ahead now, you bastard?".

What I tried to say is : if you really want to include Nero encoding in one of both test, put it in a "mass" or "popular" part.
I really want to see QT@128 compete with other formats. It was designed as the winner, confirming the positive impressions I had before the public test, and I felt again during this test. There are no reason for me to eliminate QuickTime. On contrary : I have more difficulties to conceive good reasons for giving to Nero encoder a second chance - especially on a dedicate "HA test", - designed to be the high-quality-and-quality-only test. I like Ivan's work ; I know that Nero's codec was handicaped by a forced CBR encoding during test ; I'm aware that some progress were made since first test... but without public testing, proving its superiority on QT 6.3 on most samples, introducing Ahead in the next test is for me something really odd.
  • Last Edit: 19 July, 2003, 08:56:33 PM by guruboolez

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Pre-Test discussion
Reply #87
Quote
@ff123: I'll contact you as soon as I have something ready (probably this tuesday, when I return home)
Have you found out the Blade offset when decoded by Lame?

Yes, the offset is 0

ff123

  • rjamorim
  • [*][*][*][*][*]
Pre-Test discussion
Reply #88
Quote
Yes, the offset is 0

ff123

Thanks



OK, so here's the deal:

-Sonicstage almost fried my Windows installation (and I used to think QuickTime is buggy). Also, I couldn't find out if I should rather use the SonicStage encoder or the Real Producer. As I said before, they are targeted at different purposes, and their encodings differ. I couldn't chose the portable one or the multipurpose one, and noone seemed to help in that decision.

Bottom line: Atrac3 is out.

-Given that Atrac3 is out, it's being replaced with Musepack and I won't split the tests.

-WMA standard won't be tested because, knowing Microsoft, it'll be quickly ditched and replaced with it's superior brother. So, only WMA pro wil be tested. Lack of hardware support isn't an argument, if that was an argument, MPC and Vorbis would be out too. I can expect that, in some months, there'll be more hardware supporting WMA Pro than Vorbis, or AAC.

-What about Nero? I didn't think about it yet, so don't ask. For the time being, it won't be tested (at least by me)

-most important reason not to split tests (and I quote Guruboolez)
"Test is boring. Two tests will be a real pain."

Test starts in about one hour, if all goes right while uploading.

Best regards;

Moi.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • bond
  • [*][*][*][*][*]
Pre-Test discussion
Reply #89
Quote
Atrac3 is out ... WMA standard won't be tested ... not to split tests

good news,
(i guess we will see wma9 std and atrac/realaudio again in the 64kbps test to come anyway...)

Quote
What about Nero? For the time being, it won't be tested

so quicktime is used instead? hm, i think so 

Quote
Test starts in about one hour, if all goes right while uploading.

gimme, gimme 
I know, that I know nothing (Socrates)

  • rjamorim
  • [*][*][*][*][*]
Pre-Test discussion
Reply #90
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • den
  • [*][*][*][*]
Pre-Test discussion
Reply #91
Good stuff Roberto. I think you have made the right decisions.  B)

Quote
Sonicstage almost fried my Windows installation (and I used to think QuickTime is buggy).


I also had a near death experience with Sonicstage  , hence my continuing use of OpenMG Jukebox, which has the same internals as Sonicstage, but uses a slightly less funky/sexy interface, and a more reliable install routine.

Welcome to the world of NetMD, Sony style. :x

Looking forward to this test, and will get my listening done over the weekend...

Den.

  • rjamorim
  • [*][*][*][*][*]
Pre-Test discussion
Reply #92
Quote
Good stuff Roberto. I think you have made the right decisions.  B)

Thanks

Quote
I also had a near death experience with Sonicstage   , hence my continuing use of OpenMG Jukebox, which has the same internals as Sonicstage, but uses a slightly less funky/sexy interface, and a more reliable install routine.

Welcome to the world of NetMD, Sony style. :x


True.

maybe I'll test it one of these days. In this occasion, I would probably create a Win98 partition with VirtualPC and install SonicStage inside, and later delete it. But it sure as hell has a very low priority now.

Quote
Looking forward to this test, and will get my listening done over the weekend...


Great, looking forward to your test results.

Regards;

Roberto.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org