HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: rjamorim on 2003-07-16 04:07:21

Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 04:07:21
Hello.

As most of you already know, I'm planning an extension test for the AAC@128kbps test. It'll be the AAC winner (QuickTime) versus Musepack, Vorbis, Lame ABR, WMAv9 and maybe Atrac3 (more on that later). The test should start (if everything goes right) in a week (the 23rd)

There are some subjects where I'm still not sure how to proceed, so I would like to rely on the forum users' knowledge:

1 - Should I include Atrac3 (used on Sony MiniDiscs and Real Audio at mid-high bitrates) in this test? Is there anyone out there interested in this format? (besides Den ;) )

2 - I plan to use anchors in this test. For those that don't know, anchors are samples that suffer no processing from the original apart from lowpass. They are used to put the results into perspective and avoid participants rating the encoded samples too high and/or too low. The questions that arise are:
a) 2 anchors (high and low) are needed, or only one anchor is enough? (IMO, two anchors are more useful in low bitrate tests, and don't make much sense in high bitrate ones)
b) JohnV votes for one lowpass for each sample, else some samples might sound better lowpassed than their original versions. ff123 votes for one fixed lowpass for all samples, to put things into perspective. What's your personal opinion?
c) Anyone has any idea of good lowpass(es) for the test? It's a difficult choice since low lowpasses might sound too unpleasant to the fellas with good hearing and pro gear, but high lowpasses might sound too transparent to the fellas with not-so-good hearing or simple gear.

Thanks for your help.

Regards;

Roberto Amorim.
Title: Pre-Test discussion
Post by: rpop on 2003-07-16 04:22:05
I'd go along with JohnV on this one, since I've had problems with samples sounding better lowpassed. As for the frequency, I used the PCABX Training Room (http://www.pcabx.com/training/index.htm) to test my limits, and I'd say either the 15 or the 18 kHz, as those are the ones where I actually had to focus.

Edit: I'm a newbie, so don't take my opinion as being worth very much. Just an extra push one way or the other, if the opinions are split even. I'm sure some of the regulars could make much better arguments than I did (actually, I don't think I made any arguments).

I look forward to the test! 
Title: Pre-Test discussion
Post by: ff123 on 2003-07-16 04:30:59
I was thinking of a 10 kHz lowpass, which should sound pretty bad to most people on most samples.  The idea is to prevent a codec, which may sound pretty good if evaluated by itself, from being given unfairly poor ratings when compared with the other codecs.  For example, lame might be predicted to be the loser of this test.  But surely it doesn't deserve to be rated near 1.0, which might happen sometimes if there's nothing to keep things in perspective (what then about truly crappy codecs like blade?)

I understand JohnV's concern, though.  Some types of music (for example solo piano) might still sound good with a 10 kHz lowpass.

Actually, another possible idea is to use blade as the anchor.  I'm pretty sure this would always be rated the worst, and is likely to occupy the position near a rating of 1.0

I think only one anchor is needed (the low anchor).  The hidden reference, which must be chosen from in each comparison, should be functioning as the high anchor.

ff123
Title: Pre-Test discussion
Post by: ff123 on 2003-07-16 04:33:32
Quote
I'd go along with JohnV on this one, since I've had problems with samples sounding better lowpassed. As for the frequency, I used the PCABX Training Room (http://www.pcabx.com/training/index.htm) to test my limits, and I'd say either the 15 or the 18 kHz, as those are the ones where I actually had to focus.

Either 15 kHz or 18 kHz would be useless for me as a participant (and I presume for some others as well).  Those lowpasses are transparent to me in music.

ff123
Title: Pre-Test discussion
Post by: boojum on 2003-07-16 05:41:49
I would be interested in seeing how ATRAC stacks up (alliterative pun not intended) only because I have an MD player and find the sound pretty good.  I would like to see how it rates against the other compression systems.  Not a big deal; only if it is not a pain in the butt.   
Title: Pre-Test discussion
Post by: Joseph on 2003-07-16 05:59:29
Someone should really create a Listening Tests Forum.  All these tests are hard to find when they get old.

The following topics could be moved to the "Listening Tests" forum:

Link 1 (http://www.hydrogenaudio.org/forums/index.php?act=ST&t=5&f=14&view=findpost&p=48)
Link 2 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134)




<edit: added links>
Title: Pre-Test discussion
Post by: den on 2003-07-16 06:31:12
Quote
Should I include Atrac3 (used on Sony MiniDiscs and Real Audio at mid-high bitrates) in this test? Is there anyone out there interested in this format? (besides Den  )


and boojum! Wow, I'm not alone! 

In all seriousness. My first thought is to give ATRAC3 a miss because of the limited interest and usefulness. On the other hand, it would be interesting to see just how it does compare with others at this bitrate, not that your average Minidisc user can do anything about it. Mind you, there are the new Sony mp3/ATRAC3 CD Discmans (or is that Discmen  ) now, so maybe it would be useful for somebody out there.

Roberto, how are you planning to make the ATRAC3 files and then present them for testing decoded into wavs? If you use the Sony software, there is no export as wav option. If you use RealProducer, my understanding is that this version of ATRAC3 is not as polished as the latest versions used in the Sony software... 

Den.
Title: Pre-Test discussion
Post by: ff123 on 2003-07-16 06:53:49
Quote
If you use the Sony software, there is no export as wav option.

Sounds like a job for TotalRecorder.  It's a pain in the ass, but it can be done.

ff123
Title: Pre-Test discussion
Post by: den on 2003-07-16 07:26:55
Quote
Sounds like a job for TotalRecorder. It's a pain in the ass, but it can be done.


Does this approach leave any opportunities for noise being added to the file due to the recording step, or is it noiseless/lossless?

I don't want to seem picky, I'm just curious.   

Den.
Title: Pre-Test discussion
Post by: ErikS on 2003-07-16 07:36:28
1. I'm also interested in how ATRAC performs compared to other alternatives for portable use like mp3 and aac.

2. Sounds like a good idea to use an anchor. And for an anchor to be useful it should be fixed, right? Anything below 12 kHz would be fine for me. Not too low though - if less than say 6 kHz I think the gap to the tested codecs will be too big to actually be able to relate their quality...
Title: Pre-Test discussion
Post by: Gabriel on 2003-07-16 08:14:22
Even if atrac3 does not seems really interesting for us, I think that it should be included because Sony is trying to push it on portable devices instead of mp3.
Title: Pre-Test discussion
Post by: den on 2003-07-16 10:01:41
Quote
Sounds like a job for TotalRecorder. It's a pain in the ass, but it can be done.


It's amazing what you can find if you really, really put your mind to it. I've tried to find ways before of playing with ATRAC3 outside of the Sony software for my own educational purposes, but I always lucked out.   

I just tried again, and stumbled across this... 

ATRAC3.zip (http://www.minidisc.org/atrac3.zip)

It is an acm style ATRAC3 codec, and once installed you can encode and decode ATRAC3 in the three common bitrates using the windows audio tools. I quickly tried it with Audiograbber (it was handy), and hey presto it spewed out an ATRAC3 132 kbits file inside a wav wrapper, which I could then playback/replaygain/write to wav etc with trusty foobar.  B)

I then encoded the same file with OpenMG, and tried to do a file comparison to look for differences, but the Sony version is an .omg file and there is a slight different in size ~4k in a 8.5 Mb track (@132 kbits). The Open MG generated file has tag data on the end when you check it in a hex editor. Someone else who know's what they are doing might want to look at this more closely and see if this is a reasonable way to generate some files for testing that are representative of what the Sony software churns out.

I also had a quick look at the decoded file in EAC. Has a radical haircut above 17500 Hz...  :x

This should make things easier for you Roberto if you do decide to include ATRAC3 in the test. I'm also going to have a play with this further just for my own interest...It makes ABX tests with ATRAC a shed load easier.

Den.
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-16 11:01:07
The ACM ATRAC3 encoder is totally different from SonicStage's one.
Speed, first (on my Duron 800) :

ACM = x4
SonicStage = x16

Then, quality :

LP4 with atrac3 ACM is a joke. I rarely heard something worst. SonicStage encodings are totally different.
LP2 is more problematic. SonicStage encoding are not very good : lack of details, easy to ABX on classical instrument for exemple. It sounds a bit synthetic to me.
AMC LP2 encodings are crisp, more detailed and more natural : ABXing is therefore more difficult on non-critical parts. Nevertheless, I noticed that cymbals are suffering with this codec (ACM). SonicStage, in comparison, has less problems.


In my opinion, ATRAC3 is interesting to include in this test for one reason only : Minidiscs performance. If the codec doesn't match with MD device, it's useless. In other words : sonicstage (+ Total Recorder) or nothing.
Title: Pre-Test discussion
Post by: bond on 2003-07-16 11:10:07
i vote for atrac3 too (there is an interest from the ripping community side too)
Title: Pre-Test discussion
Post by: den on 2003-07-16 12:33:18
Quote
In my opinion, ATRAC3 is interesting to include in this test for one reason only : Minidiscs performance. If the codec doesn't match with MD device, it's useless. In other words : sonicstage (+ Total Recorder) or nothing.


Dammit... You really know how to let a guy down gently guru... 

Shame about the ACM codec being not representative. I really had my hopes up there for a while. Oh well, back to TotalRecorder. I thought it was a bit suss when the codec file dates were 2001. 

Thanks for the clarification guru. 

Den.

PS: It would be nice to know what the differences are though...
Title: Pre-Test discussion
Post by: S_O on 2003-07-16 12:39:34
I would like to see atrac3 in the test, too.
Why don´t you use helix-producer as encoder? You can download it for free.
Also it is easy to decode in GraphEdit when RealOne and Gabest RealMedia Splitter is installed:

atracfile.rm -> Real Media Splitter -> Real Audio Decoder -> WAV Dest -> out.wav

Real supports Atrac3 at these bitrates, all only at 44100Hz, Stereo:

66 kbps Lowpass: 12,4kHz (flavor 0)
94 kbps Lowpass: 15,1kHz (flavor 1)
105 kbps Lowpass: 13,7kHz (flavor 2)
132 kbps Lowpass: 16,5kHz (flavor 3)
132 kbps Lowpass: 16,5kHz (surround mode) (flavor 8)
146 kbps Lowpass: 16,5kHz (flavor 4)
146 kbps Lowpass: 16,5kHz (surround mode) (flavor 9)
176 kbps Lowpass: 19,2kHz (flavor 5)
176 kbps Lowpass: 19,2kHz (surround mode) (flavor 10)
264 kbps Lowpass: 22kHz (flavor 6)
264 kbps Lowpass: 22kHz (surround mode) (flavor 11)
352 kbps Lowpass: 22kHz (flavor 7)
352 kbps Lowpass: 22kHz (surround mode) (flavor 12)

This information is from the Codec Information Table by Real. I don´t know what the surround mode means, maybe a different (none?) channel coupling. Also I don´t know if it´s a mistake that the lowpass of 105kbps is lower than the of 94 kbps. "flavor" is the number you tell the encoder (write in the audience-file) on what mode/bitrate to encode.
Title: Pre-Test discussion
Post by: Tripwire on 2003-07-16 12:41:21
When you'll be trying WMA9, be sure to test both Std and Pro. The latter one is very well able to compress 44khz stereo at 128kbit, and it seems to sound better. It's also a PC playable format, neglecting it would be disappointing.
Title: Pre-Test discussion
Post by: JohnV on 2003-07-16 13:25:46
Quote
When you'll be trying WMA9, be sure to test both Std and Pro. The latter one is very well able to compress 44khz stereo at 128kbit, and it seems to sound better. It's also a PC playable format, neglecting it would be disappointing.

I think if both WMA versions will be tested, Atrac must be dropped. Otherwise there are way too many codecs to rate. Imo 5codecs+anchor is already pretty much maximum.

Personally I'd be ready to drop MusePack from this low bitrate test. Imo MPC lowbitrate has no practical meaning to most people or even to MusePack users outside "nice to know"-factor. I don't believe many people will be using MusePack at this low bitrate even if it does well in the test.
Title: Pre-Test discussion
Post by: kl33per on 2003-07-16 13:58:41
Quote
Personally I'd be ready to drop MusePack from this low bitrate test. Imo MPC lowbitrate has no practical meaning to most people or even to MusePack users outside "nice to know"-factor. I don't believe many people will be using MusePack at this low bitrate even if it does well in the test.

My thoughts exactly.

I would also like to see ATRAC3 tested.

Most people around here are quality freaks so I don't see the point in testing the standard WMA9.  Anybody who is after the best quality with WMA9 will use the Pro version, therefore only WMA9 Pro should be included in the test.

I would also like to see QT AAC at 128kbps up against Nero 6 -streaming.
Title: Pre-Test discussion
Post by: dev0 on 2003-07-16 14:01:16
Personally I'd rather include Musepack than ATRAC3, but I can understand that people, who already own a minidisc player, might be interested in the results.

Including both versions of WMA is pointless IMO. Most users won't be using Pro and it does not yet have the hardware- and softwaresupport of the earlier StreamVersion (AFAIK the streamversion is officially called WMA2), so its inclusion would be just as pointless as Musepacks.

dev0
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 16:10:07
Hello, people. Thanks a lot for your help.

About atrac3: Well, so I guess it's decided, atrac3 will be tested.

@Den: I already expected the ACM to be worse, I had it for years, so it can only be deprecated.

About SonicStage? Is it freely available or only comes with Sony gear? In this case, I might have to ask someone to encode-decode the samples for me.


About Musepack: I know the results might not interest much the HA community, but keep in mind they will be published on several boards I participate, at the front page of CD-rw.org and maybe even Slashdot.

Besides, it's known Musepack does well at this bitrate, but I still believe a comparision to other formats will be welcome. Specially since portable support isn't a dream anymore, it seems.


About WMA: Should I test WMA PRO (that has better quality but worse support) or WMA Standard (That has worse quality but much better support)? I know the ideal would be testing both, but IMO testing more than 6 formats, plus anchor, will be unbearable. (And these results will _really_ not interest much people, due to the bias against MS and whatnot)


About the anchor: Do people agree with ff123, about using Blade to create the anchor? (lol)


@kl33per: Nero will be tested against other formats later this year/early next year (maybe). Again, I would like to test it now, but few people would be inclined to test a multitude of samples.


About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?


Well, I guess these are enough issues raised.

Again, thank-you very much.

Regards;

Roberto.
Title: Pre-Test discussion
Post by: djemberay on 2003-07-16 16:18:54
I'd be very interested in seeing how the encoder from Nero 6 stacks up as well. It sounds as though this may not be in the spirit of this particular test, but I thought I'd throw it out there...

-raymond
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-16 17:10:27
Quote
About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?

I suggest to replace BeautySlept - harpsichord sound, but maybe synthetic - by a real instrument recording (as Bachpsichord, or many others).

Nevertheless, I think that this test should include a most common classical sound : orchestra, choral, piano... something familiar. Two at least are on ff123 website : Macabre.wav, and fossile.wav - this last one was used on ff123's 128 kbps test. Fossile is ABXable. So why not this one, as common classical sound, and a harpsichord, as a more problematic classical sample?
Title: Pre-Test discussion
Post by: ErikS on 2003-07-16 17:16:57
Quote
About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?

You mean we have to listen to the same samples once again? I got tired of them already in the first test  Is it because it's a good test suite or because you want to be able to compare results between the tests? Anyway, I probably can't take this test, so I wonder why I even care...
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-16 17:24:53
I forgot this one :
SonicStage can be downloaded ; the software is big (70 MB), and I don't know any link for the moment. Recently, many were dead.

SonicStage produce .omg file : atrac3 + encryption. I can't sent to somebody an .omg file. DRM system of Sony forbid to anyone to read an .omg file that wasn't created on his PC.
Therefore, if someone want to upload you the files, they must be in PCM format (decoded + encoded losslessly). It's difficult for me to do it : I'm limited this summer to a 56K connection.

Keep in mind that we had to cheat with SonicStage in order to play the ATRAC3 file. If the original (.wav) file is still present on the system (same path, same name), this one will be played (and captured by TotalRecorder). It's easy to do a mistake, and to conclude to the superiority of Sony's format (I saw an occurence on a MD dedicated board, where the LP2 encoded was identical to the original).
Title: Pre-Test discussion
Post by: ff123 on 2003-07-16 17:25:44
I vote both for keeping Musepack and for using WMA9pro.

ff123

1. BTW, I verified that the time offset of WMA9pro using dbpoweramp to decode is 0.

2. The filesizes of the downloads will be much larger this time around, one reason is the higher bitrate, the other reason is the proprietary formats.  You'll probably have to flac the wma9 and the atrac3.  Are you going to batch encode the ogg, lame, and other encodeable files as well as batch decode?  That would make things smaller.

3.  On classical, fossiles.wav is a possibility, although Filburt once told me he couldn't stand to listen to it because he said it was recorded so badly (I personally am not annoyed by this clip).  So I would vote to replace LifeShatters with fossiles.wav or macabre.wav, and also second guruboolez's suggestion to replace beautyslept.wav with bachpshichord.wav.
Title: Pre-Test discussion
Post by: ErikS on 2003-07-16 17:26:50
Quote
I suggest to replace BeautySlept - harpsichord sound, but maybe synthetic - by a real instrument recording (as Bachpsichord, or many others).

Just FYI: BeutySlept is one sample where MPC:s PNS screws up. Except a trumpet/horn from the SQAM collection, I haven't heard that as clearly as with this sample, so I think it would be nice to leave it in. Maybe the other "real" harpsichord is as tough though... I wouldn't know. Would you mind doing a quick test, Guruboolez? Could seem unfair to the other codecs to remove a problemsampe fro MPC...?
Title: Pre-Test discussion
Post by: boojum on 2003-07-16 17:28:42
Ah, there's that harpsichord again.  B)  I agree with Guruboolez that there should be some classical sounds in the test.  Choral would be great: voice, as we all know what that sounds like, and the mass of the symphony orchestra.  How about something from Beethoven's 9th?  Freude! 
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-16 17:30:22
One 'problem' with bachpsichord (and with beautyslept too, but at lower degree), is the bitrate reached with --radio profile : near 200 kbps...

P.S. What to you mean by "screw up" ?
Title: Pre-Test discussion
Post by: ErikS on 2003-07-16 17:31:31
Quote
One 'problem' with bachpsichord (and with beautyslept too, but at lower degree), is the bitrate reached with --radio profile : near 200 kbps...

P.S. What to you mean by "screw up" ?

That it's a clear difference with and without PNS active.
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-16 17:36:47
Quote
Quote
One 'problem' with bachpsichord (and with beautyslept too, but at lower degree), is the bitrate reached with --radio profile : near 200 kbps...

P.S. What to you mean by "screw up" ?

That it's a clear difference with and without PNS active.

Is PNS lowering sound quality ? Or is it an exemple of the benefit of Klemm PNS algorithm ?
Title: Pre-Test discussion
Post by: ErikS on 2003-07-16 17:42:39
Quote
Is PNS lowering sound quality ? Or is it an exemple of the benefit of Klemm PNS algorithm ?

With PNS sounds worse than without. For me I should add... But I don't always trust my own ears - will you try it?
Title: Pre-Test discussion
Post by: bond on 2003-07-16 17:46:30
if it is generally accepted that wma9 pro will produce better results than wma9 standard then i will vote for using wma9 pro (although hm i dont really want that the codec is getting pushed if it is really good compared to the others  )
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-16 17:46:56
No problem. But for the moment, I must shut my computer down : thunder is more than menacing...
Title: Pre-Test discussion
Post by: Gecko on 2003-07-16 19:02:46
I'm not in favor of adding any other form of lowpassing other than the codecs use themselves. Most importantly we should be testing real world performance imo. Also expect alot of people who mistake hifi with lots of high frequencies to complain about overall dull sound. It will be a mess to explain this step to all those self appointed experts (especially outside of HA) who will disregard such a testing method. It all depends on what you want this test to be.

Another problem of artificial lowpassing would be readjusting the bitrate of the vbr codecs.

One valid reason for an artificial lowpass might be to reduce ogg Vorbis' hf boosting bug.

I'm not sure about adding mpc. People will extrapolate the results of the 128k test to other bitrates and this might shed a wrong light on mpc. Mpc doesn't suck at this bitrate, but I expect ogg or aac to sound better by quite a margin. Mpc is not intended to be used at 128 and probably noone will, so why bother testing? On the other hand, maybe mpc will present itself as a viable alternative and this is a good chance to find out.

Oh yeah, I think trance and celtic music is definately underrepresented in this test.
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 19:46:28
@Gecko: I think you didn't understand the purpose of the anchor. The anchor is only one file, and it suffers lowpass (or any other process to degrade quality) in order to put things into perspective and to avoid codecs being rated too low.

The samples encoded with the competing codecs suffer no preprocessing like lowpass.

About MPC: From some preliminary tests conduced by Den (and Guruboolez?), I would expect MPC to win this test, or at least come close to the winner. :B

Regards;

Roberto.
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 19:57:57
Quote
I vote both for keeping Musepack and for using WMA9pro.

OK... unless someone comes with good reasons to use WMA std or to sack Musepack, I'll go with your vote.

Quote
1. BTW, I verified that the time offset of WMA9pro using dbpoweramp to decode is 0.


OK. Thanks a lot.

Quote
2. The filesizes of the downloads will be much larger this time around, one reason is the higher bitrate, the other reason is the proprietary formats.  You'll probably have to flac the wma9 and the atrac3.  Are you going to batch encode the ogg, lame, and other encodeable files as well as batch decode?  That would make things smaller.


Yeah, I would also batch process the anchor, now I only need to know if I'll use Blade or Sox for that. >_<  (blade = encode, sox = lowpass)

If we use blade, I would also have to find out if there's any offset introduced when decoding it's encodes with Lame. <hint>

Quote
3.  On classical, fossiles.wav is a possibility, although Filburt once told me he couldn't stand to listen to it because he said it was recorded so badly (I personally am not annoyed by this clip).


So, I think we can use Macabre instead, just to be safe.

Quote
So I would vote to replace LifeShatters with fossiles.wav or macabre.wav, and also second guruboolez's suggestion to replace beautyslept.wav with bachpshichord.wav.


Good. It's settled then.

@ErikS: I'm using (mostly) the same sample suite because it's a good suite (representing several different styles) and because I'm lazy.

I'm thinking about calling for new samples for the 64kbps extension test, but it's still too early to talk about that.
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 20:05:45
Ok, so what's (nearly) settled (unless someone comes with good arguments)

- Atrac3 will be tested
- Musepack will be tested
- WMA PRO will be tested instead ow WMA std
- LifeShatters will be replaced with macabre and beautyslept with bachpshichord

What's not settled:

- Will the anchor use lowpass or Blade? If it uses lowpass, one lowpass for all samples or one for each sample? If it uses Blade, plain 128kbps CBR?

Regards;

Roberto.
Title: Pre-Test discussion
Post by: JohnV on 2003-07-16 20:08:07
Quote
Quote
I vote both for keeping Musepack and for using WMA9pro.

OK... unless someone comes with good reasons to use WMA std or to sack Musepack, I'll go with your vote.

What kind of "voting" that is? 
If this will be solved by voting, you should create a poll...

Anyway, I vote exactly opposite to ff123: sack wma9pro and MPC and keep Atrac. 
Title: Pre-Test discussion
Post by: JohnV on 2003-07-16 20:14:00
Quote
Ok, so what's (nearly) settled (unless someone comes with good arguments)

- Atrac3 will be tested
- Musepack will be tested
- WMA PRO will be tested instead ow WMA std

6 codecs+anchor+original is hell of a job for those who want to do the full test with all samples.
Title: Pre-Test discussion
Post by: ErikS on 2003-07-16 20:23:45
Where can I find this Bachpsichord samle?
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 20:37:43
Quote
sack wma9pro and MPC and keep Atrac. 

Sack WMA and MPC, favouring Atrac??? Heck, Atrac3 loses to other codecs (in features) in nearly every imaginable way (not multi platform (only Windows)), only CBR, no open decoder or encoder, not usable in media players like Winamp or Foobar, not usable in movies besides Real ones (unless you do hacks)...

I don't understand what is going on here. :B

And I agree 6 codecs is nearly too much, but, IMO, most of them aren't "sackeable". I see good reasons for including MPC, but, still, I think a poll would be the way to go.



@ ErikS: Ask Guruboolez
Title: Pre-Test discussion
Post by: ff123 on 2003-07-16 20:45:22
I'll try out blade tonight to see what it will sound like on the test suite and also to see if it has an offset

ff123
Title: Pre-Test discussion
Post by: Garf on 2003-07-16 20:47:29
Too bad Lifeshatters is out ... I found the clip 'interesting'..
Title: Pre-Test discussion
Post by: bond on 2003-07-16 20:53:33
Quote
Atrac3 ... not usable in media players like Winamp

no legal way

Quote
not usable in movies besides Real ones

should be possible with gabest's filter, i think...
Title: Pre-Test discussion
Post by: Sebastian Mares on 2003-07-16 21:04:09
 I would like Atrac3 comparision!
Title: Pre-Test discussion
Post by: ScorLibran on 2003-07-16 21:14:48
Is the testing something a newbie like myself could help with?  I actually tried to get in on the bitrate analysis phase, and had everything I needed to do it downloaded and ready to go.  Then I got really busy re-encoding my collection and didn't have the time or HD space to follow through.

Now that I'm finishing up all my encoding, I'd like to take part in the ABX phase, unless a group has already been identified to do the testing.

Title: Pre-Test discussion
Post by: Dibrom on 2003-07-16 21:31:39
I've only briefly skimmed this thread, so I could be missing something, but here's my opinion:  I would personally like to see MPC included in this test, especially if this test is going to be geared more towards the users of this board than perhaps the general populace (and let's face it, it probably will be even if designed otherwise).  There are a lot of MPC users here, and there has always been an open ended question about it's low bitrate performance.  It'd be nice to see it put to the test once again.

I think the same argument above probably would work against atrac.  While it is fairly widely used overall, I don't think it is particularly widely used on this board.  Furthermore, it'd almost seem like a purely academic excercise to include it since I figure that it doesn't have a good chance of doing very well at all considering the more advanced and more highly tuned codecs it will be competing with.  It could perhaps provide some useful results to point people to, but I believe most people who are serious about audio compression (which are the people who will appreciate these results the most) already realize that the alternatives are better on just about every front.

As for WMA vs WMA Pro, I'm not really sure.  If we test WMA Pro, it has to be made very clear that there is a distinction between this and normal WMA, and that WMA Pro will not work on most (any?) portable devices.  I think maybe it'd be best to actually test both, but that might push the codec count too high.

Personally, I'd favor dropping Atrac first out of those available.

Perhaps it'd be best to really define the target audience of the test before deciding which to include and which not to.  Are we testing for the masses here?  If so, then MPC doesn't make sense and WMA Pro probably doesn't either.  Atrac would though.  If we are testing for the more enthusiastic and technically minded users (like the people here on HA  -- again, those who I think would appreciate these results the most also), then WMA Pro makes sense over normal WMA and MPC makes more sense than Atrac I think.  If we're trying to strike a balance, then maybe it'd be best to go with my original suggestion:  WMA/WMA Pro and MPC instead of Atrac.

Just my $0.02...
Title: Pre-Test discussion
Post by: bond on 2003-07-16 21:54:20
imho the audience will be all people who transcode audio cause it will be the first independent representative codec comparison @128 (as i know)

on any discussion on the net (professional or not) about codec quality this test could be quoted (which makes it very usefull)

so including the most used codecs will be necessary to keep the test as much usefull for as most people as possible (not only for some professional super-minds  )
Title: Pre-Test discussion
Post by: Gecko on 2003-07-16 22:00:37
Quote
@Gecko: I think you didn't understand the purpose of the anchor. The anchor is only one file, and it suffers lowpass (or any other process to degrade quality) in order to put things into perspective and to avoid codecs being rated too low.
The samples encoded with the competing codecs suffer no preprocessing like lowpass.

Ok, thx for clearing that up. I guess this was confusing me. Heh, actually, it still is:
Quote
b) JohnV votes for one lowpass for each sample, else some samples might sound better lowpassed than their original versions. ff123 votes for one fixed lowpass for all samples, to put things into perspective.
Title: Pre-Test discussion
Post by: S_O on 2003-07-16 23:20:57
Quote
Heck, Atrac3 loses to other codecs (in features) in nearly every imaginable way (not multi platform (only Windows)), only CBR, no open decoder or encoder, not usable in media players like Winamp or Foobar, not usable in movies besides Real ones (unless you do hacks)...

This is not completly right:
-not multi-platform:
Everybody is allowed to create a en/decoder for every platform. Real will compile the libraries for every platform they said. This is part of the HelixCommuinty-project.
-Playable in every DirectShow supporting player thanks to Gabest Filters
-In movies it is useable, also thanks Gabest and it´s not a hack, Real has nothing against using their codecs in other containers. By the way, the new RealVideo 9 EHQ Video Codec has won in several tests (against XviD, WMV9, DivX Kauehi), I would like to know how the audio quality is. In the last month I got a new picture of Real, not the evil-format anymore.

And for wma9pro:
-Nearly no player, because no DirectShow support (only WMP7+???)
-Only windows
-No open-source en/decoder
-Not useable in any movies except wmv

For encoding I would use the Helix-Producer, because it´s free for everybody, not too big and creates unencrypted files. Also most people know Real (and maybe Helix), but who knows SonicStage and omg-files? If you ask Karl Lillevold (a Real developer) over Doom9 board, I think he will tell you exactly what kind/version etc. of Atrac3 they´ve used.

Quote
no legal way

I think it should be legal to create a winamp-plug-in if it´s legal to create DirectShow-Filters. You are not allowed to rip the codecs from RealOne, so the player must be installed, or you must use the binaries from the helix-project. But nobody has done a plug-in yet.
Title: Pre-Test discussion
Post by: bond on 2003-07-16 23:38:18
Quote
By the way, the new RealVideo 9 EHQ Video Codec has won in several tests (against XviD, WMV9, DivX Kauehi), I would like to know how the audio quality is.

Me too
although i dont know any representative tests which prooves ehq to be better than xvid... (ok this is far too of topic)

Quote
I think it should be legal to create a winamp-plug-in if it´s legal to create DirectShow-Filters.

there is a plugin, but it was always said that it was illegal (although it needs realone installed like gabest's filter) dont know if the filter are legal or just tolerated (till now)
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-16 23:52:39
Here I come again.

@Dibrom: about the target audience, I think it'll be mixed. We'll have the main audience here at HA for sure, but we must also consider the <cough> guys at Slashdot that will also know about the results if things go as I hope.

Another option would be creating a test for widespread advertizing, that would appeal to the masses, and another one for HA mostly, that would focus on the codecs used at HA. This one I'll start next wednesday would be the "masses" one, because it already attracted quite some attention due to the AAC test. (including attention from Dolby labs and Apple  )

Anyone with thoughts on this idea? (of separating the test in two)

@Gecko: What I mean that is that JohnV votes for a different lowpass for each one of the 12 samples. E.G, ATrain gets a lowpass of 11kHz, and Waiting gets 9.5kHz. ff123 would vote for ATrain and Waiting (and everything else) using a fixed lowpass.

@Bond: There is a Real Audio plugin for Winamp called InnoReal. But it's "illegal" for some reason (Real demanded that Nullsoft remove it from their plugin database some years ago). I think that it's because they don't want people decoding their streams (what's stoopid, IMO).
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-17 00:01:24
Quote
This is not completly right:
-not multi-platform:
Everybody is allowed to create a en/decoder for every platform. Real will compile the libraries for every platform they said. This is part of the HelixCommuinty-project.
-Playable in every DirectShow supporting player thanks to Gabest Filters
-In movies it is useable, also thanks Gabest and it´s not a hack, Real has nothing against using their codecs in other containers. By the way, the new RealVideo 9 EHQ Video Codec has won in several tests (against XviD, WMV9, DivX Kauehi), I would like to know how the audio quality is. In the last month I got a new picture of Real, not the evil-format anymore.

Hrm, I guess we have a problem here.

We need to define if, for our purposes, Atrac3 = RA or = Sony's version. That matters in the aspect that only one of these is usable in movies and playable in several environments, and only one is playable on hardware players. If both encoders are using the same encoding libraries, we can go ahead. But if they aren't, we'll have to chose between the multipurpose one of the hardware-playable one.

I'll mail Karl asking for clarification.

Regards;

Roberto.
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-17 00:20:57
It's me again.

I just want to inform you that I'll travel to the middle of nowhere (for those that know about Brazilian geography: Mato Grosso) in some minutes, and I'm not sure yet if I'll be able to access the internet there. So, if I don't reply to this thread until next tuesday, fear not! - I'm still alive, and the test hopefully starts on time.

Best regards, and thank-you very much for everyone that helped so far.

Roberto.
Title: Pre-Test discussion
Post by: bond on 2003-07-17 00:27:46
Quote
And for wma9pro:
-Nearly no player, because no DirectShow support (only WMP7+???)
-Not useable in any movies except wmv

nope it is possible to mux wma9 pro into other containers (i tested it with gabest matroskamuxer (.mkv) and with m$ avi mux filter (.avi)) and to playback it with a m$ filter called "wmaudio decoder dmo" in directshow players

Quote
Anyone with thoughts on this idea? (of separating the test in two)

hm, so you would do 2 different tests (and in use both aac, vorbis and lame)? or one big "mass test" and the second one just a small test with some pros?

which codecs would participate in the mass test? aac, vorbis, lame, wma9 (std or pro?) [and realaudio or sony's atrac3 or real's cook  ]

Quote
@Bond: There is a Real Audio plugin for Winamp called InnoReal. But it's "illegal" for some reason (Real demanded that Nullsoft remove it from their plugin database some years ago). I think that it's because they don't want people decoding their streams (what's stoopid, IMO).

yup that was the plugin i meant, and that also was the reason why i am sceptical if real really "likes" gabest filter (although the should be wise enough to do so)...

edit: nice holiday rjamorim
Title: Pre-Test discussion
Post by: JohnV on 2003-07-17 00:36:57
Quote
@Gecko: What I mean that is that JohnV votes for a different lowpass for each one of the 12 samples. E.G, ATrain gets a lowpass of 11kHz, and Waiting gets 9.5kHz. ff123 would vote for ATrain and Waiting (and everything else) using a fixed lowpass.

Actually, I meant that the results after lowpassing the anchor should be checked, because if using fixed lowpass, on some samples "the effect" can be much smaller than on some other samples. If there's a case where the chosen lowpass is clearly not enough to serve its purpose, it should be adjusted..
(I didn't give any values)
Title: Pre-Test discussion
Post by: den on 2003-07-17 01:34:58
@Roberto and others

My understanding is that the Real version of ATRAC3 is not representative of the Sony ATRAC3 used in OpenMG and Sonicstage. For one, I don't think the same low pass is used.

Your best bet will be to either download a copy of SonicStage yourself, or get someone else to record the wavs on your behalf. If you go to www.minidisc.org and visit the various forums within, and then go to the NetMD sections, there is usually a sticky with links to downloads. Some of these are legal links, through Sony retailers, and some of them are warez.    You've just got to weave your way around all the 15 year olds posting "MD rulz!" 

Despite the plethora of samples, I feel that MPC and ATRAC3 are worth including. Many people on this board are not using mpc at 128 kbits, but how many have actually tried it?    Not many I suspect. As you mentioned, the few tests I've done have shown that mpc can be very competitive at 128 kbit... and I'm not alone with this impression. 

Can't wait for this test!

Den.
Title: Pre-Test discussion
Post by: den on 2003-07-17 01:43:12
Sorry to double up, but there are also ways to play ATRAC3 .omg files in Winamp 2.81, or 2.95 (lite only) using a video plug in. The details are in one of the threads in the www.minidisc.org forums, but in essence you still need to have OpenMG or Sonicstage installed, and then you register the omgdec.ax file in Windows.

Once you have done that, you can either play the .omg file in Windows Media Player as is or Winamp with a plug in. Because it is still using the Sony omgdec file, I think you will find that the encryption restrictions still exist, ie you must have the original file present somewhere on you system, or have ripped it from an audio CD, but I haven't tested this to be sure.

If you already have Sonicstage/Open MG installed though, I can't see much benefit in doing this. My attempts to get Winamp to write a wav file to disk while playing an .omg file failed dismally. 

Den.
Title: Pre-Test discussion
Post by: den on 2003-07-17 04:02:16
Me again... Last time this time, really... 

I just took Kraftwerks - The Robots and encoded it in Open MG (variant of Sonicstage), Real Producer and the crappy ATRAC3 acm codec. Encoded all three at 132 kbits using the standard "profile" in each case.

The ACM codec has a clear hi freq cut off at 17500 Hz.
Real Producer has a clear hi freq cut off at 16500 Hz.
Open MG also cuts off at 17500 Hz (edit: I was fooled before by OpenMG playing the wav instead of the omg file...>_<).

My personal vote is for Soundstage/OpenMG, but I have a vested interest. 
It is the one being pushed by Sony to replace LAME mp3 for portables, and is now being targetted by Apple and their I-pods (AAC), but it all depends what your target audience is for this test comparison.  B)

Den. 

Edit: Contained false info before regarding high freq cut off with Open MG.
Title: Pre-Test discussion
Post by: magic75 on 2003-07-17 07:27:03
Quote
Quote
I vote both for keeping Musepack and for using WMA9pro.

OK... unless someone comes with good reasons to use WMA std or to sack Musepack, I'll go with your vote.

One reason from a newbie...
WMA std is supported on a lot of portables and WMApro is not?
Wasn't the use of Atrac in Sonys portables the initial reason for including it?
My personal opinion is that bitrates around 128 kbps is used typically on portables rather than played on a computer, and therefore formats supported by portables should be prioritized. That would mean dropping MPC and WMA9pro, but that's just me...
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-17 11:24:30
Short reply :

- Sorry, I haven't test BeautySlept with and without PNS encoding with musepack.
- Bachpsichord sample is available here :
http://membres.lycos.fr/guruboolez/AUDIO/samples/ (http://membres.lycos.fr/guruboolez/AUDIO/samples/)
- WMApro encodings are readable on foobar2000.
- I can confirm that winamp playback of encrypted and protected .omg files can't be possible on another computer. Another solution is to install a directshow filter for WMP, but the problem remain. Before playback, the plug/filter check the DRM rights.
Title: Pre-Test discussion
Post by: ErikS on 2003-07-17 12:18:51
Quote
Short reply :

- Sorry, I haven't test BeautySlept with and without PNS encoding with musepack.
- Bachpsichord sample is available here :
http://membres.lycos.fr/guruboolez/AUDIO/samples/ (http://membres.lycos.fr/guruboolez/AUDIO/samples/)

I did a quick test, and bachpsichord has the same problem with but not as clear.

ABX results
Beauty slept: 12/12, would give it a score somewhere around 3.5 on the 5 point scale.

Bachpsichord: 10/13, score around 4.0.
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-17 19:57:32
I compared the sample Beautyslept at radio profile, and the consequences of the PNS tool (mpc 1.14). I couldn't differenciate the two encodings.

Code: [Select]
1R = C:\Temp\Pns\Sample03\BeautySlept_Q4xlev.wav
2R = C:\Temp\Pns\Sample03\BeautySlept_Q4xlevNOpns.wav

ABX Results:
Original vs C:\Temp\Pns\Sample03\BeautySlept_Q4xlev.wav
   8 out of 8, pval = 0.004
Original vs C:\Temp\Pns\Sample03\BeautySlept_Q4xlevNOpns.wav
   8 out of 8, pval = 0.004
C:\Temp\Pns\Sample03\BeautySlept_Q4xlev.wav vs C:\Temp\Pns\Sample03\BeautySlept_Q4xlevNOpns.wav
   5 out of 12, pval = 0.806
[CODE]


Tried with Bachpischord :

[CODE]
1R File: C:\Temp\Pns\Sample03\Bachpsichord_Q4xlev.wav
1R Rating: 2.0
1R Comment: very grainy
---------------------------------------
2L File: C:\Temp\Pns\Sample03\Bachpsichord_Q4xlevNOpns.wav
2L Rating: 3.0
2L Comment:
---------------------------------------
ABX Results:
C:\Temp\Pns\Sample03\Bachpsichord_Q4xlev.wav vs C:\Temp\Pns\Sample03\Bachpsichord_Q4xlevNOpns.wav
   14 out of 20, pval = 0.058


PNS encodings is worst in this case : sound is grainy, and more unpleasant. I didn't noticed it with BeautySlept. But I must say that I had some difficulties to hear difference between original and -radio (notation : 4.4 and 4.6), less pronounced to my ears than with Bachpsichord (2 and 3/5).
Title: Pre-Test discussion
Post by: Tripwire on 2003-07-17 20:50:15
The reason why I'm asking to include WMA9Pro is that I expect MS to drop WMA9Std in a not so far future in favor of their apparently better Pro version of the codec, for sound quality reasons and also because the backwards compatibility of WMA9Std hindering any further tuning. At least in Q<=25 VBR modes it produces 48-80kbit results (depending on input) that sound pretty OK. I wouldn't wonder if in a year or two, thanks to growing DSP performance in portables, that the Windows Media logo requirements will ask for two channel WMA9Pro support.

Or maybe not. Who knows.
Title: Pre-Test discussion
Post by: amano on 2003-07-18 03:48:27
hmm. maybe it sounds like a complete newbie question.

but...
how do I produce a wma 9 pro file????
there is no option in wmp9. or is "windows media audio (variable bitrate)" the one I am seeking?
Title: Pre-Test discussion
Post by: karl_lillevold on 2003-07-18 06:06:06
Quote
We need to define if, for our purposes, Atrac3 = RA or = Sony's version. That matters in the aspect that only one of these is usable in movies and playable in several environments, and only one is playable on hardware players. If both encoders are using the same encoding libraries, we can go ahead. But if they aren't, we'll have to chose between the multipurpose one of the hardware-playable one.

I'll mail Karl asking for clarification.

Hello everybody, so nice to see many familiar names from over on Doom9.

I enjoyed participating in the AAC test and reading the lively discussion, and now I am looking forward to the next test. I think it it is a good idea to include ATRAC3. In many listeners' opinion it does sound better than MP3 at 128 kbps.

RealAudio ATRAC3 is the same as Sony's reference ATRAC3. However, we do not know if this has been updated by Sony since our licensing of ATRAC3. The reason there may be differences is that the reference code does not govern all the settings at various bitrates, which I have been told, may affect for instance the cut-off frequency. My suggestion is to use the ATRAC3 codec that most conveniently enables test participants to create their WAV samples, which would probably be the ACM codec, but I do believe the ATRAC3 flavors are close enough that any choice you are most comfortable with will work out well.

Most of my RV9-EHQ (http://forum.doom9.org/showthread.php?s=&threadid=40392) video encodes use RA8 cook at 96 kbps, and I my knowledge and experience in the audio space could have been better.. I did recently put on some high quality head phones and it was nice to hear that my ears could still detect which clips were coded at 128 kbps with AAC  So don't get too upset if anything I say here is not correct. If you need more information I will try to help out.
Title: Pre-Test discussion
Post by: ff123 on 2003-07-18 06:18:08
Quote
hmm. maybe it sounds like a complete newbie question.

but...
how do I produce a wma 9 pro file????
there is no option in wmp9. or is "windows media audio (variable bitrate)" the one I am seeking?

Using Windows Media Encoder:

New Session -> Convert a File -> Choose input and output files -> File Archive -> Choose an Audio quality setting (which you will change later)

Go to the Properties tab -> Compression tab -> Edit

Choose WMA Pro as the codec -> Choose bitrate mode (bitrate VBR is the two-pass vbr mode)

Go to the tab next to "General" (mine says 777, standing for the bitrate)

Choose 128 kbps, 44 kHz, 2 channel 24 bit VBR.  The tab will change to 135 (128 kbit/s  audio bitrate plus 7 kbit/s script overhead)

Back in the Compression tab, hit "Apply"

Hit the "Start Encoding" button at the top

This is not very user friendly, is it?  But it seems to be the only way to encode two-pass vbr files, since dbpoweramp doesn't appear to offer this mode.  dBpoweramp is the most convenient way to decode, however, since WME doesn't offer this capability.

ff123
Title: Pre-Test discussion
Post by: amano on 2003-07-18 15:04:59
ok. thanks. didn't use the wme ever. so I thought, this option would be integrated in wmp 9.

so, now when I know, how to choose the pro format, I vote to use wma standard in the test, because I guess 99% of the wma tracks out there in the wild will be standard ones, ripped and encoded with wma 8 or 9.

the wme approach seems to be only used by sound pro's that wouldn't use the pro codec with other than experiamental reasons.
and as pointed above, @ 128 kbps hardware compatibility should be one of several reasons.
so drop wma 9 pro and use wma std. just my 2 cents.
Title: Pre-Test discussion
Post by: Nic on 2003-07-18 15:37:14
The OpenMG software does come with a directshow filter for decoding the ATRAC3 omg files it makes. In graphedit you could easily make WAV files for the ATRAC3 encoded files. Although getting the bitrate to be 128kbps might be a problem.

-Nic
Title: Pre-Test discussion
Post by: den on 2003-07-19 00:27:58
Quote
The OpenMG software does come with a directshow filter for decoding the ATRAC3 omg files it makes. In graphedit you could easily make WAV files for the ATRAC3 encoded files. Although getting the bitrate to be 128kbps might be a problem.


I didn't realise there was a directshow filter in there somewhere, but I haven't exactly gone looking for it either. 

I know some people are a little sensitive to the 128 kbit vs 132 kbit situation, but there is no everyday use of ATRAC at 128 kbit, so trying to make it so is pointless.

Surely the additional 4 kbit is not that big a deal for this test.

And hey, if ATRAC3 turns out to be the best, those who love the other formats can always use the extra 4 kbits as an excuse... 

Den
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-19 01:54:59
Quote
Quote
Anyone with thoughts on this idea? (of separating the test in two)

hm, so you would do 2 different tests (and in use both aac, vorbis and lame)? or one big "mass test" and the second one just a small test with some pros?

which codecs would participate in the mass test? aac, vorbis, lame, wma9 (std or pro?) [and realaudio or sony's atrac3 or real's cook  ]

Well, my idea would be to do two separate (and independent)  tests, one to appeal the masses with codecs of mass consumption, and other to appeal HA with codecs mostly used here.

The Masses test would compare QuickTime, Lame, Sony´s Atrac3, WMA STD and Vorbis

The HA test would compare Nero AAC, Lame, Musepack, WMA PRO and Vorbis.


What do you guys think of this idea?

Regards;

Roberto.


(Damn lousy cybercoffee, connection speed is around 2kBps)
Title: Pre-Test discussion
Post by: amano on 2003-07-19 02:33:05
sounds great to me. and by having mp3 and vorbis in both tests, we could perhaps draw some conclusions (ok, I know, not really valid ones) between both tests.

I vote, to perform it as suggested, but to begin with the test for the masses.



EDIT:
upps, we forgot our anchor.

new proposal masses: Lame, QuickTime, WMA STD, Vorbis and Blade as anchor

new HA: ATRAC3, Nero AAC, Musepack, WMA PRO and Blade as anchor.
Title: Pre-Test discussion
Post by: den on 2003-07-19 06:05:32
Quote
The Masses test would compare QuickTime, Lame, Sony´s Atrac3, WMA STD and Vorbis

The HA test would compare Nero AAC, Lame, Musepack, WMA PRO and Vorbis.


I agree in principle, but why QT in one and Nero in the other?    I must be missing something.

Do we really need an anchor?

Den.
Title: Pre-Test discussion
Post by: bond on 2003-07-19 11:16:05
Quote
The Masses test would compare QuickTime, Lame, Sony´s Atrac3, WMA STD and Vorbis
The HA test would compare Nero AAC, Lame, Musepack, WMA PRO and Vorbis.

sounds ok,
but i would use quicktime in both tests (it won in a fair match against nero, and nero will get its chance again in the 64kbps test)

or you prepare the two tests in a way so that they can also be compared, for example by using the same test samples, so that it would be like one big test splitted into two parts...


but, i personally would only need a test which includes the best "versions" each codec has to offer (at 128kbps of course):
lame
quicktime
wma pro
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-19 15:49:54
Quote
I agree in principle, but why QT in one and Nero in the other?    I must be missing something.

Because I'm facing lots of pressure from people wanting a listening test with Nero AAC since it didn't do so well. See it as a revenge attempt  (And no, I'm not talking about Ivan)

Besides, if I don't fuck up, the results of both tests would be comparable among themselves. You wouldn't be able to draw definitive conclusions (that would require same set of samples, same listener group, same listening conditions for each listener...), but you could at least have an idea.

Quote
Do we really need an anchor?


IMO we do, as I said before, to put things into perspective.

Low bitrate tests are even worse, they need two anchors. :B


@ff123: Have you found out if Blade is suitable for anchor?

Regards;

Roberto.


Edit: Yes, the mass-test would be the first.
Title: Pre-Test discussion
Post by: ff123 on 2003-07-19 17:11:12
Quote
@ff123: Have you found out if Blade is suitable for anchor?

I ran through some comparisons, and here's how I would find things:

Code: [Select]
                   10kHz                   blade
41_30sec           easy                    very easy
atrain             moderate                moderate
beautyslept        easy                    easy
blackwater         moderate to difficult   easy
death2             moderate to difficult   very easy
flooressence       moderate to difficult   very easy
layla              moderate                easy
lifeshatters       moderate                very easy


I think blade would make a better anchor than 10 kHz, even though some samples may be harder to find problems in than others.

Note:  Probably some people would find 10 kHz easy to detect on all samples.  Some people might find it very hard to tell the difference on some.  Maybe somebody else could ring in and try out a few?

ff123
Title: Pre-Test discussion
Post by: rpop on 2003-07-19 17:32:09
So the masses test is the one scheduled to start Wednesday? How much later will the HA test start?
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-19 19:55:33
Quote
So the masses test is the one scheduled to start Wednesday?

Yes. Unless someone comes with a very good reason not to split the test in two.

(BTW, big thanks to Dibrom for pointing out the issue of the target audience)

Quote
How much later will the HA test start?


Much later. Before that, I plan to test codecs at 64kbps, and then do some vocodec tests with the help from JM Valin. Expect the HA test to happen somewhere around late this year, early next year (maybe).

With the extension test, it's enough 128kbps tests for now. :B

Regards;

Roberto.
Title: Pre-Test discussion
Post by: ff123 on 2003-07-19 22:11:24
Roberto,

Before you start the test, could you upload a test archive, so that I can doublecheck volume level and time alignment?

ff123
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-19 23:34:39
Quote
The Masses test would compare QuickTime, Lame, Sony´s Atrac3, WMA STD and Vorbis

The HA test would compare Nero AAC, Lame, Musepack, WMA PRO and Vorbis.

Just an apprehension : do you really consider QuickTime (and his codec) as a solution for the masses ? I know there's iTune [span style='font-size:8pt;line-height:100%'](really same encoder : not sure)[/span], but it's for the next 6 months limited to MAC world (not really a mass-system). On the other side, QuickTime doesn't accept CDRipping or even batch encoding. And I prefer forget mass-tagging issues here... Mass or experts, QT isn't a useful tool. Last but not least, QT isn't intuitive for most people (save as... dosn't work : you need to export, and of course to know it).

Ahead encoder is so simple to use in comparison. And so many people have Nero installed on their computer. And popularity will certainly increase with the HE-AAC encoder, and its impressive quality at ~64 kbps.


If you really want to put both encoders on two different tests, I find more coherent to keep Nero VBR for the mass-test, and QT CBR for the HA one.
Title: Pre-Test discussion
Post by: bond on 2003-07-19 23:39:24
1) plz dont forget that the tests are about sound quality!!!!
and not about usability of the encoding tool or something else (which has nothing to do with the purpose mentioned above)...

2) codecs which were judged worse than another in a fair, scientific test (like quicktime - nero) are out at the tested bitrate (no need for a revenge until there are serious changes on the encoder side)

3) first of all everybody should ask himself:
"hey, what's the sense of a codec comparison @128kbps?"

-> people want to know which codec gives the best quality output at that bitrate

-> only the best codecs should be compared
-> quicktime, lame, vorbis, wma9 pro
(if you want to add additionally atrac3 or musepack or any other codec isnt really important to reach the goal of the comparison)

sorry to hear that rjamorim plans this (imho most) important comparison very late this year (or even next year) 
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-19 23:51:25
Quote
plz dont forget that the tests are about sound quality!!!!

and not about usability or something else (which has nothing to do with the purpose mentioned above)...

Indeed. But in my opinion, the difference between mass and 'expert' presuppose some ergonomy consideration.
Maybe the idea wasn't well explained :
- a PC only test (= HA test)
- a portable-use test (= mass test)

Even here, I have some reticences. I can't imagine someone filling an iPod with QT installed on his PC. Just try to encode one album only : it's a real pain.
A listening test should be useful, especially when practical purpose is in mind (call it portable-test or mass-test, it doesn't change anything).

When Doom9 performed nice codec comparison, he doesn't maximise perceptual quality with impossible avisyth script, and he's not using H264 encoder family - better theorical quality, but so slooooow... He have quality in mind, but doesn't forget to be in harmony with his readers behaviour and possibility
Title: Pre-Test discussion
Post by: bond on 2003-07-20 00:03:54
ups crosspost

added point 2 and 3 in my last post...


ok, if you call it "portable-test" than the purpose of the test changed from "quality only" to "usability and quality", in this case i would add, like proposed from guruboolez, nero to the "mass-portable test" instead of quicktime...


am i the only one who thinks that the "ha-(only the best)-test" is more important and should be done first?
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-20 00:17:40
Hello.

@Guruboolez: I understand your concern about QT not being really usable. But it has the best quality, it won the AAC test, and I'm not willing to fathom the criticism I expect will come from all sides about "WTF, QuickTime won the AAC test but you used Nero on the extension test though? Are you working for Ahead now, you bastard?".

You see, doom9 is a good example, but what is tying my hands now is the AAC test result.

@Bond: IMO, the HA test can't be conduced now. Even though it might be more important, still it would be wasting all the attention the AAC@128kbps test gathered.I plan to keet the HA test more "low profile", and not announce it everywhere else, at most at CD-rw.org

And no, this is a quality test. Even though usability is cool, it's an utterly subjective concept, and I want to keep these tests objective.

And, of course, I can be convinced to hurry up the HA test a little.


@ff123: I'll contact you as soon as I have something ready (probably this tuesday, when I return home)
Have you found out the Blade offset when decoded by Lame?

Thank-you all.

Regards;

Roberto.
Title: Pre-Test discussion
Post by: bond on 2003-07-20 00:37:49
Quote
"WTF, QuickTime won the AAC test but you used Nero on the extension test though? Are you working for Ahead now, you bastard?"



Quote
IMO, the HA test can't be conduced now. Even though it might be more important, still it would be wasting all the attention the AAC@128kbps test gathered.I plan to keet the HA test more "low profile", and not announce it everywhere else, at most at CD-rw.org

i dont think that it will waste attention
-> the only "big" difference between the two tests is wma9 pro and std
-> the most important purpose for using an audio codec is storing music clips on the hd (portables are much less important imho compared to storing on hd (especially @128) and i dont want to know how many people from the masses reencode their stuff to lower bitrates anyways before using it in a portable)...

-> so if wma9 std will be replaced by the pro version, this will not lead to a loss in attention imo even the dumbest user will think: "hey, of course the pro version is better than the standard version and rjamorim also used the pro version, so there can be no doubt that its better, lets drop wma9 std @128"

-> i dont think that it will be a fault to let the pro version out on the masses (and the wma9 std codec will be important in the 64kbps test anyways (especially for portables))

Quote
And, of course, I can be convinced to hurry up the HA test a little.

would be really great
Title: Pre-Test discussion
Post by: guruboolez on 2003-07-20 01:28:53
Quote
@Guruboolez: I understand your concern about QT not being really usable. But it has the best quality, it won the AAC test, and I'm not willing to fathom the criticism I expect will come from all sides about "WTF, QuickTime won the AAC test but you used Nero on the extension test though? Are you working for Ahead now, you bastard?".

What I tried to say is : if you really want to include Nero encoding in one of both test, put it in a "mass" or "popular" part.
I really want to see QT@128 compete with other formats. It was designed as the winner, confirming the positive impressions I had before the public test, and I felt again during this test. There are no reason for me to eliminate QuickTime. On contrary : I have more difficulties to conceive good reasons for giving to Nero encoder a second chance - especially on a dedicate "HA test", - designed to be the high-quality-and-quality-only test. I like Ivan's work ; I know that Nero's codec was handicaped by a forced CBR encoding during test ; I'm aware that some progress were made since first test... but without public testing, proving its superiority on QT 6.3 on most samples, introducing Ahead in the next test is for me something really odd.
Title: Pre-Test discussion
Post by: ff123 on 2003-07-22 04:20:45
Quote
@ff123: I'll contact you as soon as I have something ready (probably this tuesday, when I return home)
Have you found out the Blade offset when decoded by Lame?

Yes, the offset is 0

ff123
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-23 23:16:38
Quote
Yes, the offset is 0

ff123

Thanks



OK, so here's the deal:

-Sonicstage almost fried my Windows installation (and I used to think QuickTime is buggy). Also, I couldn't find out if I should rather use the SonicStage encoder or the Real Producer. As I said before, they are targeted at different purposes, and their encodings differ. I couldn't chose the portable one or the multipurpose one, and noone seemed to help in that decision.

Bottom line: Atrac3 is out.

-Given that Atrac3 is out, it's being replaced with Musepack and I won't split the tests.

-WMA standard won't be tested because, knowing Microsoft, it'll be quickly ditched and replaced with it's superior brother. So, only WMA pro wil be tested. Lack of hardware support isn't an argument, if that was an argument, MPC and Vorbis would be out too. I can expect that, in some months, there'll be more hardware supporting WMA Pro than Vorbis, or AAC.

-What about Nero? I didn't think about it yet, so don't ask. For the time being, it won't be tested (at least by me)

-most important reason not to split tests (and I quote Guruboolez)
"Test is boring. Two tests will be a real pain."

Test starts in about one hour, if all goes right while uploading.

Best regards;

Moi.
Title: Pre-Test discussion
Post by: bond on 2003-07-23 23:27:24
Quote
Atrac3 is out ... WMA standard won't be tested ... not to split tests

good news,
(i guess we will see wma9 std and atrac/realaudio again in the 64kbps test to come anyway...)

Quote
What about Nero? For the time being, it won't be tested

so quicktime is used instead? hm, i think so 

Quote
Test starts in about one hour, if all goes right while uploading.

gimme, gimme 
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-24 00:32:02
Quote
gimme, gimme 

Of course.

http://rarewares.hydrogenaudio.org/test/12...esentation.html (http://rarewares.hydrogenaudio.org/test/128extension/presentation.html)
Title: Pre-Test discussion
Post by: den on 2003-07-24 01:34:17
Good stuff Roberto. I think you have made the right decisions.  B)

Quote
Sonicstage almost fried my Windows installation (and I used to think QuickTime is buggy).


I also had a near death experience with Sonicstage  , hence my continuing use of OpenMG Jukebox, which has the same internals as Sonicstage, but uses a slightly less funky/sexy interface, and a more reliable install routine.

Welcome to the world of NetMD, Sony style. :x

Looking forward to this test, and will get my listening done over the weekend...

Den.
Title: Pre-Test discussion
Post by: rjamorim on 2003-07-24 02:58:03
Quote
Good stuff Roberto. I think you have made the right decisions.  B)

Thanks

Quote
I also had a near death experience with Sonicstage   , hence my continuing use of OpenMG Jukebox, which has the same internals as Sonicstage, but uses a slightly less funky/sexy interface, and a more reliable install routine.

Welcome to the world of NetMD, Sony style. :x


True.

maybe I'll test it one of these days. In this occasion, I would probably create a Win98 partition with VirtualPC and install SonicStage inside, and later delete it. But it sure as hell has a very low priority now.

Quote
Looking forward to this test, and will get my listening done over the weekend...


Great, looking forward to your test results.

Regards;

Roberto.