Skip to main content ↑ ↓
Topic: Pre-Test discussion (Read 33123 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Pre-Test discussion

Hello.

As most of you already know, I'm planning an extension test for the AAC@128kbps test. It'll be the AAC winner (QuickTime) versus Musepack, Vorbis, Lame ABR, WMAv9 and maybe Atrac3 (more on that later). The test should start (if everything goes right) in a week (the 23rd)

There are some subjects where I'm still not sure how to proceed, so I would like to rely on the forum users' knowledge:

1 - Should I include Atrac3 (used on Sony MiniDiscs and Real Audio at mid-high bitrates) in this test? Is there anyone out there interested in this format? (besides Den ;) )

2 - I plan to use anchors in this test. For those that don't know, anchors are samples that suffer no processing from the original apart from lowpass. They are used to put the results into perspective and avoid participants rating the encoded samples too high and/or too low. The questions that arise are:
a) 2 anchors (high and low) are needed, or only one anchor is enough? (IMO, two anchors are more useful in low bitrate tests, and don't make much sense in high bitrate ones)
b) JohnV votes for one lowpass for each sample, else some samples might sound better lowpassed than their original versions. ff123 votes for one fixed lowpass for all samples, to put things into perspective. What's your personal opinion?
c) Anyone has any idea of good lowpass(es) for the test? It's a difficult choice since low lowpasses might sound too unpleasant to the fellas with good hearing and pro gear, but high lowpasses might sound too transparent to the fellas with not-so-good hearing or simple gear.

Thanks for your help.

Regards;

Roberto Amorim.

Pre-Test discussion

Reply #1 –
I'd go along with JohnV on this one, since I've had problems with samples sounding better lowpassed. As for the frequency, I used the PCABX Training Room to test my limits, and I'd say either the 15 or the 18 kHz, as those are the ones where I actually had to focus.

Edit: I'm a newbie, so don't take my opinion as being worth very much. Just an extra push one way or the other, if the opinions are split even. I'm sure some of the regulars could make much better arguments than I did (actually, I don't think I made any arguments).

I look forward to the test! 
Happiness - The agreeable sensation of contemplating the misery of others.

Pre-Test discussion

Reply #2 –
I was thinking of a 10 kHz lowpass, which should sound pretty bad to most people on most samples.  The idea is to prevent a codec, which may sound pretty good if evaluated by itself, from being given unfairly poor ratings when compared with the other codecs.  For example, lame might be predicted to be the loser of this test.  But surely it doesn't deserve to be rated near 1.0, which might happen sometimes if there's nothing to keep things in perspective (what then about truly crappy codecs like blade?)

I understand JohnV's concern, though.  Some types of music (for example solo piano) might still sound good with a 10 kHz lowpass.

Actually, another possible idea is to use blade as the anchor.  I'm pretty sure this would always be rated the worst, and is likely to occupy the position near a rating of 1.0

I think only one anchor is needed (the low anchor).  The hidden reference, which must be chosen from in each comparison, should be functioning as the high anchor.

ff123

Pre-Test discussion

Reply #3 –
Quote
I'd go along with JohnV on this one, since I've had problems with samples sounding better lowpassed. As for the frequency, I used the PCABX Training Room to test my limits, and I'd say either the 15 or the 18 kHz, as those are the ones where I actually had to focus.

Either 15 kHz or 18 kHz would be useless for me as a participant (and I presume for some others as well).  Those lowpasses are transparent to me in music.

ff123

Pre-Test discussion

Reply #4 –
I would be interested in seeing how ATRAC stacks up (alliterative pun not intended) only because I have an MD player and find the sound pretty good.  I would like to see how it rates against the other compression systems.  Not a big deal; only if it is not a pain in the butt.   
Nov schmoz kapop.

Pre-Test discussion

Reply #5 –
Someone should really create a Listening Tests Forum.  All these tests are hard to find when they get old.

The following topics could be moved to the "Listening Tests" forum:

Link 1
Link 2




<edit: added links>

Pre-Test discussion

Reply #6 –
Quote
Should I include Atrac3 (used on Sony MiniDiscs and Real Audio at mid-high bitrates) in this test? Is there anyone out there interested in this format? (besides Den  )


and boojum! Wow, I'm not alone! 

In all seriousness. My first thought is to give ATRAC3 a miss because of the limited interest and usefulness. On the other hand, it would be interesting to see just how it does compare with others at this bitrate, not that your average Minidisc user can do anything about it. Mind you, there are the new Sony mp3/ATRAC3 CD Discmans (or is that Discmen  ) now, so maybe it would be useful for somebody out there.

Roberto, how are you planning to make the ATRAC3 files and then present them for testing decoded into wavs? If you use the Sony software, there is no export as wav option. If you use RealProducer, my understanding is that this version of ATRAC3 is not as polished as the latest versions used in the Sony software... 

Den.

Pre-Test discussion

Reply #7 –
Quote
If you use the Sony software, there is no export as wav option.

Sounds like a job for TotalRecorder.  It's a pain in the ass, but it can be done.

ff123

Pre-Test discussion

Reply #8 –
Quote
Sounds like a job for TotalRecorder. It's a pain in the ass, but it can be done.


Does this approach leave any opportunities for noise being added to the file due to the recording step, or is it noiseless/lossless?

I don't want to seem picky, I'm just curious.   

Den.

Pre-Test discussion

Reply #9 –
1. I'm also interested in how ATRAC performs compared to other alternatives for portable use like mp3 and aac.

2. Sounds like a good idea to use an anchor. And for an anchor to be useful it should be fixed, right? Anything below 12 kHz would be fine for me. Not too low though - if less than say 6 kHz I think the gap to the tested codecs will be too big to actually be able to relate their quality...

Pre-Test discussion

Reply #10 –
Even if atrac3 does not seems really interesting for us, I think that it should be included because Sony is trying to push it on portable devices instead of mp3.

Pre-Test discussion

Reply #11 –
Quote
Sounds like a job for TotalRecorder. It's a pain in the ass, but it can be done.


It's amazing what you can find if you really, really put your mind to it. I've tried to find ways before of playing with ATRAC3 outside of the Sony software for my own educational purposes, but I always lucked out.   

I just tried again, and stumbled across this... 

ATRAC3.zip

It is an acm style ATRAC3 codec, and once installed you can encode and decode ATRAC3 in the three common bitrates using the windows audio tools. I quickly tried it with Audiograbber (it was handy), and hey presto it spewed out an ATRAC3 132 kbits file inside a wav wrapper, which I could then playback/replaygain/write to wav etc with trusty foobar.  B)

I then encoded the same file with OpenMG, and tried to do a file comparison to look for differences, but the Sony version is an .omg file and there is a slight different in size ~4k in a 8.5 Mb track (@132 kbits). The Open MG generated file has tag data on the end when you check it in a hex editor. Someone else who know's what they are doing might want to look at this more closely and see if this is a reasonable way to generate some files for testing that are representative of what the Sony software churns out.

I also had a quick look at the decoded file in EAC. Has a radical haircut above 17500 Hz...  :x

This should make things easier for you Roberto if you do decide to include ATRAC3 in the test. I'm also going to have a play with this further just for my own interest...It makes ABX tests with ATRAC a shed load easier.

Den.
 

Pre-Test discussion

Reply #12 –
The ACM ATRAC3 encoder is totally different from SonicStage's one.
Speed, first (on my Duron 800) :

ACM = x4
SonicStage = x16

Then, quality :

LP4 with atrac3 ACM is a joke. I rarely heard something worst. SonicStage encodings are totally different.
LP2 is more problematic. SonicStage encoding are not very good : lack of details, easy to ABX on classical instrument for exemple. It sounds a bit synthetic to me.
AMC LP2 encodings are crisp, more detailed and more natural : ABXing is therefore more difficult on non-critical parts. Nevertheless, I noticed that cymbals are suffering with this codec (ACM). SonicStage, in comparison, has less problems.


In my opinion, ATRAC3 is interesting to include in this test for one reason only : Minidiscs performance. If the codec doesn't match with MD device, it's useless. In other words : sonicstage (+ Total Recorder) or nothing.

Pre-Test discussion

Reply #13 –
i vote for atrac3 too (there is an interest from the ripping community side too)
I know, that I know nothing (Socrates)

Pre-Test discussion

Reply #14 –
Quote
In my opinion, ATRAC3 is interesting to include in this test for one reason only : Minidiscs performance. If the codec doesn't match with MD device, it's useless. In other words : sonicstage (+ Total Recorder) or nothing.


Dammit... You really know how to let a guy down gently guru... 

Shame about the ACM codec being not representative. I really had my hopes up there for a while. Oh well, back to TotalRecorder. I thought it was a bit suss when the codec file dates were 2001. 

Thanks for the clarification guru. 

Den.

PS: It would be nice to know what the differences are though...

Pre-Test discussion

Reply #15 –
I would like to see atrac3 in the test, too.
Why don´t you use helix-producer as encoder? You can download it for free.
Also it is easy to decode in GraphEdit when RealOne and Gabest RealMedia Splitter is installed:

atracfile.rm -> Real Media Splitter -> Real Audio Decoder -> WAV Dest -> out.wav

Real supports Atrac3 at these bitrates, all only at 44100Hz, Stereo:

66 kbps Lowpass: 12,4kHz (flavor 0)
94 kbps Lowpass: 15,1kHz (flavor 1)
105 kbps Lowpass: 13,7kHz (flavor 2)
132 kbps Lowpass: 16,5kHz (flavor 3)
132 kbps Lowpass: 16,5kHz (surround mode) (flavor 8)
146 kbps Lowpass: 16,5kHz (flavor 4)
146 kbps Lowpass: 16,5kHz (surround mode) (flavor 9)
176 kbps Lowpass: 19,2kHz (flavor 5)
176 kbps Lowpass: 19,2kHz (surround mode) (flavor 10)
264 kbps Lowpass: 22kHz (flavor 6)
264 kbps Lowpass: 22kHz (surround mode) (flavor 11)
352 kbps Lowpass: 22kHz (flavor 7)
352 kbps Lowpass: 22kHz (surround mode) (flavor 12)

This information is from the Codec Information Table by Real. I don´t know what the surround mode means, maybe a different (none?) channel coupling. Also I don´t know if it´s a mistake that the lowpass of 105kbps is lower than the of 94 kbps. "flavor" is the number you tell the encoder (write in the audience-file) on what mode/bitrate to encode.

Pre-Test discussion

Reply #16 –
When you'll be trying WMA9, be sure to test both Std and Pro. The latter one is very well able to compress 44khz stereo at 128kbit, and it seems to sound better. It's also a PC playable format, neglecting it would be disappointing.

Pre-Test discussion

Reply #17 –
Quote
When you'll be trying WMA9, be sure to test both Std and Pro. The latter one is very well able to compress 44khz stereo at 128kbit, and it seems to sound better. It's also a PC playable format, neglecting it would be disappointing.

I think if both WMA versions will be tested, Atrac must be dropped. Otherwise there are way too many codecs to rate. Imo 5codecs+anchor is already pretty much maximum.

Personally I'd be ready to drop MusePack from this low bitrate test. Imo MPC lowbitrate has no practical meaning to most people or even to MusePack users outside "nice to know"-factor. I don't believe many people will be using MusePack at this low bitrate even if it does well in the test.
Juha Laaksonheimo

Pre-Test discussion

Reply #18 –
Quote
Personally I'd be ready to drop MusePack from this low bitrate test. Imo MPC lowbitrate has no practical meaning to most people or even to MusePack users outside "nice to know"-factor. I don't believe many people will be using MusePack at this low bitrate even if it does well in the test.

My thoughts exactly.

I would also like to see ATRAC3 tested.

Most people around here are quality freaks so I don't see the point in testing the standard WMA9.  Anybody who is after the best quality with WMA9 will use the Pro version, therefore only WMA9 Pro should be included in the test.

I would also like to see QT AAC at 128kbps up against Nero 6 -streaming.

Pre-Test discussion

Reply #19 –
Personally I'd rather include Musepack than ATRAC3, but I can understand that people, who already own a minidisc player, might be interested in the results.

Including both versions of WMA is pointless IMO. Most users won't be using Pro and it does not yet have the hardware- and softwaresupport of the earlier StreamVersion (AFAIK the streamversion is officially called WMA2), so its inclusion would be just as pointless as Musepacks.

dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

Pre-Test discussion

Reply #20 –
Hello, people. Thanks a lot for your help.

About atrac3: Well, so I guess it's decided, atrac3 will be tested.

@Den: I already expected the ACM to be worse, I had it for years, so it can only be deprecated.

About SonicStage? Is it freely available or only comes with Sony gear? In this case, I might have to ask someone to encode-decode the samples for me.


About Musepack: I know the results might not interest much the HA community, but keep in mind they will be published on several boards I participate, at the front page of CD-rw.org and maybe even Slashdot.

Besides, it's known Musepack does well at this bitrate, but I still believe a comparision to other formats will be welcome. Specially since portable support isn't a dream anymore, it seems.


About WMA: Should I test WMA PRO (that has better quality but worse support) or WMA Standard (That has worse quality but much better support)? I know the ideal would be testing both, but IMO testing more than 6 formats, plus anchor, will be unbearable. (And these results will _really_ not interest much people, due to the bias against MS and whatnot)


About the anchor: Do people agree with ff123, about using Blade to create the anchor? (lol)


@kl33per: Nero will be tested against other formats later this year/early next year (maybe). Again, I would like to test it now, but few people would be inclined to test a multitude of samples.


About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?


Well, I guess these are enough issues raised.

Again, thank-you very much.

Regards;

Roberto.

Pre-Test discussion

Reply #21 –
I'd be very interested in seeing how the encoder from Nero 6 stacks up as well. It sounds as though this may not be in the spirit of this particular test, but I thought I'd throw it out there...

-raymond

Pre-Test discussion

Reply #22 –
Quote
About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?

I suggest to replace BeautySlept - harpsichord sound, but maybe synthetic - by a real instrument recording (as Bachpsichord, or many others).

Nevertheless, I think that this test should include a most common classical sound : orchestra, choral, piano... something familiar. Two at least are on ff123 website : Macabre.wav, and fossile.wav - this last one was used on ff123's 128 kbps test. Fossile is ABXable. So why not this one, as common classical sound, and a harpsichord, as a more problematic classical sample?

Pre-Test discussion

Reply #23 –
Quote
About the samples: Guruboolez raised a question that there is no real classical sample (both classical samples in the 64kbps test were replaced with problem cases). So, do people agree with exchanging, say, LifeShatters (that is a bit redundant, stylewise, to BlackWater and Thear1) with Bachpsichord? Or people have another suggestion about a sample to be replaced?

You mean we have to listen to the same samples once again? I got tired of them already in the first test  Is it because it's a good test suite or because you want to be able to compare results between the tests? Anyway, I probably can't take this test, so I wonder why I even care...

Pre-Test discussion

Reply #24 –
I forgot this one :
SonicStage can be downloaded ; the software is big (70 MB), and I don't know any link for the moment. Recently, many were dead.

SonicStage produce .omg file : atrac3 + encryption. I can't sent to somebody an .omg file. DRM system of Sony forbid to anyone to read an .omg file that wasn't created on his PC.
Therefore, if someone want to upload you the files, they must be in PCM format (decoded + encoded losslessly). It's difficult for me to do it : I'm limited this summer to a 56K connection.

Keep in mind that we had to cheat with SonicStage in order to play the ATRAC3 file. If the original (.wav) file is still present on the system (same path, same name), this one will be played (and captured by TotalRecorder). It's easy to do a mistake, and to conclude to the superiority of Sony's format (I saw an occurence on a MD dedicated board, where the LP2 encoded was identical to the original).
 
SimplePortal 1.0.0 RC1 © 2008-2018