HydrogenAudio

Starting again, after last week's failure...

I'd like to announce the opening of my Multiformat at 128kbps public listening test.

The test consists of comparing how 6 encoders: iTunes AAC, Ogg Vorbis, Lame MP3, Musepack, WMA Std. and Atrac3 behave compressing 18 different samples representing a wide spectrum of musical styles.

People interested in participating are invited to visit this page:
http://www.rjamorim.com/test/multiformat12...esentation.html (http://www.rjamorim.com/test/multiformat128/presentation.html)

The test will end on May 23rd and results will be posted soon after.

Best regards;
Roberto.

For those using BitTorrent, you can check stats here:
http://www.rarewares.org/tracker/mystats.php (http://www.rarewares.org/tracker/mystats.php)

Big thanks to dev0 for helping me set up the torrents.

You should use the new iTunes 4.5 / QT 6.5.1 AAC encoder.. It brought a lot of optimizations compared to the one in 4.2

You should use the new iTunes 4.5 / QT 6.5.1 AAC encoder.. It brought a lot of optimizations compared to the one in 4.2

While I agree for a different reason, several forum members complained of new artifacts in QuickTime 6.5.1 encoded AAC files that weren't in QuickTime 6.5.0 encoded AAC files.  Thus the AAC files encoded via the newer version of QuickTime were of a lesser quality.

You should use the new iTunes 4.5 / QT 6.5.1 AAC encoder.. It brought a lot of optimizations compared to the one in 4.2

http://www.hydrogenaudio.org/forums/index....ndpost&p=207158 (http://www.hydrogenaudio.org/forums/index.php?showtopic=21148&view=findpost&p=207158)

lol 

You should use the new iTunes 4.5 / QT 6.5.1 AAC encoder.. It brought a lot of optimizations compared to the one in 4.2

You must be confusing this thread with this one (http://www.hydrogenaudio.org/forums/index.php?showtopic=20301). The listening test is open now, the discussion about encoders is over. I mean, really, you had more than a month time to discuss this! 

Besides, as Roberto already mentioned in the thread he linked to, those "optimizations" were purely for graph watchers and didn't seem to bring any audio quality improvements (quite the contrary).

Already downloading at ~55 kbps, uploading ~25 kbps overall with all packages.

I'll be leaving my broadband but firewalled connection at the service of the community for a while, so leech in.

Dont seem to be able to access the test page
Anyone care to share the links for the .torrents ?

Dont seem to be able to access the test page
Anyone care to share the links for the .torrents ?

That's very weird...

The torrents are here:
http://www.rarewares.org/torrents/ (http://www.rarewares.org/torrents/)

And you can get abc-ht_bin.zip here:
http://pessoal.onda.com.br/rjamorim/abc-hr_bin.zip (http://pessoal.onda.com.br/rjamorim/abc-hr_bin.zip)

i also cant access your listening test pages (including http://www.rjamorim.com/test (http://www.rjamorim.com/test))

Hrm... I wonder if 1and1 is down.

Yep, down here as well.

Hmmm... At first, the samples were screwed, now the page is down... Maybe it's a sign from above.

Maybe it's a sign from above.

It must be!!!

First, I postpone because of a surgery
Second, I postpone because ff123 started a parallel test
Third, I cancel the test because it was all bork.
Fourth, I arrive home from a trip to find my main NTFS partition nearly completely corrupt. Spent monday and tuesday on their entireties fixing this issue.
And now, 1and1 goes bork and stops serving my pages.

I must have thrown stones at the cross.


Oh, well, there's nothing I can do now. Let's just wait 1and1 get their stuff fixed ASAP.

They promise 99.9% (http://order.1and1.com/xml/order/HostingDevConnectivity;jsessionid=3D69B0CB62CEA4B7482A0C0A08FB3399.TC63b?__frame=_top) uptime...

Weee! Seems to be back online.

Downloaded. This time I take the tests more carefully. (I did cancelled one in a day.)

They promise 99.9% (http://order.1and1.com/xml/order/HostingDevConnectivity;jsessionid=3D69B0CB62CEA4B7482A0C0A08FB3399.TC63b?__frame=_top) uptime...

Well, that's nearly 9 hours per year. Looks like the still got some downtime left :-)

Well, if no grave issues are discovered until tomorrow, I would like to invite anyone with "contacts" to post about this test on Slashdot, Kuro5hin, RAO... and anywhere else you deem appropriate.

Thank-you.

Regards;

Roberto.

Esse eu vou participar

Hmmm... At first, the samples were screwed, not the page is down... Maybe it's a sign from above.

Being the inquisitive beings that we are, I think all these signs makes us even more determined to carry it out and see the results that we probably weren't supposed to see.

In case any Mac OS X users besides myself try to participate:

You need the latest Java (1.4.2). Earlier versions may hang after you ABX for a little while. You can control-click on ABC-HR's dock icon and tell it to force quit if that happens.

If the Java 1.4.2 installer complains you need version 10.3.1 or higher of Mac OS X and you already have that, simply control-click on the installer package, Show Package Contents from the contextual menu, and delete the script named VolumeCheck.

You can easily enough get command-line utilities for all the required decoders except for Ogg Vorbis. For that, I used Amadeus to convert .ogg to .wav, since I already have it. There is a demo version available. Fink makes it easy to get flac, madplay and mppdec. Maybe Fink can give you oggdec, but I didn't see a way, unless it's a part of libogg. The pre-compiled faad mentioned in the instructions Roberto included works, but you need to add a line to clicodecs.cfg, like this one:

m4a, faad -o %o %i

Otherwise faad will not process AAC files with the .m4a extension.

Once you have all the decoders, and your sample files (unzipped inside the abc-hr_bin directory) go ahead and chmod 755 decodeall.sh as directed, but also remember to move it into the abc-hr_bin directory (from the Terminal):

mv decodeall.sh ..

That should do the trick. Or you can just drag and drop it in the Finder if you prefer, but you're going to execute it by typing this in the Terminal, anyway:

./decodeall.sh

I don't know if I will submit any results. I have downloaded sample packages 1, 3, 5, and 12 and I can so far reliably ABX only one sample from package 1. Guess I am deaf, after all. :-) Or maybe these encoders are mostly pretty good.

Hope this helps some Mac users.

Great guide! Thank-you very much for posting it

About oggdec/ogg123 for MacOS X:
http://fink.sourceforge.net/pdb/package.php/vorbis-tools (http://fink.sourceforge.net/pdb/package.php/vorbis-tools)

Hi

I think I'm about to take part in my first listening test  Starting downloads right now

Thank you rjamorin for your persistance and hard work in making this test happen

By the way, I don't want to be picky, but wouldn't it be better if Musepack's link (on the test's presentation page) was pointing to Musepack Open Source Project (http://www.musepack.net) ? I've noticed just yesterday, that it's already up and running, though still a bit "work in progress"

Best regards,

Crocodil.

Is it necessary to download the ABC/HR again?  I only ask because I think i may have some bad news.  I downloaded sample 2 zip (today) and there appears to be some problems with one sample being noticably louder than the others and one with a glitch that clearly identifies it.  I know this is the reason why the test was scrapped and restarted, but oddly enough last time i listened i didnt have any problems with the sample 2 package - now i do.  Perhaps I have missed something.  Just thought i should let you know.

Is it necessary to download the ABC/HR again?

Yes, definately. Check this thread:
http://www.hydrogenaudio.org/forums/index....howtopic=21448& (http://www.hydrogenaudio.org/forums/index.php?showtopic=21448&)

My bad.  Just thought i should flag it.  Cheers

The news on RW point to the wrong page (the old 128 listening test).

The news on RW point to the wrong page (the old 128 listening test).

Fixed. Thanks for reporting.

Would you like us to keep our BT clients open to help when the floodgates of slashdot and the like are opened?  They seem to be well seeded at the moment, I'm seeing 8 fellow seeders on the large package

Would you like us to keep our BT clients open to help when the floodgates of slashdot and the like are opened?  They seem to be well seeded at the moment, I'm seeing 8 fellow seeders on the large package

Well, that would surely be welcome.

Now, the seeder at rarewares.org is pumping files at 64kBps (more than enough currently). If the test gets announced at Slashdot, I'll probably kill it and start seeding at 256kBps :B

Thank-you very much for everyone's help in seeding

Thank-you very much for everyone's help in seeding

I thought I could join in for a while and make the samples.zip package available, but the upload speed doesn't actually burst through the roof. I have two clients open (Shadow's and ABC) and they upload with about 17kB/s each. Is nobody downloading this package or am I doing something wrong as I don't really know alot about BitTorrent in the first place. Maybe it's because of a different timezone (I'm in Europe)?

I thought I could join in for a while and make the samples.zip package available, but the upload speed doesn't actually burst through the roof. I have two clients open (Shadow's and ABC) and they upload with about 17kB/s each. Is nobody downloading this package or am I doing something wrong as I don't really know alot about BitTorrent in the first place. Maybe it's because of a different timezone (I'm in Europe)?

I think it's because there aren't many people downloading this package ATM. You can check here:
http://www.rarewares.org/tracker/mystats.php (http://www.rarewares.org/tracker/mystats.php)
that currently only 3 participants are leeching. That's why not all of your bandwidth it being requested

@rjamorim: Thanks for clarifying. I'll just leave it as is then...

Roberto, the script bin/decodeall.sh in the package is incorrect at one place:
inside the main loop, at the cmd="..." line cut delimiter is ', ' where it should be ','
(my coreutils 5.2.0 bitched about it (delimiter must be single character),
after fixing the script has executed correctly)

EDIT: Another error: cliconfig.cfg file contains info for mp4 extension... all AAC files are .m4a

OK. Fixed the script, moved it to the same folder as abc-hr (as I understand, that's where it should be run from), and added m4a to the clicodecs.cfg. New abc-hr_bin.zip is already available at the mirrors.

Thanks for reporting, AstralStorm

Hi,
I'm a little bit surprised, i cannot see anymore the BT client  recommanded by HA.. The original one isn't very col (no resume)...

I'm a little bit surprised, i cannot see anymore the BT client  recommanded by HA.. The original one isn't very col (no resume)...

AFAIK, there is no BT client recommended by HA.

In the test readme, I link to TorrentStorm (http://www.torrentstorm.com/). That's the client I use, and I like it. I can't recommend others because I have no experience with them, except BT++ (which I don't like)

Maybe http://foobar2000.org/beta.html (http://foobar2000.org/beta.html) has the information you are looking for.
I'm currently using BitTornado, since it's easy to use, powerful and close to the original codebase.
The original client does support resuming, just restart the download and it'll resume.

Azureus is also a pretty good client for bittorent:

http://azureus.sourceforge.net (http://azureus.sourceforge.net)

The biggest problem with it is that it is a resource hog.

It is written in Java so most systems should support it.

ABC just works for me.

Personally, I use Shareaza (http://www.shareaza.com/), which is a fantastic BitTorrent client IMO.  Easy as pie to set up and use, and it also can be used for Gnutella2 and eDonkey.  This may say more about me than it does about the client  , but Shareaza is the only client that I was able to understand and use intuitatively, unlike the other BitTorrent clients I tried.

i just wanted to say 'im using the bittorrent link and i'm gona help out since i'm on a university connection!
definitely want to try this out.
btw, i'm a longtime lurker and member of head-fi, so i decided to try this

i liked shad0w's (now bittornado) the best until Azureus got stable, as i usually ran more than 2-4 at once and Azureus can throttle all your torrents to a certain overall speed, not to mention make it so that only a certain number are seeding/downloading at one time. i still seed to at LEAST 1.0-1.2 if not 2 (and waaaaay more if i love what i downloaded)

I've listened to 3 samples so far and just wanted to comment that the test is definitely doable (i.e., not impossibly difficult).  Don't forget to listen once you've seeded the torrents!

ff123

I've listened to 3 samples so far and just wanted to comment that the test is definitely doable (i.e., not impossibly difficult).

I agree with this. I've finished for 10 samples so far.

The Slashdot people seem to start downloading.
2nd Multi-Format 128kbps Public Listening Test (http://slashdot.org/articles/04/05/13/2144252.shtml?tid=126&tid=141&tid=172&tid=188&tid=93)

Edit: Fixed mismatched quote.

The Slashdot people seem to start downloading.
2nd Multi-Format 128kbps Public Listening Test (http://slashdot.org/articles/04/05/13/2144252.shtml?tid=126&tid=141&tid=172&tid=188&tid=93)

Edit: Fixed mismatched quote.

Impressive. They are not saying this test is being conduced by HA

And yes, they are seriously downloading stuff. Just check the BT stats:
http://www.rarewares.org/tracker/mystats.php (http://www.rarewares.org/tracker/mystats.php)
More than 18Gb transferred so far...  If only 10% of the people downloading would actually send me results... 

I have a concern about a test with this high bitrate. As far as I can tell they all sound like wave files at this bitrate, so won't the result just be random, as people can't tell the difference?

In addtion have I two questions:

(a) Will the file size be taking in consideration in the conclusion?
(b) Why isn't Real Audio 10, VP6, and normal Vorbis in the test?

Best regards,
Martin

The Slashdot people seem to start downloading.
2nd Multi-Format 128kbps Public Listening Test (http://slashdot.org/articles/04/05/13/2144252.shtml?tid=126&tid=141&tid=172&tid=188&tid=93)

I hope that they can contribute: http://slashdot.org/comments.pl?sid=107516&cid=9145429 (http://slashdot.org/comments.pl?sid=107516&cid=9145429)

The whole thing is a pseudoscience wannabe filled to the brim with vendors lauding useless gear before gullible fools. I will not participate.

(b) Why isn't Real Audio 10, VP6, and normal Vorbis in the test?

Best regards,
Martin

Normal Vorbis was tested in the previous 128 kbps test and has not had any quality improvements since then.  Many of the third party tunings have fixed some of the problems in the Xiph version and according to the Vorbis Listening Test (http://www.hydrogenaudio.org/show.php/showtopic/20389), they all outperformed the Xiph version.

Quote

The whole thing is a pseudoscience wannabe filled to the brim with vendors lauding useless gear before gullible fools. I will not participate.

Are you really surprised?

The discussion there has actually been pretty civil so far, considering /.'s standards.

Quote

(b) Why isn't Real Audio 10, VP6, and normal Vorbis in the test?

Best regards,
Martin

Normal Vorbis was tested in the previous 128 kbps test and has not had any quality improvements since then.  Many of the third party tunings have fixed some of the problems in the Xiph version and according to the Vorbis Listening Test (http://www.hydrogenaudio.org/show.php/showtopic/20389), they all outperformed the Xiph version.

Right. Besides, 6 codecs is more than enough. Testing more than that at 128kbps would be painful and mean.

Some interesting comments:

http://slashdot.org/comments.pl?sid=107516&cid=9144780 (http://slashdot.org/comments.pl?sid=107516&cid=9144780)
http://slashdot.org/comments.pl?sid=107516&cid=9144833 (http://slashdot.org/comments.pl?sid=107516&cid=9144833)
http://slashdot.org/comments.pl?sid=107516&cid=9144785 (http://slashdot.org/comments.pl?sid=107516&cid=9144785)
http://slashdot.org/comments.pl?sid=107516&cid=9144976 (http://slashdot.org/comments.pl?sid=107516&cid=9144976)  <-- r3mix ruelez!!!
http://slashdot.org/comments.pl?sid=107516&cid=9145079 (http://slashdot.org/comments.pl?sid=107516&cid=9145079)
http://slashdot.org/comments.pl?sid=107516&cid=9144713 (http://slashdot.org/comments.pl?sid=107516&cid=9144713)
http://slashdot.org/comments.pl?sid=107516&cid=9146187 (http://slashdot.org/comments.pl?sid=107516&cid=9146187)

I wonder if the guy that submitted the news article did that to hurt me  (j/k)

You say having good quality cans is important.  Is something like a Grado SR-60 good enough to hear the differences?

Man if you look, i've responded to all of those threads, and been moderated -1 pretty much every time 

You say having good quality cans is important.  Is something like a Grado SR-60 good enough to hear the differences?

I guess so. Even my very crappy Philips HP250 can catch artifacts here and there.

Man if you look, i've responded to all of those threads, and been moderated -1 pretty much every time 

Are you really surprised?

You say having good quality cans is important.  Is something like a Grado SR-60 good enough to hear the differences?

Absolutely.  I have those headphones and I was able to ABX nearly all the samples in the last 128k test.

Some interesting comments:

It's been a while since I've read slashdot - It's amazing how bad it's gotten.  My brain was making this terrible squealing noise because of the pain I inflicted by even looking at some of those comments.

Amazing. harashin finished all samples again

Congratulations, man. Now go get some sleep.

Don't forget to listen once you've seeded the torrents!

ff123

you mean we're supposed to do something once i've downloaded it? i just download/burn/delete

I've listened to 3 samples so far and just wanted to comment that the test is definitely doable (i.e., not impossibly difficult).

*pets his shiny new HD590s* 


edit: omg, began the test. i'm going to cry... these are truely kicking my ass 

Roberto, would it be usefull for me to mirror the sample packages on my server?
Our firewall won't allow me to use BitTorrent, but I would be more than happy to help take some of the load off rarewares.org.

Roberto, would it be usefull for me to mirror the sample packages on my server?

Well, so far, RW has been managing to keep hosting the packages.

Actually, I guess that we pumped so much data today, that it broke the host's bandwidth meter. It's reporting 910Mb transferred, while on a normal day (without distributing samples, RareWares alone), about 2Gb are consumed.

Besides, it seems people are going really mad on BitTorrent instead of HTTP. More than 50Gb (!!!!) transferred so far.

Anyway, yes, it would be good to have an emergency place in case my host goes down again (wouldn't surprise me, I bet it has never been Slashdotted)

But be prepared to transfer a handful of gigabytes/day.

Thank-you.

Regards;

Roberto.

As far as I can tell they all sound like wave files at this bitrate, so won't the result just be random, as people can't tell the difference?

The results will be deeply analyzed. If people can't tell the difference and the results are random, the analysis will show it.

For example, look at the results of the old test : http://www.rjamorim.com/test/128extension/results.html (http://www.rjamorim.com/test/128extension/results.html)

The analysis at the bottom showed that the results were indeed random exept for MP3, thus the other codecs were rated equal.

Quote

Some interesting comments:

It's been a while since I've read slashdot - It's amazing how bad it's gotten.  My brain was making this terrible squealing noise because of the pain I inflicted by even looking at some of those comments.

It's the first time I go to slashdot. I don't understand what you find bad it these links. The first one about frequency plots is actually better than all the FAQs we have had about it here. It covers quite all the reasons why graphs shouldn't be used to compare codecs.

I started doing the test. Only three samples done so far.

Some samples are much too hard for me. Will probably have to leave 5/5 for those samples as a quick listening doesn't reveal anything to my quite untrained ear. For more thorough work I can only do about 2 sample sets a day. This is hard work.

One question:

Does anybody know of a way around the Java ABC/HR limitation that it only allows the standard Windows Directsound output to be used?

I can't use my better gear, because it really is good only through ASIO and I can't select that from ABC/HR. Not that better sound card would change the results, but I'd just like to use the gear that I'm used to.

regards,
halcyon

Quote

You say having good quality cans is important.  Is something like a Grado SR-60 good enough to hear the differences?

Absolutely.  I have those headphones and I was able to ABX nearly all the samples in the last 128k test.

Okay great.  I got everything installed last night and I am going to try and do the test over the weekend.  I am using Linux so it took me a little longer to get everything set up, but it is still pretty easy.

FWIW, I am using Mandrake 10.0 and there were packages for everything but the mpc decoder (although the xmms plugin was available), so I had to use the precompiled binary from the link in the readme.  The decode script ran flawlessly.

Also, why did my post with the question get deleted/modded.  Is it against the TOS to call headphones cans?

It's the first time I go to slashdot. I don't understand what you find bad it these links. The first one about frequency plots is actually better than all the FAQs we have had about it here. It covers quite all the reasons why graphs shouldn't be used to compare codecs.

It wasn't necessarily those comments in particular.  There were a bunch of comments along the lines of "Just some guy trying to hock useless expensive audio gear" or "You can get the same size file with a much higher bitrate using codec X" or "mp3 sounds like AM radio even at 368kbps" or other nonsense.  Of course, the most nonsensical ones were modded way up.

Yeh, and i got modded down as overrated for countering that the --alt-preset setting achieves transparency at only around 190kbps. I even mentioned that this was verified in double-blind tests found on the linked-to webpage.

*sigh* oh well, many /.ers seem to have their minds made up on certain issues and mod you down when you don't support their preconceived notions.

Also, why did my post with the question get deleted/modded.  Is it against the TOS to call headphones cans?

I can still see your post :B

Quote

Also, why did my post with the question get deleted/modded.  Is it against the TOS to call headphones cans?

I can still see your post :B

Clearly visible here.
Cans are as good as any word for the earcups IMO

The results will be deeply analyzed. If people can't tell the difference and the results are random, the analysis will show it.

How can you tell whether people can't tell the difference and thereby just choose something, or the codecs are equally good?

As I see it, it is a statistical problem, that can't be ignored when the bitrate is this high.

Normal Vorbis was tested in the previous 128 kbps test and has not had any quality improvements since then.  Many of the third party tunings have fixed some of the problems in the Xiph version and according to the Vorbis Listening Test (http://www.hydrogenaudio.org/show.php/showtopic/20389), they all outperformed the Xiph version.

I see. Will any of the third parties versions be ported to the Xiph version?

Right. Besides, 6 codecs is more than enough. Testing more than that at 128kbps would be painful and mean.

Could VP6 and Real Audio 10 be considered for the next test?

I wouldn't be supprised if they preform very well at both low and high bitrates.

How can you tell whether people can't tell the difference and thereby just choose something, or the codecs are equally good?

When they choose to rank a sample out of guessing, there's 50% chance of picking the reference. Multiply that for 6 samples, and you'll see that the chance of picking the reference in one of the six samples is pretty high.

Result files with ranked reference are discarded.

I see. Will any of the third parties versions be ported to the Xiph version?

It only depends on Xiph themselves. But my guess is no. From the past experience, they always chose to ignore tunings done by third parties and go their own way instead.

Could VP6 and Real Audio 10 be considered for the next test?

The next test performed by me will be at 48kbps. Real Audio 10 is under consideration.

About VP6: do you mean this (http://www.on2.com/avc_codec.php3)?

I can see nowhere to download that codec.

When they choose to rank a sample out of guessing, there's 50% chance of picking the reference. Multiply that for 6 samples, and you'll see that the chance of picking the reference in one of the six samples is pretty high.

Result files with ranked reference are discarded.

Why? That's puts very good codecs in a bad light.

If someone indeed can't tell the difference between the original and the compressed, the compressed must thereby be VERY good.

About VP6: do you mean this (http://www.on2.com/avc_codec.php3)?

I can see nowhere to download that codec.

I think you'll have to encode your audio files from VP6 (http://www.on2.com/vp6.php3) like you would for a video file.

Why? That's puts very good codecs in a bad light.

If someone indeed can't tell the difference between the original and the compressed, the compressed must thereby be VERY good.

If someone can't tell the difference between a codec and the original, he must give 5/5 to both files, and his result is taken into account.
To "rank the reference" means to think that the original sounds worse than the encoded file, and give it a bad mark, which is absurd. That's why the result is discarded.

If someone can't tell the difference between a codec and the original, he must give 5/5 to both files, and his result is taken into account.
To "rank the reference" means to think that the original sounds worse than the encoded file, and give it a bad mark, which is absurd. That's why the result is discarded.

Phew. Now I am more confident

I thought you ment that, if a codec was rated as the original, then the vote would be discarded. This would be a serious statistical error to do!

I think you'll have to encode your audio files from VP6 (http://www.on2.com/vp6.php3) like you would for a video file.

From what I see, that is a VCM video encoder only. It doesn't encode audio. (or am I blind?)

From what I see, that is a VCM video encoder only. It doesn't encode audio. (or am I blind?)

Nope. You're fine. Nothing audio related to VP6... 

Quote

Quote

Also, why did my post with the question get deleted/modded.  Is it against the TOS to call headphones cans?

I can still see your post :B

Clearly visible here.
Cans are as good as any word for the earcups IMO

Okay, I see it now.  I hope my ears are better than my eyes.

Alright. I've just finished the test and I'm about to submit to Roberto. It was not so painful.

Anyway, I have some questions. First is that the decoded wavs from Vorbis and MP3 are producing the exact same filesize than the original (reference) wav. I mean exact, down to the last byte. As far as I compared, that's on all samples. OK, I know that shouldn't mean anything but I find it to be quite strange.

There is one other thing. I need to know which codec was used for the *_5.flac files. Was it WMA or Atrac3? In the Kraftwork Test (Sample10) there is a huuuuge artifact (that I won't mention for the sake of this test) with this codec. If someone could please PM me about the codec I'd be really thankfull.

There is one other thing. I need to know which codec was used for the *_5.flac files. Was it WMA or Atrac3?

I'll reply to your e-mail with the answer. Don't ask these things publicly


And yes, rjamorim.com is down again, in case anyone noticed. Let's all collectively blame 1and1.

And yes, rjamorim.com is down again, in case anyone noticed. Let's all collectively blame 1and1.

It is up for me.

I’ve encountered many problems on the few samples I’ve already tested. There were many pops and skipping during playback. I’ve changed many times buffer length, but on some samples (like the Bartòk one), problems remains. It’s really disturbing, especially on difficult samples, where differences are hard to perceive. I gave up ABX phase when problems were too disturbing. On the first sample, I’ve also renounce to rate the files: too angry for undisturbed evaluation! The problems are also audible on reference playback.

These problems are maybe due to my little CPU (Duron 800), I don’t know. I remember that I’ve encountered similar ones during the last AAC test (Hongroise.wav), but the test was easier, and the problems were therefore less critical (easy test = less concentration).
Am I alone to have these troubles?

I think i can understand you: because of skipping i lost concentration many times while ABXing (successful) the first sample. With my CPU (Celeron 1000), some pops and skipping occur but not enough to make me "angry".
In a such critical test a perfect reproduction is necessary to concentrate and give an undisturbed judge. The ITU-R BS.1116-1 infact says: "There should be no audible artefacts (e.g. “clicks”) of the switching system, since such artefacts can seriously interfere with the assessment process." That said, i think we have to make a compromise because we can't exclude non-Windows users from an OPEN listening test.

Certainly. Nevertheless, what will happen to my (or someone else) results if I rank 3.0 file 3L or 6R, and if I'm failing on ABX due to disturbance of playback stage? I've abandoned the idea of confirming my results by ABX on one sample, simply because I feared to obtain unsignificant results due to concentration lost. Results can't be considered as serious, if ABX scores are bad, and if files have notation inferior to 5.0.
I've just finished 6 archives, but I think that I'll keep the same attitude for the 12 other: better skipping the ABX phase than risking to lose the significance of my work [i.e. careful listening and evaluating] previously done.

The click problem is back !

I use ABCHR 0.4b4, 10 march 2004 (file abchr.jar 05/08/2004 14:01).

It is especially present in the Debussy sample if you select a part beginning into the violins, between 2 and 3 seconds, for example. Once you get the click, browse into the different sample. Some have the same click, some have a different one.

But there is no click at all if the sample is played from the beginning.

I've tested the four first samples without noticing anything wrong. Debussy is the 5th one.

The click problem is back !

I use ABCHR 0.4b4, 10 march 2004 (file abchr.jar 05/08/2004 14:01).

It is especially present in the Debussy sample if you select a part beginning into the violins, between 2 and 3 seconds, for example. Once you get the click, browse into the different sample. Some have the same click, some have a different one.

But there is no click at all if the sample is played from the beginning.

I've tested the four first samples without noticing anything wrong. Debussy is the 5th one.

I confirmed the noticeable click in the debussy sample if it is started between 2 and 3 seconds in both abchr-java and abchr (native windows).  abchr-java adds a second, softer click shortly after the first one, as I reported earlier, although it seems worse in this sample for some reason.

I wonder if the clicking can possibly be reduced or eliminated if a fade-in is used?

ff123

Quote from: Pio2001,May 15 2004, 06:11 PM

I wonder if the clicking can possibly be reduced or eliminated if a fade-in is used?

ff123

Do you think that fad-in/fade-out are a good idea for listening tests based on A-B conditions? From my experiences, difference in volume level between the end of the file and the beginning are highly disturbing. When I begin the listening of the file X, I always think it's the encoded one, because low-volume sound muffled, a bit lifeless, etc...


Anyway, am I alone to have problems with the first sample (string quartet)? I also had problems with the Debussy sample, but the encoding problems were really severe with a couple of file that I wasn't really annoyed.

Such fade-in/out must not exeed 1/40th of a second. In fact to avoid a click, the fades should be half the period of the lowest frequency present at the beginning (or at the end). It it short enough to be unnoticeable, and long enough not to introduce audible harmonics (=click).
To find it, just look at the waveform, and set the fade lenght approximately the same lenght as the longest visible oscillation, so that it doesn't look disturbed by the process.

What do you mean with "having problems" ? Problems for testing (bugs, clicks), or hearing artifacts (pre echo, ringing) ? The latter should be discussed after the test is done. I'm not finished yet.

Just wanted to mention that the test is quite hard to do under OSX, as I am experiencing many sound problems with the application: some clicks, and often the sound is stopping for about half a second.

What do you mean with "having problems" ? Problems for testing (bugs, clicks), or hearing artifacts (pre echo, ringing) ? The latter should be discussed after the test is done. I'm not finished yet.

I'm not talking about encoding problems (I knew some of them before the test, because I've submitted this sample), but playback problems (clicks, but also short moment of silence during playback).
With the Debussy sample, I'm not too annoyed, because I don't really need too much attention for hearing artifacts (or ABX them, but I don't ABX too obvious artifacts). But for more subtle artifacts, the playback problems (pops...) are disturbing.

To "rank the reference" means to think that the original sounds worse than the encoded file, and give it a bad mark, which is absurd. That's why the result is discarded.

OT? First, hearing is subjective. Second, I may like the sound of a piano after a low pass filter was applied to the recording, because it removed some unwanted nosie from it. Maybe in this test the negative effects of discarding data outweights the positive effects. One still can not state in absolute terms that preferring the encoded file over the original is absurd.

One still can not state in absolute terms that preferring the encoded file over the original is absurd.

In this test, it's all about sounding as close to the original as possible. What people prefer is another matter. If you prefer to lowpass at 11 kHz is totally up to you

Quote

One still can not state in absolute terms that preferring the encoded file over the original is absurd.

In this test, it's all about sounding as close to the original as possible. What people prefer is another matter. If you prefer to lowpass at 11 kHz is totally up to you

Because compression discards information and adds other artefatcs to the original and the samples were choosen to find such artefacts easier (as far as I know people worked hard to find such), than discarding those results where the encoded version "rank the reference" is the right thing to do.
But in general I can prefer the sound of the compressed one, because my hearing is just as subjective as yours. If you would wave a $100 bill in front of my eyes every time the compressed one is played, I bet I would prefer the sound of that one (but that would not be a blind test would it )

In this test, it's all about sounding as close to the original as possible.

But he doesn't know which is the original

So, yes, you can, in some cases, have encoded samples sounding better than the reference. Even experienced listeners get fooled sometimes. Check the private listening test conduced by Menno comparing HE AAC to other codecs.

The problem is, if a listener ranks the reference, you can never know if he actually listened to the samples or was just guessing (unless he did ABX, but that's another story). And guessing is bad for the test. That's why I usually choose the safe route and discard results with ranked references, unless the listener ABXd the samples with confidence.

Regards;

Roberto.

Is it just me, or rjamorim.com is down again?

It's OK here

OK now, but I couldn't access to your site 10 hours ago.

We've got to the end of the week to test 18 samples?
Please Roberto give us some more room... I've got a job!

We've got to the end of the week to test 18 samples?
Please Roberto give us some more room... I've got a job!

I assume it's not necessary to test all 18 samples. At the end of the week, just send the results of the tests you managed to complete.

I know that, but I would like to test them all. And I guess that the larger number of results the more valid the result should be...

dunno if this was discussed already, but the .torrents seem to be removed, from the normal sample download, sample16 doesnt seem to be downloadable

I removed the .torrents because if someone is going to start the test now, he should be hurried, and therefore he should use the http links. But the tracker is still working.

About the sample packages: they are there:
http://www.rarewares.org/samples/ (http://www.rarewares.org/samples/)

I guess the problem you experienced is related to rarewares.org moving to another server.

24 hours to go

In about 24 hours, this test will be closed.

If you don't plan to test more samples, please start sending your results.

Regards;

Roberto.

results sent

Could somebody enlighten me on details of kraftwerk sample? (Sample10)

I need album and track name.

Thank-you in advance.

Album: The Man Machine
Title: The Robots
Tracknumber: 01
Date: 1978

A properly remastered reissue of this album (and other Krafwerk's albums) is expected soon.

Amount of results I got so far, for those interested:

Sample01: 21
Sample02: 23
Sample03: 22
Sample04: 23
Sample05: 17
Sample06: 17
Sample07: 15
Sample08: 17
Sample09: 19
Sample10: 25
Sample11: 13
Sample12: 12
Sample13: 13
Sample14: 14
Sample15: 14
Sample16: 11
Sample17: 15
Sample18: 13

That is before screening for ranked references, mind you

I'm afraid that testing 18 samples wasn't a good idea, after all.

Amount of results I got so far, for those interested:

Sample01: 21
Sample02: 23
Sample03: 22
Sample04: 23
Sample05: 17
Sample06: 17
Sample07: 15
Sample08: 17
Sample09: 19
Sample10: 25
Sample11: 13
Sample12: 12
Sample13: 13
Sample14: 14
Sample15: 14
Sample16: 11
Sample17: 15
Sample18: 13

That is before screening for ranked references, mind you

I'm afraid that testing 18 samples wasn't a good idea, after all.

Why not?  Numbers around 12 per sample aren't unexpected or even necessarily bad.  My opinion before the test was that it would be better to have numbers like this on more samples than a higher number of listeners on fewer samples.  Also, the test can be extended to help increase numbers if you process them and still think it's problematic.

ff123

LAST CALL

The test will finish in less than one hour. If you still have results to send, send them ASAP.

Got mine in just under the wire. 

My results are pretty pathetic, but I'm not surprised.  I think my personal transparency threshold is somewhere around 48-64kbps.  I'm convinced I have "artifact-proof hearing". 

Thanks to Roberto for conducting this test.  He's done a great job yet again. 

Test is now CLOSED.

Results will be published quite shortly. Huge thanks to all participants.

I will be a show ruiner and reveal that Vorbis (AoTuV tweak) enthusiasts will be very happy...