Greetings!
I finally managed to extract my password database from a backup set and am happily using an Ubuntu live CD until my new HDD ships. Anyways, great news: the multiformat listening test at 64 kbps is finally open!
The featured encoders, as most of you know already, are:
Nero Digital HE-AAC (VBR)
WMA Professional 10 (CBR)
Ogg Vorbis AoTuV 5 Beta (VBR)
The anchors used in this test are:
iTunes LC-AAC at 96 kbps VBR as high anchor
iTunes LC-AAC at 48 kbps CBR as low anchor
The test is scheduled to end on August 5th, 2007 August 12th, 2007 August 15th, 2007. Its address is: http://www.listening-tests.info/mf-64-1/ (http://www.listening-tests.info/mf-64-1/)
Happy testing!
Is the WMA Encoder the "plain" one or the one from WMP11 that uses some kind of SBR?
It's 10 Professional that comes with Vista and WMP 11.
I downloaded the Torrents and I'm seeding them now. The samples 1-17 have been downloaded a few times, but the sample 18 may have a problem.
Azureus reports this when I try to update tracker for the sample 18:
Connection Error (IOException:Server returned HTTP response code: 503 for URL: http://www.rarewares.org/tracker/tracker.p...Hi&azver=3) (http://www.rarewares.org/tracker/tracker.php?info_hash=%18%00%C0%9E%26%82%3FE%CB%83%11%A5%15%B1%07X%1A%B2%81U&peer_id=-AZ2504-VmD55HI29RFV&supportcrypto=1&port=1062&azudp=1549&uploaded=0&downloaded=4593370&left=0&event=started&numwant=8&no_peer_id=1&compact=1&key=nFxPH1Hi&azver=3))
All other samples show "OK" tracker status.
I'll try redoing that torrent file. If it still won't work, you guys can always download through http...
Redone the torrent file, seems to be working now! (please delete the old torrent file and download the new one:
http://www.rarewares.org/sebastian/Sample18.zip.torrent (http://www.rarewares.org/sebastian/Sample18.zip.torrent) )
Yay! I might actually get time to contribute results this time.
Redone the torrent file, seems to be working now! (please delete the old torrent file and download the new one:
http://www.rarewares.org/sebastian/Sample18.zip.torrent (http://www.rarewares.org/sebastian/Sample18.zip.torrent) )
Thanks. It works now.
Great job Sebastian, I hope we will get a lot of results.
# Nero HE-AAC Jul 20 2007
-q 0.24 (hard-coded value in this version of the encoder)
That last part can be left out I guess. The uploaded package has the complete encoder, including the linux binaries!!
I removed the info about the hardcoded value.
God damn it, it's becoming increasingly difficult to detect even 64 kbps encodings. Either codecs are now really good or I'm going deaf. On the tests I've performed so far, the only sample I can easily and 100% reliably detect seems to be the low anchor. And I'm using $300 Alessandro MS-2 headphones with headphone amp and M-Audio sound card.
These tests are an excellent way to lower one's self-esteem
Hi all,
just to mention that the more accurate results on this test are, better quality of future Nero AAC Encoder you can expect. This is because results from listening tests are used to tune the encoder settings. So, please concentrate very hard when you perform the test. I don't have special hearing, but I can hear differences in these encodings and thus I believe that any of you with normal hearing can hear the differences. Thus please concentrate. At least you must be able to discover low anchor, but please try to grade all samples and all encoders.
And it is not important that Nero wins (though I would like it ), only that there are as many as possible results and that you tried really hard to hear differences.
Cheers, Goran.
This is my first time participating in listening tests (though I have done rather a lot of ABXing with foobar2000).
To be sure:
I have to send the SampleXX.erf files generated through menu File>Save Test Results ?
(so that would a RAR file containing 18 .erf files)
Second question: Why isn't included iTunes AAC 7.3.1.3 64 VBR? Seems to me unfair not to include this speed.
Third question: Will I be able to see my own results?
Thx
Edit: Added second and third question
God damn it, it's becoming increasingly difficult to detect even 64 kbps encodings. Either codecs are now really good or I'm going deaf.
...
No, you're not going deaf - codecs are becoming better and better
I have to send the SampleXX.erf files generated through menu File>Save Test Results ?
(so that would a RAR file containing 18 .erf files)
Correct - at least if you want to test all samples. You can also send me an archive containing only five results if you only tested five samples.
Second question: Why isn't included iTunes AAC 7.3.1.3 64 VBR? Seems to me unfair not to include this speed.
Because that is LC-AAC which is not really optimized for low bitrates like HE-AAC. LC-AAC and HE-AAC will be featured in an upcoming 80 kbps listening test.
Third question: Will I be able to see my own results?
Not until after the test when I publish the encryption / decryption key.
Second question: Why isn't included iTunes AAC 7.3.1.3 64 VBR? Seems to me unfair not to include this speed.
Because that is LC-AAC which is not really optimized for low bitrates like HE-AAC. LC-AAC and HE-AAC will be featured in an upcoming 80 kbps listening test.
Ok, then what sense has to use VBR iTunes AAC 7.3.1.3 at 96 kbps and not higher speed? Why are high and low anchor included?
Just learning
God damn it, it's becoming increasingly difficult to detect even 64 kbps encodings. Either codecs are now really good or I'm going deaf. On the tests I've performed so far, the only sample I can easily and 100% reliably detect seems to be the low anchor. And I'm using $300 Alessandro MS-2 headphones with headphone amp and M-Audio sound card.
These tests are an excellent way to lower one's self-esteem
Typical. I was hoping this one would be a walk-over. The last one I participated in, the huge ~128 kbps test around new year 05/06(?), was a nightmare.
Great job Sebastian, I hope we will get a lot of results.
I think that I may participate this time. This will be the first listening test in which I will be involved, but, as the old saying goes, practice leads to perfection.
That last part can be left out I guess. The uploaded package has the complete encoder, including the linux binaries!!
Where can I find the linux binaries, please? I would love to encode my collection with your encoder, as it seems to be the "best of breed". And speaking of that, I can even provide Debian (also usable by Ubuntu Linux users) packages for that and I would be glad to provide packages for amd64 and powerpc too.
Regards, Rogério Brito.
Where can I find the linux binaries, please? I would love to encode my collection with your encoder, as it seems to be the "best of breed".
http://www.listening-tests.info/mf-64-1/NeroDigitalAudio.zip (http://www.listening-tests.info/mf-64-1/NeroDigitalAudio.zip)
And speaking of that, I can even provide Debian (also usable by Ubuntu Linux users) packages for that and I would be glad to provide packages for amd64 and powerpc too.
PowerPC?
The Nero encoder is distributed binary-only. Unless you get Nero's sources through some sort of NDA, you won't be able to create binaries for architectures other than i386.
PS: still not interested in creating a PowerPC repository at RareWares?
Wow, this really was much harder than I thought. I never thought I'd have to listen so hard to 64 kbps music to hear the artifacts.
One thing is certain: It's definitely usable!
Ok, then what sense has to use VBR iTunes AAC 7.3.1.3 at 96 kbps and not higher speed? Why are high and low anchor included?
Just learning
You can read about use of anchors in ITU-R BS.1116. Also try searching on this forum, there were already discussions about anchors.
Typical. I was hoping this one would be a walk-over. The last one I participated in, the huge ~128 kbps test around new year 05/06(?), was a nightmare.
Fortunately transparency at 64 kbps is highly dependent on the sample. I had huge problems identifying the encodes with classical (Paganini_Allegro_spirituoso) and "normal" music (Senor), but it was much easier with a kind of music that is most familiar to me (symphnoy_metal [sic]).
Anyway I had to do several testing rounds and multiple listening passes to learn what to listen to; concentrate on cymbals, sibilants and on other high-frequency content and the encodes should be easier to spot.
Anyway at least two of the three codecs are really good.
Fortunately transparency at 64 kbps is highly dependent on the sample. I had huge problems identifying the encodes with classical (Paganini_Allegro_spirituoso) and "normal" music (Senor), but it was much easier with a kind of music that is most familiar to me (symphnoy_metal [sic]).
Anyway I had to do several testing rounds and multiple listening passes to learn what to listen to; concentrate on cymbals, sibilants and on other high-frequency content and the encodes should be easier to spot.
Anyway at least two of the three codecs are really good.
Yes, I basically narrow in on a second or two, with lots of high pitch detail, and listen over and over and over again.
Same here =)
This is awesome, first time I participate in such test <3
The Deutsch man was an interesting soundclip, first I thought this is gonna be impossible to distinguish but turned out it was one of the easiest so far
Same here =)
This is awesome, first time I participate in such test <3
The Deutsch man was an interesting soundclip, first I thought this is gonna be impossible to distinguish but turned out it was one of the easiest so far
Interesting. I thought he was pretty tricky. At least as difficult as the rest.
(5/18 samples done, almost 6.)
Same here =)
This is awesome, first time I participate in such test <3
The Deutsch man was an interesting soundclip, first I thought this is gonna be impossible to distinguish but turned out it was one of the easiest so far
Interesting. I thought he was pretty tricky. At least as difficult as the rest.
(5/18 samples done, almost 6.)
I'm a first time listening test participant too, using an old TB Santa Cruz with Sennheiser HD465. Currently 12/18 samples done. I'm completely surprised at how good 64KBps sounds! And the high anchor is transparent most of the time for me.
Last time I tried 64kbps sounded less than impressive, but if the improvements keep up, I'm sure it'll sound transparent for me in a not too distant future.
edit: all samples tested. I'm shocked that I for the most part only hear the difference in distortions that exist in the original but somehow get mangled in the lossy versions. The music itself doesn't sound much different in my ears (which admittedly aren't that great)
BTW, thanks to everyone who submitted results so far. I will check them out as soon as my HDD arrives.
Working on a few now. It's definitely still do-able at this bitrate, even though the codecs have significantly improved since the last time I tried one of these. I don't like the signature sound of one of the competitors (and I have my suspicions on which one it is).
I showed my 12 year old boy how the testing works. His young ears can hear more acutely than mine, but I still have the better ability to pick out differences in codecs. A lot of it is knowing where to listen.
Three real competitors seems like a good number to compare.
The high anchor's got me, can't abx ~96 kbps confirmed.
The test is not finished... Why is there no link to the samples?
It's in the readme: http://www.listening-tests.info/mf-64-1/ (http://www.listening-tests.info/mf-64-1/)
It's in the readme: http://www.listening-tests.info/mf-64-1/ (http://www.listening-tests.info/mf-64-1/)
Thanks!
But I think the Unix scripts were not included, don't know why. They seem to still work perfectly.
How to for MacOS X:
* Get the Ogg Vorbis, FAAD and WavPack binaries from rarewares.org
* Rename faad_OSX to faad and make it executable:
chmod +x faad
* Make sure that the vorbis, faad and wavpack decoders are in your path. If you copied them in the 'bin' folder:
export PATH=${PATH}:bin
* Download the abc-hr_bin.zip from Roberto's listening test (http://www.rjamorim.com/test/multiformat128/presentation.html)
* Copy decodeall.sh to Sebastian's listening test folder
* Edit clicodecs.cfg:
wv, wvunpack -q -o %o %i
flac, flac -d -s -o %o %i
mp3, madplay -Q -o %o %i
mpc, mppdec --silent %i %o
mp4, faad -o %o %i
m4a, faad -o %o %i
ape, mac %i %o -d
ogg, oggdec -Q -b 16 -o %o %i
* Run the script:
./decodeall.sh
You will need to have all the samples unzipped before running the script, as described in the doc.
Edit: wavpack was missing from the config file
Thank you for the tutorial.
I would suggest using Nero AAC Decoder instead of faad, just like in windows setup.
Thank you for the tutorial.
I would suggest using Nero AAC Decoder instead of faad, just like in windows setup.
There does not seem to be any MacOS X version on the Nero website or rarewares.org. Do you know a place to get it?
Oh, yes you are right.
Sorry.
Can I be one of the first to request an extension to the test as I've just got home from an overseas holiday (Fiji - very nice ) & won't have time to listen properly until this weekend, pretty please?
Man!
Is it just me or is anyone else having a tough time differentiating between Ogg Vorbis AoTuV 5 Beta and Nero HE-AAC Jul 20 2007? This is a nice test. I'll have to sit down and really go through all the samples but man... I'm impressed with how well these two codec's have been developed.
Man!
Is it just me or is anyone else having a tough time differentiating between Ogg Vorbis AoTuV 5 Beta and Nero HE-AAC Jul 20 2007? This is a nice test. I'll have to sit down and really go through all the samples but man... I'm impressed with how well these two codec's have been developed.
Well I atleast "think" vorbis and he-aac goes kinda on the same level
I also have my suspicion that its the wma that sounds crap (next to low anchor ofc) ^^
Time will tell when the test is finished, I still got 7 tests to go through...gaah!
I would like to see as many listeners as possible participating in this test and thus it would be great if Mares would postpone closing for a week or more.
Thank you for the tutorial.
I would suggest using Nero AAC Decoder instead of faad, just like in windows setup.
Thinking of it since I am using FAAD my Nero HE-AAC results will be useless... I will be testing only the AAC part in it.
Will be like a 48kbps contender!Sebastian should I continue testing?
Can you decode files on some other system and then only copy wav files?
Can somebody put decoded files online?
Thank you for the tutorial.
I would suggest using Nero AAC Decoder instead of faad, just like in windows setup.
Thinking of it since I am using FAAD my Nero HE-AAC results will be useless... I will be testing only the AAC part in it. Will be like a 48kbps contender!
Sebastian should I continue testing?
FAAD2 also decodes the SBR part, no problem. Only problem is that the delay is not compensated and therefore it might be easier for you to pick out the AAC files.
I can check what the delay is
Hmm actually it seems the MacOS version on rarwares is pretty old. Maybe it doesn't have SBR?
The delay you should cut from the beginning of your wav files is 2624 samples.
Isn't FAAD capabile of decoding HE-AAC?
Bleh, too slow. That's what happens when you open sites in new tabs and view them only after minutes...
As for postponing, I will think about that. There are some days left, so no hurry.
Regarding the HE-AAC problem - bear in mind that the WAV files you produce have to be identical to the ones produced by the Nero decoder since the encrypted configuration files also contain hashes of the WAV files. Maybe someone who has some webspace can put the WAV files online.
The ABC-HR program did not complain when I used Quicktime-decoded HE-AAC files (which do sound different from FAAC-decoded files)
Maybe someone who has some webspace can put the WAV files online.
Yes please! WavPack format would be top.
Now that is strange. Then what are the verification hashes good for?
I uploaded the decoded Nero files somewhere. If there are more people who need this (so users that don't use x86 linux or windows), please send me a PM and I will give you the link (don't want too much traffic).
Regards,
Menno
I for one could use a few extra days. Each sample takes quite a while, and you only have so much free time after work.
Are we pressed for time? If so, why?
I woild like that too, im very short on hours this week
Just a note for people who have just installed ubuntu (like me). The default gnu java is compatible with sun java jre 1.4, but abchr-java needs jre 1.5 or later (also called version 5). I decided to go for broke and install sun java version 6:
sudo apt-get install sun-java6-jre sun-java6-plugin
and then change the configuration to use this:
sudo update-alternatives --config java
and finally from within the abchr application I changed the playback device to be the Java Sound Audio Engine.
It seems to work fine!
Geez, I could party that I finally got rid of Ubuntu Live CD and others install it voluntarily...
OT warning:
Hehe, actually for all its supposed user-friendliness, I did have to tweak it quite a bit. I didn't like the fonts, which seemed big and fuzzy, so I found out how to install/configure the Microsoft ones. Samba sharing (using cifs and not the deprecated smbfs) took me quite a while to get right, so that I could get my XP Share to mount/unmount as it came online or went offline. Using cifs without having to explicitly use the IP address of the share took more tweaking (installing winbind), and did I mention Feisty has some sort of shutdown bug when using cifs? Had to fix that too. I wouldn't have had to use samba to browse my kids' XP laptop in the first place if the GnomeFS was actually able to stream files properly over a Microsoft network.
Getting my wireless network to connect on bootup was kind of a pain. No, it's not obvious how to do that and why, but it wasn't as easy as just playing with the Network Manager settings inside a GUI of some sort.
MPlayer configuration took a lot of tweaking (it doesn't even use X11 by default).
Wanted to write to ntfs volumes -- more fiddling.
And of course the java thing.
Good things: my streaming video media plays faster, but maybe that's only because I have not installed a software firewall; I could have uninstalled zonealarm on my Win2k boot and I might have seen the same speed improvement. There's no resident anti-virus program sucking up resources. Openoffice works well enough, and it does most things (I don't play games) I need to do as well as my Win2k installation does.
/OT warning
Oh, and BTW, I have 4 more samples to test, and I'll email them off to you.
And speaking of that, I can even provide Debian (also usable by Ubuntu Linux users) packages for that and I would be glad to provide packages for amd64 and powerpc too.
PowerPC?
Yes, PowerPC. That was my intention originally. But given that we can't even redistribute the binaries (as I've read in the license), I don't think that I'm going to package them. It is a pity for those of us with multiple platforms.
The Nero encoder is distributed binary-only. Unless you get Nero's sources through some sort of NDA, you won't be able to create binaries for architectures other than i386.
PS: still not interested in creating a PowerPC repository at RareWares?
I do and I'm talking about this with Mike. I already told him by private e-mail what my plans are here with the layout of the repository and this would make mirroring the repository
much easier.
Anyway, back to the topic of the listening test, this will be my first time participating on the test. I'm having a really hard time (as others) doing the listenings.
I think that it is easier to recognize problems with genres of music that I'm familiar with (like Metal and Harpsichord), but with others, it is quite hard even to detect a low pass (say, with Techno).
One thing that would be nice to catalog in the Knowledge Base of HA would be the types of artifacts that usually arise with different encoders/formats.
It is really strange here that I have Blind Guardian's "Nightfall in Middle Earth" encoded with Apple's latest version of Quicktime at 128CBR and I
can listen to artifacts that are here, but that I'm having a hard time in participating of this test...
I think that I may have a "killer" sample to contribute...
Regards, Rogério.
I would like to see as many listeners as possible participating in this test and thus it would be great if Mares would postpone closing for a week or more.
Well, I have only had the chance (you, know, that thing called "Real Life" actually takes our time) to try 3 samples. With much difficulty, I should add.
Regards, Rogério.
I'm now working on sample 11, having worked backwards from 18 (to even out the number of results for the different samples ) . Another 10 to go, in other words.
Thanks for all submissions so far and good luck to all of you who are still testing.
Anyway, back to the topic of the listening test, this will be my first time participating on the test. I'm having a really hard time (as others) doing the listenings.
I think that it is easier to recognize problems with genres of music that I'm familiar with (like Metal and Harpsichord), but with others, it is quite hard even to detect a low pass (say, with Techno).
One thing that would be nice to catalog in the Knowledge Base of HA would be the types of artifacts that usually arise with different encoders/formats.
So far I've noticed: lowpass (of course), mushy transients, wheezy voices, narrowing of stereo, fluttering or a type of low-frequency noise in which the music doesn't sound "clean." In one case, I had a "wtf!?" experience, in which an entire instrument virtually disappeared. That was kind of cool; I've never heard that one before. I should add some of these to my artifact training page, but I'm so lazy these days. The HA knowledge base would be good too.
ff123
(you, know, that thing called "Real Life" actually takes our time)
I thought that was cancelled for poor ratings.
Hey, I just noticed this test is running!
Aren't we suppose to receive a mail from Roberto for those tests? I thought he had a ML dedicated to this.
Yeah, Roberto is so unreliable these days...
Extending the test until August 12th.
(you, know, that thing called "Real Life" actually takes our time)
I thought that was cancelled for poor ratings.
Click, but don't hurt me! (http://en.wikipedia.org/wiki/Real_life)
Interesting, really interesting. So far I haven't been aware of something like this. Once again I learned something new.
Tried a few sample....
GUYS, WOULD YOU PLEASE STOP IMPROVING YOUR ENCODERS (as LAME now seems quite lame)
Tried a few sample....
GUYS, WOULD YOU PLEASE STOP IMPROVING YOUR ENCODERS (as LAME now seems quite lame)
I'll throw in an LOL @ that.
Extending the test until August 12th.
Great! That should give me enough time to do all the samples.
Edit: This strategy of setting a short date and then extending it is probably a good one, because if you had said the 12 of August from the very start, people (me) might have put it off and not finished in time all the same.
Did somebody succeed finding (abx-ing) all encoded samples except maybe high anchor?
Did somebody succeed finding (abx-ing) all encoded samples except maybe high anchor?
I've found basically all compressed samples so far (samples 9-18) including high anchor. Possibly the odd 5.0, I can't remember now.
But I often find, when ABXing, that I can get the first 3 or 4 attempts right, but then suddenly, I can't hear the difference any more... then I have to rest my ears for little while and continue... In other words: not really artifacts you would notice during normal listening!
BTW: When should you set a score between 4.0 (perceptible) and 5.0 (imperceptible)? How do you define something that is between perceptible and imperceptible? I'm ending up giving nearly all my samples 4.0. Some slightly lower, and the odd just above 4.0 - when I have to twist my brain round and swivel my eyes into my head to hear any difference.
I'm also spending a lot of time trying to spot. There is important difference between 48 and 64 kbit test.
But I often find, when ABXing, that I can get the first 3 or 4 attempts right, but then suddenly, I can't hear the difference any more... then I have to rest my ears for little while and continue... In other words: not really artifacts you would notice during normal listening!
I'm also noticing when I ABX in test mode and I suddenly lack feedback, I become unsure at first, and then just say "oh well" and click through the trials so fast that I have little time to consciously decide whether I heard the artifact or not. I'm quite surprised afterwards when I find I scored something like 14/15, which is what usually happens.
I heard before that very fast decisions can be better than those you had time to think about, maybe this is a similar effect
I'm also noticing when I ABX in test mode and I suddenly lack feedback, I become unsure at first, and then just say "oh well" and click through the trials so fast that I have little time to consciously decide whether I heard the artifact or not. I'm quite surprised afterwards when I find I scored something like 14/15, which is what usually happens.
I heard before that very fast decisions can be better than those you had time to think about, maybe this is a similar effect
Well, sometimes I can do 10 or so abx choices quickly in a row, but it has to be a very clear artifact, or at least I have to be able to define it very clearly in my head. (I.e. "the cymbal on beat 3/8, to the left, sounds mushier than it should".)
Aren't we suppose to receive a mail from Roberto for those tests? I thought he had a ML dedicated to this.
Gah! Blame my teacher that wants me to code a huge Java program by yesterday.
Sorry about that. I wonder if it's still worth announcing?
Yeah, Roberto is so unreliable these days...
FYAD
How do you define something that is between perceptible and imperceptible?
Uhm... I'd say you could call it ... perceptible?
How do you define something that is between perceptible and imperceptible? I'm ending up giving nearly all my samples 4.0.
When ABXing, I use that range to indicate how difficult is for me to spot the problem. If i have to concentrate a lot, and might even fail at times, it's almost imperceptible, so 4.8, 4.9....
You get it?
Perceptible (4.0) means that i hear the difference, but if i didn't have the original to compare, i wouldn't think that's an artifact, or wouldn't notice it at all. (say, some lowpass filter that doens't affect the sound quality, an artifact that makes the noise more noisy without affecting the tonal parts...)
' date='Aug 4 2007, 14:11' post='508416']
When ABXing, I use that range to indicate how difficult is for me to spot the problem. If i have to concentrate a lot, and might even fail at times, it's almost imperceptible, so 4.8, 4.9....
You get it?
Well, yeah. But my kind of anal point is that if you can tell the difference, then you can't set a score higher than 4.0. I know, I know, I perhaps shouldn't take the scale so literally.
Well, you said yourself that sometimes you can hear a difference and then suddenly it's gone after you ABXed a few times. So, it's between perceptible and imperceptible.
Well, you said yourself that sometimes you can hear a difference and then suddenly it's gone after you ABXed a few times. So, it's between perceptible and imperceptible.
Ok, I'll buy that.
Off-topic,
Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.
If according to this test, WMA 10 pro be either superior/inferior to any of the other codecs, then in the next discussions, the moderator and other members may state that:
"IMHO, codec X is better/worse than WMA 10 pro"
without being thrown with TOS#8. Just don't disclose the results online (but hey, you can PM anytime! just pass the message).
It's better this way than being thrown with a subpoena or court orders or warrants of arrest...
Let the MS guys worry about the hidden result, and let us just have our final announcement of the results without WMA like saying (for example):
"HE-AAC came out No. 1. WMA 10 pro is included in the test, but we are not going to disclose its score. HE-AAC is still No. 1. Don't use WMA 10 Pro--it sucks" or if it's favourable to them, just keep a mum.
They won't stand a chance with that one! Har Har Har!
Off-topic,
Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.
I must have missed that discussion... why would publishing the results of this listening test be illegal? Because Microsoft pwnz the internets?
Edit: I've just looked through the pre-test thread and couldn't find any such discussion... could you fill me in?
There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?
There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?
Well, you could zoom in on some other part of the track and ignore the click.
Well, you could zoom in on some other part of the track and ignore the click.
That's true, but, personality-wise, it's not easy for me to give up advantages when I know they exist. It's like pretending to be shocked at a movie's end when you solved it in the first act. Now that this artifact is known, we cannot be confident others aren't also ABXing based on it, consciously or unconsciously. I'll sleep on it, but I'll probably declare this one a five-way tie (by not submitting anything).
So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function. I understand that some people focus on certain parts of the music and rate a coded depending how that specific part sounds like, but rating a codec based on something that isn't even intended to be part of the music (and wouldn't disturb your listening experience) is weird.
Edit: Bear in mind that you don't have to ABX only, but also rank each codec.
Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.
Pardon? I would also like to know where you got that information from regarding this listening test in 2007. There are no such claims that wouldn't allow me to disclose the results in the WMP 11 final package. And even if, in Germany, EULAs have no legal meaning.
So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function.
Were I evaluating eighteen "pristine" cars and one of the cars had a noticeable scratch, I'd remove it from testing because the other seventeen lacked obvious physical defects, just like I am doing with this sample set.
Edit: Bear in mind that you don't have to ABX only, but also rank each codec.
5-way tie. Honestly, it's not far from it in any case. I had a more difficult time with that particular sample than others.
So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function.
Were I evaluating eighteen "pristine" cars and one of the cars had a noticeable scratch, I'd remove it from testing because the other seventeen lacked obvious physical defects, just like I am doing with this sample set.
Edit: Bear in mind that you don't have to ABX only, but also rank each codec.
5-way tie. Honestly, it's not far from it in any case. I had a more difficult time with that particular sample than others.
I'm with you on this one. If there is a CD-style click in one of the samples (haven't got to nr 2 myself yet) then I don't understand why it has been included...
but rating a codec based on something that isn't even intended to be part of the music (and wouldn't disturb your listening experience) is weird.
Not so weird if you ask me. Simple human nature to pick the easy way out. The question is, why was this sample included, if it contains a ripping error? (Does it?)
I didn't notice there was an issue and the sample was available for use in listening tests. And as I said - you are supposed to rate how the audio files sound, it's not all about ABXing. Yes, you may "cheat" now by successfully ABXing and therefore making ABC/HR disable the reference sliders, but then do you rate the codecs based on a few milliseconds? On the other hand, you can test for pre-echo in this way or how encoders deal with vinyl clicks / ripping errors. Personally, I don't see this as a huge issue.
I didn't notice there was an issue and the sample was available for use in listening tests. And as I said - you are supposed to rate how the audio files sound, it's not all about ABXing. Yes, you may "cheat" now by successfully ABXing and therefore making ABC/HR disable the reference sliders, but then do you rate the codecs based on a few milliseconds? On the other hand, you can test for pre-echo in this way or how encoders deal with vinyl clicks / ripping errors.
Well, I may be doing things wrong, but that's basically what I do. I find a position that I can ABX and rate the samples at this position.
If I would just listen to the whole samples and rate them overall, they would all be 5.0 except the low anchor.
Focusing on something like the first couple of seconds is OK in my opinion, but focusing on one single click to rate a sample that has several seconds - well, this is not the way it should be done. That click has only several milliseconds - even it the click was 1 second long, it would only make out 4% of the sample. And like I said, what if the sample was ripped off a vinyl?
Edit: And BTW, if you think all 5 tracks sound the same, why not submit the results even if it's a 5-way-tie?
Edit: And BTW, if you think all 5 tracks sound the same, why not submit the results even if it's a 5-way-tie?
Because I want to prove to myself that I can tell the difference. That's half the reason we're all doing this, right?
Edit: Also, I wan't to help find the best codec. And if I don't zoom in on the details, how will I know which is best?
OK, your POV. As I said, I don't see this as a problem. I have no idea if that rip is from a vinyl or not, so maybe it's not clipping or an extraction error at all. You can rate that click if you want and focus on the click alone like you are trying to teach me that it's the best way, or you can think about whether or not that click is really so important for your overall listening experience / pleasure. Personally, I would either ignore it or rate it together with the rest of the music (or at least some seconds if not the whole sample) and deal with possible artifacts like with any other sample (microattacks, preecho...). I would also like to end the debate about sample 2 at this point since I see no point in continuing such a discussion.
OK, your POV. As I said, I don't see this as a problem. I have no idea if that rip is from a vinyl or not, so maybe it's not clipping or an extraction error at all. You can rate that click if you want and focus on the click alone like you are trying to teach me that it's the best way, or you can think about whether or not that click is really so important for your overall listening experience / pleasure. Personally, I would either ignore it or rate it together with the rest of the music (or at least some seconds if not the whole sample) and deal with possible artifacts like with any other sample (microattacks, preecho...). I would also like to end the debate about sample 2 at this point since I see no point in continuing such a discussion.
No need to get all upset or angry. I'm not trying to teach anyone anything, I was merely joining in a discussion I thought was interesting, which is what I thought forum boards were for. I won't rate sample 2 from that single click if it is an obvious fault of some kind.
And yes, I also listen to a couple of seconds when rating the samples. (Although I would say that the deciding sound is usually only some fraction of a second - i.e. a snare, a cymbal, etc.
Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".
Edit: Also, I wan't to help find the best codec. And if I don't zoom in on the details, how will I know which is best?
Finding the best codec is a question of averages, and discarding worst-case scenarios. ABXing is only useful to detect differences, not quality.
I think rating this sample taking how the encoders encode the artifact is useful, just like how we rate preecho on some hard-samples. What cannot be done is rating the whole sample only by this, if there are other artifacts present aswell. (don't know, haven't done it yet).
There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?
I'm the 'author' of this sample, ripped several years ago.
If you listen closely to this sample you can ear several clicks that occur on the
whole sample - and on the whole album if you got it. It's not a ripping issue nor a random noise: it's a mechanical sound. Like most instruments on earth harpsichord sound isn't electronically generated and the process required to create a note from the instrument isn't completely noise-free. This kind of annoyance are very common with solo instrumental music and are often magnified by the recording conditions (microphones close to the performer/instrument): fingers on the flute, "breath" of the organ, fingers on the bow of guitars/luth, etc...
Ah yes, these small details may cause annoying artefacts on some situations like low-bitrate encoding.
Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".
Yes, and everyone, including myself, do that too. However, our views of "weak spots" seem to differ.
[...]
Hey, you're alive! Long time since I last saw you online here.
Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".
Yes, and everyone, including myself, do that too. However, our views of "weak spots" seem to differ.
Umm... ok...? BTW, now that we know the mysterious "click" is part of the recording, I'd say it's quite fair to use it as part of the rating. I wouldn't have otherwise, though.
That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not? Maybe it was intentionally there to see how codecs behave when fed with such material? In my first test we even had a transcoded sample.
That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not? Maybe it was intentionally there to see how codecs behave when fed with such material? In my first test we even had a transcoded sample.
Oh, right. I misunderstood you. I thought you wouldn't either... hmm.. well, I guess I feel a sound codec shouldn't have to perform well on such things as ripping errors. Anyway, no big deal. I guess, in general, a codec which copes well with a click will also cope well with certain types of sounds in music, so I guess you're right
If you listen closely to this sample you can ear several clicks that occur on the whole sample
Thanks. Upon further investigation I discovered a few more of these clicks. I had suspicion they existed in the source but couldn't pass judgment without original material. I am familiar with orchestral noise and pitfalls of "one take" recordings, but the clicks in this case seem magnified.
I'd say it's quite fair to use it as part of the rating. I wouldn't have otherwise, though.
I agree. Now that the artifacts are known to be part of the original recording and not extraction errors, its testing use is acceptable.
That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not?
Correct me if I misunderstand, but the goal of this test is to see how well these codecs perform on typical well mastered, well ripped general consumer material. Certainly the bulk of the samples within reflect that viewpoint; is anything in here recorded from tape, vinyl, or anything that deviates significantly from the Redbook format? Analog and extraction artifacts force the listener to focus on outlier cases not typical of modern mastering processes. If it is indeed your intention to gauge low bitrate performance on mastering errors, then I misunderstood the goal. I brought the issue up because I believed I found a flaw in the test.
In any case, I disagree with the suggestion to ignore the clicks and trudge through as if they don't exist. Doing so forces the listener to be "honest", which blind testing seeks to avoid, while simultaneously handcuffing the tester by forcing him to use the second best detection method. As it stands, Id' say this particular sample is valid though not ideal for generic testing. I believe the average (well mastered) orchestral track to be less noisy but am not an orchestral aficionado so I will defer to the experts on this one.
To clear up some things.
I did not notice those clicks just until you pointed out that there is one after the first 4 seconds. Then I listened carefully and noticed it indeed. I did not include the sample because of those clicks, but simply because it features a solo instrument (harpsichord) and because the sample is also quite old.
Even if the click was an extraction error, I fail to see why that would be a huge problem in the test. The only advantage one might have (if he even notices those clicks) is that he can, more or less, successfully ABX the original and all samples making ABC/HR disable the reference sliders. Then it's up to the user again to rate and I highly doubt that he can give fair ratings by considering only a couple of milliseconds of a 25 seconds track, and those milliseconds not even containing anything especially related to music. What would you have done if I included a sample taken from a vinyl? Aren't we allowed to use lossy codecs for that task? Personally, I copied most of my audio tapes to 80 kbps MP3 because it's suitable for my ears.
Correct me if I misunderstand, but the goal of this test is to see how well these codecs perform on typical well mastered, well ripped general consumer material.
Really? No where did I mention well mastered and well ripped general consumer material. Red Hot Chili Pepper's "Californication" is everything else than well mastered, but that didn't keep me from encoding it to MP3, even though all tracks clip like hell.
So - all in all it's your choice if you rate the click or not, but not submitting the results for the track only because of some clicks is something I don't understand. I also wouldn't even give much importance to those damn clicks, even if they were extraction errors (which they are not thanks to guru's clarification), simply because it doesn't change my listening experience if a certain click suffers from pre-echo or not. The annoying thing is the click itself that is also present in the original and not some artifact that changes the sound of a simple click if the rest of the file is transparent.
@Sebastian
I'm sorry I got confused regarding the legality of the disclosure of the results of this test using WMA Pro 10. I just picked up from where the other (closed) thread left off. At any rate, since there would be any issue regarding it...
then ROCK AND ROLL!!!
Can't wait to see the results. Much as I would like to participate, I don't have the minimum equipment.
Much as I would like to participate, I don't have the minimum equipment.
You might be surprised. I'm plowing through with $35 Sennheiser PX-100 headphones and a noisy air conditioner a few feet away.
Can't wait to see the results. Much as I would like to participate, I don't have the minimum equipment.
IMO there is no need for high quality equipment for this test. Headphones (<$30) and descent integrated soundcard should be enough. Noisy environment is bigger problem IMO.
To illustrate the debate about noise and clicks I uploaded a new harpsichord sample (recorded in the 60') with much more mechanical clickety-clack :
http://www.megaupload.com/?d=WEJXUM18 (http://www.megaupload.com/?d=WEJXUM18)
I also have a fresh and expensive opera live recording with several stage noise (feet or object on the parquet floor) which were so close to ripping artefacts that I assimilated them first to mastering issues. I also have an excellent opera in Hybrid-SACD with weird pops on one track I can't explain. And to finish, a new CD re-issue (licensed by a label to a second one) with a ripping click noise: the CD is new, pressed but with a flaw typical of amateur ripping...
I don't think we should include samples that weren't correctly ripped unless with explicitely want to test how will react different encoders to a common issue. Better use a good rip of interesting samples rather than poor ones. But those clicks are far to be unsignificant: millions peoples have the same ripping artifacts on their library and the exemples I listed above are showing that even EAC-maniac can have the same or similar ones.
12.8. is near... How is the test going? How many results are expected?
Will it be possible to get the key file for the result files when the test has finished? It would be interesting to compare your own results with the averages.
Will it be possible to get the key file for the result files when the test has finished? It would be interesting to compare your own results with the averages.
Normally it has been published after the test has finished. Also, the complete test data must be publicly available for anyone who wants to verify the results or further analyze the data.
BTW, I mailed my result files today.
Will it be possible to get the key file for the result files when the test has finished?
Absolutely, yes!
I received both results, BTW. So far, I haven't been able to sort the results, but I hope to do so tomorrow. After that, I can also tell you which samples have a higher priority. What I can tell is that I have 22 mails in my inbox.
It will be very interesting to see the results. I felt I was finding more serious artifacts in the lower-numbered samples, ~1-5. Will be interesting to see if other people were doing the same.
Sample 8 is the only one with less than 10 results. For the other samples, here is the number of (valid) results so far:
Sample 01: 18
Sample 02: 18
Sample 03: 16
Sample 04: 17
Sample 05: 15
Sample 06: 14
Sample 07: 11
Sample 08: 9
Sample 09: 12
Sample 10: 10
Sample 11: 11
Sample 12: 11
Sample 13: 13
Sample 14: 11
Sample 15: 10
Sample 16: 10
Sample 17: 12
Sample 18: 11
Unfortunately, there were several invalid results with ranked references which I am going to ignore.
Unfortunately, there were several invalid results with ranked references which I am going to ignore.
Oops. Hope none where mine
Unfortunately, there were several invalid results with ranked references which I am going to ignore.
Can you describe invalidities? I read ABC/HR's documentation, but it is out of date. I got through five rounds of samples or so before I understood the behavior of the ABX "Testing Mode" but went back and corrected or reran previous trials (good thing I wrote notes for each sample). I should be okay now, but one never knows.
EDIT: I see that I've downloaded the Java version yet documentation points to the Win32 version, so it's not out of date as much as mildly incompatible.
Can you describe invalidities?
As I said, results with ranked references.
Unfortunately, there were several invalid results with ranked references which I am going to ignore.
Can you describe invalidities? I read ABC/HR's documentation, but it is out of date. I got through five rounds of samples or so before I understood the behavior of the ABX "Testing Mode" but went back and corrected or reran previous trials (good thing I wrote notes for each sample). I should be okay now, but one never knows.EDIT: I see that I've downloaded the Java version yet documentation points to the Win32 version, so it's not out of date as much as mildly incompatible.
Whenever you are not sure whether you can hear differences or not, you should do ABX in testing mode. When you do enough correct trials (5 out of 5, or 7 out of 8) then reference is grayed out and you can only move slider for coded sample (you can only rank coded sample). Before doing ABX in testing mode it is advisable first to do it in training mode until you are sure that you can hear differences.
IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.
And one more thing. Come on people please do a bit more on samples 7-18! At least sample 7 is very easy even for high anchor.
Whenever you are not sure whether you can hear differences or not, you should do ABX in testing mode. When you do enough correct trials (5 out of 5, or 7 out of 8) then reference is grayed out and you can only move slider for coded sample (you can only rank coded sample). Before doing ABX in testing mode it is advisable first to do it in training mode until you are sure that you can hear differences.
IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.
Great summary. This information would be helpful to newcomers were it included somewhere in the test summary or linked elsewhere.
IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.
That is precisely what I do. I have no problem in giving 5 to an encode. If I cannot spot any obvious difference after listening to the sample two or three times it gets a 5 rating.
That is precisely what I do. I have no problem in giving 5 to an encode. If I cannot spot any obvious difference after listening to the sample two or three times it gets a 5 rating.
I try a bit harder than that, because I like to know that I have all the detail in the music that I can possibly hear when I dish out a big 5.0.
Editorial comment:
Your testing procedure is just too complicated for a senior citizen old codger like me. Too many installs, too many directories, to many apps... Too much.... If anyone comes up with a simple web based user interface count me in....
Note to moderators: If my comment seems overly harsh just delete it.
Woody
I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests
Well, all you need to do is simply download the ABC/HR ZIP file provided on the test page, unpack it to a folder of choice, then download the samples you want to test, unpack them in the ABC/HR folder, run the corresponding DecodeXX.bat files from the "bin" directory and that's it. All you need to do then is to start ABC/HR, load the corresponding "*.ecf" file, test, save as "*.erf" and send me the file. That's not too complicated.
I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests
Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.
Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.
I was a bit confused … the readme says "Place
the sample package ("SampleXX.zip") in the same folder as ABC/HR and uncompress it." I thought #18 was the "latest version"
Well, all you need to do is simply download the ABC/HR ZIP file provided on the test page, unpack it to a folder of choice, then download the samples you want to test, unpack them in the ABC/HR folder, run the corresponding DecodeXX.bat files from the "bin" directory and that's it. All you need to do then is to start ABC/HR, load the corresponding "*.ecf" file, test, save as "*.erf" and send me the file. That's not too complicated.
I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests
Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.
It still takes a bit of work to get started on these tests, although it's not difficult. I can think of other ways to do this. For example: have the java app start from a click on a webpage. The application could have another dialog box specifically made for such tests which lists the samples the test administrator has selected, has buttons for downloading the compressed files for each sample (either by torrent or http), and has another button for starting the test once the download has completed (the app would automatically take care of decompressing/decoding everything). The only thing you would need to specify is where to download the compressed file to, and where to load it from when you want to start/continue the test. Instead of emailing the .erf file, you could choose to submit it directly by pressing a button (perhaps with IP logging to identify possible multiple submissions from the same person).
The only downside to this is people may not like the abchr application to connect to the internet, because then there are always questions about what it does when it connects.
The only downside to this is people may not like the abchr application to connect to the internet, because then there are always questions about what it does when it connects.
The people who care about their the nature of their Internet traffic probably aren't the same who want the most convenient experience possible. I assume the application would still work manually for those inclined to run it thusly.
why can't an installer with ALL ready be made? (not that i care, but for others like the prev poster)
Someone would have to write an installer in Java or something that runs on Windows, Linux and Mac.
I've got two comments about the current abc/hr java version:
1) in ABX mode there are keyboard shortcuts Q,W,E,D to play/stop samples -- this is VERY helpful.
However, in the ABC/HR mode, the same shortcuts are available but which sample do they refer to? That is, when I press W or E - which sample of the 5 it is going to play? Had to click the buttons with the mouse to make sure it's playing the ones I wanted.
2) on my laptop the playback via the java sound engine skips every so often. Setting high priority in the task manager (windows2000) doesn't help.
Well, I cannot help in those cases since I didn't write the program, sorry. Also, there is no better thing we can use that is also available for multiple platforms.
Anyways, today is the last day you can test and submit results. I will close the test tomorrow at around 12 PM German / Central European Time (GMT +1 with daylight saving time). Once the test is closed, no results will be accepted, so please don't wait until the very last minute.
Important! If you decide to start testing now, please focus on samples 7+ since they are the ones with the least results. This only applies to people who start testing now - if you already tested samples 1 to 6 and your results are pending, submit them, of course. Again, if you start testing now, please do so for samples 7 and higher - do not start testing now while choosing samples 1 to 6!
I try a bit harder than that, because I like to know that I have all the detail in the music that I can possibly hear when I dish out a big 5.0.
That of course makes sense. Maybe I am just not patient enough.
Sebastian, can you tell me if I rated any references? Results should've arrived from vince dot hart at gmail. I fear the answer.
The key will be available after the test is over so you can check yourself.
People, about two hours left!
Hmm... I could use a bit more results for the last 10 samples. I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).
Come on people! If you feel geeky doing the tests, remember it's all in the name of science!
Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.
Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.
This is why I joined. 48 kbps is unrealistically low and thus uninteresting to me right now, but I'm having problems as early as 96 kbps. A new 128 kbps test may be more challenging than many are expecting. 64 kbps is just the right spot where you might be fooled some of the time, making it interesting for testing purposes.
Well, in case anyone is interested, the next tests will be:
MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority
Well, in case anyone is interested, the next tests will be:
MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority
It is good to see that you have plans for several tests since no one else has been interested about conducting listening tests.
The 128 kbps MP3 test will be interesting because LAME 3.98 should be soon ready and FhG has a new version too.
Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.
It would be good to keep the tests flowing in a steady stream and not allow as long breaks as we had now. I think more testers would be willing to participate if the test discussions are constantly kept alive. For example, if a test is not possible in the scheduled time frame because of codec development reasons you could start the next test from the list instead.
Also, I wonder if the test presentations, instructions/help/faq and the results presentations could be developed further. It would be nice to have an attractive and carefully organized web site where you could browse info about the ongoing test, the test schedule and the results easily. Perhaps naylor83 would be interested in helping with this. He did excellent work with the rarewares site.
I too would be more interested in a multi-format at 96 with the AAC candidates being Nero and Itunes VBR.
This bitrate would be of more interest for those using portable devices I think.
Come on people! If you feel geeky doing the tests, remember it's all in the name of science!
And science should be in the name of better life
Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.
I disagree. 48 --> 64 --> 80 = 16 kbps between each test iteration. The difference between Vorbis at 64 and 80 kbps has me very interested in this test. 80 kbps is the portable sweet spot for me when it comes to Vorby...I'd like to see if it's still #1 or if there's a similar transparency being reached across all formats.
Sebastian, you're killing me....I need to know how many references I ranked.
I'm also with vinnie97 wholeheartedly as regards 80 kbps tests - I'd also be interested in the results & suspect quite a few participants would be pleasantly surprised with the fidelity of the sounds; I for one was a little surprised in this test at the fidelity in some of the non-problem/general-music samples e.g. "Big Yellow" & "Les Voleurs" come to mind as being mostly quite listenable to my ears/head/brain IIRC.
Can't wait for the overall results!
Edit P.S.: I also dread my contributions being thrown out for whatever reason, so I tend to be a bit conservative and listen/rank like beto wrote above.
P.P.S.: Blimey, the next 128 mp3 test will be ruddy hard, I reckon!
Hmm... I could use a bit more results for the last 10 samples. I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).
It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.
people with golden ears would help us with this test
But the golden ears would have to rate all these samples between annoying and very annoying.
Whereas people with average hearing can use the full scale for the ratings. Like I did.
Hmm... I could use a bit more results for the last 10 samples.
I have a question: When somebody ranks the reference, do you disqualify the whole test for this sample or only the result for this particular encoder within the test?
Would it make any sense for those of us who ranked the references to repeat the discarded tests more carefully and re-submit?
people with golden ears would help us with this test
But the golden ears would have to rate all these samples between annoying and very annoying.
Whereas people with average hearing can use the full scale for the ratings. Like I did.
Giving low grades wouldn't make any problems.
And because someone can hear very well and find those small differences it doesn't mean that he/she will grade those differences as annoying.
It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.
I'll do it today or maybe tomorrow for all 18 samples - Sebastian will therefore have one more result for each one.
I think the amount of ranked references in the future tests could be lessened with better instructions. ff123's practise page is very useful, but an updated page (with the JAVA version screenshots) inside the test site would be better.
The following things should be said very clearly:
- If the difference is not obvious, you should do an ABX test.
- If you can't succesfully ABX you should not try to guess. Don't pull either of the rating sliders down.
- If you didn't ABX test an obvious sample it would be good to listen to it once again and verify that you moved the correct slider.
Perhaps the instructions could have some step by step examples with screenshots. For example:
[Screenshot 1, the initial view] & a speech bubble:
"Hmmm... the sample on the left has less overall clarity and the symbals are a bit diminished, could it be the lossy file? Though, I am not sure, better go to ABX."
[Screenshot 2, ABX view, configuring the test]
[Screenshot 3, ABX view after a succesful ABX test] & a speech bubble:
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."
and so on...
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."
"... Ah, that must mean this is the WMA sample! I'll bring it down an extra notch just for spite."
Edit: Seriously though... Good suggestions!
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.
I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.
I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).
That's a fantastic idea. I downloaded the program and a few samples but got bogged down in the mechanics of things. Wasn't really sure what I was doing.
A single clear " how to " manual might have helped.
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.
In my case it would mean practically re-doing the whole test from scratcg, because most of the time I didn't do abx.
I did go through abx a few times when I was in doubt, and I did manage to abx successfully every time (or almost every, cannot remember). But there is still a chance I could have ranked a reference somtimes.
It would be nice if Sebastian could notify us by e-mail which of our tests are discarded and which we should re-do and re-submit. (Of course, if such thing is allowed by the test protocol)
The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).
N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec...
The four first samples are tested on my side.
I hope you will not get bored until you get to samples 7.-18.
I don't understand why so little results for sample 8. It is easy after short 'getting used to it'. However I will not send my results because we at Nero don't want to have influence on the results.
Add to the list explanation of ABX testing behavior. It was far from obvious to me, and I had to redo several samples after experimentation (at the time I thought using the testing mode to eliminate the reference was required in all cases, but apparently this isn't true--this should be covered, too).
Not obvious is that the listener gets only one shot at testing mode unless he wishes to redo all samples within the test or load from a good session. The program also has some strange ideas of what qualifies as a commit; if I begin testing, reset my trials, then decide to come back later, I'll find the test was marked as a failed commit and will not be able to try again without starting over or loading from a successful session save. At least twice I successfully ABXed only to have my results discarded and replaced with a failed commit.
The listener should also know that he needs to run at least eight trials in testing mode. Early on I tried finalizing when ABC/HR first displayed a number (6/6, I believe), but results came up in red and the reference was not discarded, which is not helpful, especially given the retesting issue.
I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?
Had I not kept notes on each sample, I probably would have quit due to the above frustrations.
I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?
Had I not kept notes on each sample, I probably would have quit due to the above frustrations.
You play hard, it seems...
I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)
But if you say it happens *After* saving session... i've never saved session. (don't use to get interrupted in the middle of a test)
I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)
I forgot about needing to manually select sound source; that was the first workaround I instituted. I haven't experienced clicks, but the audio doesn't always start/stop in exactly the same place. The latter is known; I saw it mentioned somewhere.
Speaking of bugs, your handle breaks quoting.
Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.
and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard),
I can confirm this (JRE Version 1.6.0 (Build 1.6.0_02-b05 / Win XP Pro SP2). But I can't confirm any other problems with ABC-HR neither in this test nor in the previous three or four tests.
The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).
N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec...
Guru plays hard too. I can't imagine ABXing all files. In fact, I didn't ABX any of them. If I think I might not be hearing a true difference I just give it the benefit of the doubt and don't pull the slider down.
ff123
Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.
I have noticed this too. In my case it is enough to close the cropped "New ABC/HR Test..." window and reopen it. On the second time I get the complete window. I don't need to restart the program.
(XP SP2, jre1.6.0_02)
Maybe a mod can split this and create a new thread called "ABC/HR Bitching" or something.
Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.
I have noticed this too. In my case it is enough to close the cropped "New ABC/HR Test..." window and reopen it. On the second time I get the complete window. I don't need to restart the program.
Really?! I will have to try your solution. I was up to now always restarting application like Mares.
BTW, you can take your time on Wednesday because the test will end around 20:00 since I am in the new apartment fixing things and talking to various mechanics (or what is the correct translation of the German "Handwerker"?).
BTW, you can take your time on Wednesday because the test will end around 20:00 since I am in the new apartment fixing things and talking to various mechanics (or what is the correct translation of the German "Handwerker"?).
Handyman?
I'm looking forward to seeing the results of this test
or what is the correct translation of the German "Handwerker"?
craftsman / craftswoman
I hope you will not get bored until you get to samples 7.-18.
Samples 1 to 9 are OK. I did 50% of the job and I have 44 hours to finish the 9 others.
Guru plays hard too. I can't imagine ABXing all files. In fact, I didn't ABX any of them. If I think I might not be hearing a true difference I just give it the benefit of the doubt and don't pull the slider down.
At this bitrate I've no problem to ABX these encodings. I can ABX most of them in less than a minute (some of them were probably done in ~20 seconds - I will see it on log files in two days). It's not very stressing and I can easily afford this for all samples. It would be very different at 128 kbps.
I only failed for one sample (
bibilolo, the 7th IIRC).
__
I noticed that for most samples I already tested two are sharing the same kind of artefact: a grainy texture I only heard in the past with SBR (HE-AAC & MP3Pro). So I guess that the second one is WMAPro and that it uses a technology which seems very close to Coding Technologies' one.
At this bitrate I've no problem to ABX these encodings. I can ABX most of them in less than a minute (some of them were probably done in ~20 seconds - I will see it on log files in two days). It's not very stressing and I can easily afford this for all samples. It would be very different at 128 kbps.
I only failed for one sample (bibilolo, the 7th IIRC).
Sample7 was the easiest one for me. I could even ABX (8/8) high anchor, which on most other samples I could not (at least I think it was the high anchor that I can not ABX).
... I can ABX most of them in less than a minute (some of them were probably done in ~20 seconds ...
Guess this means even you wouldn't easily hear problems in normal listening situations outside of abxing. Not talking about most of us with our common ears.
Sounds very good for these codecs at such a low bitrate.
Guess this means even you wouldn't easily hear problems in normal listening situations outside of abxing.
I was maybe not very clear. I can ABX (8/8...12/12) most samples in a few seconds. It's only possible if obvious artefacts are audible (
i.e. when quality sounds usually poor compared to the reference). For me 64 kbps are still unpleasant even on "normal" listening conditions ("normal" has of course to be defined... as well as "unpleasant": the ears adapt themselves very fast to poor listening conditions such as AM radio, etc...).
I would say (
from my own experience) that a huge quality gap exists between this quality and the one obtained by the same competitors at 128 kbps. But without a solid experience in artefacts hunting I'm pretty sure that this quality would be amazing to my ears.
I would say (from my own experience) that a huge quality gap exists between this quality and the one obtained by the same competitors at 128 kbps. But without a solid experience in artefacts hunting I'm pretty sure that this quality would be amazing to my ears.
It is great that you are helping us with this test Big thanks from my side!
Also have in mind that results from this test will also help in improving Nero AAC at high bitrates.
Sebastian has now one new result for each sample. All must be valid (successful ABX or 5.0). I think wasn't particularly soft with the competitors, especially on the second half of the test. Now I'm waiting to see the final results and to discover mine.
Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test
I concur. It's not that the other planned tests would be invaluable (because they certainly will be), but it seems that many are very interested in a multi-format test at 96 kbps, myself included. I also think an aoTuV B5 Vorbis-only test at various bit rates, between Q1 and Q4, with non-music samples (perhaps both effects and voice) would be interesting, seeing as how many game developers are using Vorbis as a staple for audio assets, and the results of such a test may be particularly valuable for them.
I also think an aoTuV B5 Vorbis-only test at various bit rates, between Q1 and Q4, with non-music samples (perhaps both effects and voice) would be interesting, seeing as how many game developers are using Vorbis as a staple for audio assets, and the results of such a test may be particularly valuable for them.
With all due respect, these tests are made for users who want to listen to music either at home (>= 128 kbps) or outside the house (< 128 kbps). If game developers want to encode their car engine sounds or whatever to Vorbis, they should conduct their own tests - I doubt EA or any other company is going to come and pay me for conducting tests or pay you folks for investing your time in testing.
Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.
Sorry to ask something which may be highly obvious, but where are the sources of the ABC/HR program in Java?
I see that there's a directory with licenses and it contains some mix of BSD-like (both 3 and 4 clause) and LGPL 2.1. I would like to see the sources at least of the LGPL program and, quite possibly, package it for Debian (and hence, Ubuntu), so that people can use things like:
[blockquote]
aptitude install wavpack abchr vorbis-tools faad[/blockquote]
and use the corresponding programs to decode the files (with all the dependencies already pulled in).
This would simplify the process of distributing the binaries for at least one platform. I can also create a meta-package that pulls other pieces (or I can include scripts as examples) that helps in listening tests conducted by Hydrogen audio, with all the packing and unpacking going on under a Free platform.
BTW, such script in that package would help with users on MacOS X and even those on *BSD, as I always try to write programs in a a POSIX compliant shell like ash.
In summary, I think that we could indeed make this easier for at least a portion of the potential test listeners and I volunteer to make
and maintain such a package in Debian proper so that it would be there even if I get hit by a bus.
Regards, Rogério.
Sources are hosted on rarewares: http://www.rarewares.org/others.php (http://www.rarewares.org/others.php)
I've done 17 samples of 18. Only White America sample sounds strange and confuses me. I hear difference at the first try but then can't abxed with 5 tries. Original already sound artificially however it's lossless.
Sources are hosted on rarewares: http://www.rarewares.org/others.php (http://www.rarewares.org/others.php)
Thank you very much. I see that the other .jar files are already in Debian (which probably mean that they are already in Ubuntu Linux) and now, I only have to modify the build process (with ant) to get a built package.
Nice.
Thanks, Rogério.
I've done 17 samples of 18. Only White America sample sounds strange and confuses me. I hear difference at the first try but then can't abxed with 5 tries. Original already sound artificially however it's lossless.
YES. Me too!!!
I've done 17 samples of 18. Only White America sample sounds strange and confuses me. I hear difference at the first try but then can't abxed with 5 tries. Original already sound artificially however it's lossless.
YES. Me too!!!
+1
It was the only sample where I could distinguish only low anchor.
Are you people inpatient to see the results?
Test is not closed, yet! But in case anyone is still testing, please hurry up - only a few hours left (can't say for sure - having some private work to do here before I can start evaluating the results).
Are you people inpatient to see the results?
Very much so, yes.
Closing the test in 25 minutes.
my friends do the test! please 30 minutes!!!
OK Sebastian! He does the test and send to you to mail listening-tests dot info.
We wait the results!!!
Woah, somebody's excited!
Though I don't blame you; I am too!
my friends do the test! please 30 minutes!!!
OK Sebastian! He does the test and send to you to mail listening-tests dot info.
We wait the results!!!
Huh? So, your friends are still doing the test or what?
OK, the test is over! No results will be accepted any longer. A total of 43 people submitted results.
OK, no problem decrypting the results. I know that one user who also received a mail from me a few days ago had a problem with three results actually pointing to the same test session (= set of samples). This is something I encountered in all test so far (OK, I only conducted two others ) and have no idea why it happens. The encrypted results file differ, but once decrypted, you find out that several files correspond to the test for sample 15 for example.
Anyways, was wondering what to do when people submitted results in two or more "waves", but only the first wave of results contains a nickname. Should I add the nick to the other anonymous results, or mark them as anonymous? Two people are affected by this problem. Also sending out a mail...
Bleh, it's almost midnight and I've only finished sorting the results. Since I am very tired, I hope you don't mind if I just hunt the results through chunky and friedman and assemble a nice overall graph only. The graphs for each sample along with a full results webpage will be up tomorrow.
Edit: What was wrong with Sample 7 folks? So far, it has the most invalid results (ranked reference) - something like 7 IIRC!
Second Edit: Wow, a total of 40 results were invalid. However, 470 results were valid!
Third Edit: In case anyone is interested, here's the number of valid results per sample:
Sample 01: 33
Sample 02: 33
Sample 03: 29
Sample 04: 29
Sample 05: 28
Sample 06: 27
Sample 07: 20
Sample 08: 21
Sample 09: 25
Sample 10: 21
Sample 11: 25
Sample 12: 27
Sample 13: 27
Sample 14: 24
Sample 15: 25
Sample 16: 26
Sample 17: 29
Sample 18: 21
Total: 470
Fourth Edit: Ah, missed an invalid result!
We want re-sults! We want results!
I'll start the count-down!
10
.
.
.
.
.
.
.
.
9
.
.
.
.
.
.
.
8
.
.
.
Third Edit: In case anyone is interested, here's the number of valid results per sample:(...)
These results can be compared with the previous ones (from August, 9th):
Aug 9 2007, | Aug 15 2007
Sample 01: 18 | 33 (+15)
Sample 02: 18 | 33 (+15)
Sample 03: 16 | 29 (+13)
Sample 04: 17 | 29 (+12)
Sample 05: 15 | 28 (+13)
Sample 06: 14 | 27 (+13)
Sample 07: 11 | 20 (+9)
Sample 08: 9 | 21 (+12)
Sample 09: 12 | 25 (+13)
Sample 10: 10 | 21 (+11)
Sample 11: 11 | 25 (+14)
Sample 12: 11 | 27 (+18)
Sample 13: 13 | 27 (+14)
Sample 14: 11 | 24 (+13)
Sample 15: 10 | 25 (+15)
Sample 16: 10 | 26 (+16)
Sample 17: 12 | 29 (+17)
Sample 18: 11 | 21 (+10)
People were very active during the "bonus" period. It's a good point!
IT'S ALIVE!
http://www.listening-tests.info/mf-64-1-results (http://www.listening-tests.info/mf-64-1-results)
Contains only the overall ratings and the zoomed plot. Details will be available tomorrow. You can also download the encryption key.
All post-results posts moved to the thread in Validated News (http://www.hydrogenaudio.org/forums/index.php?showtopic=56851).