Greetings!
After months of endless discussion and preparation, the MP3 listening test is finally open.
The featured encoders, as most of you know already, are:
LAME 3.97
LAME 3.98.2
iTunes 8.0.1.11
Fraunhofer IIS mp3surround CL encoder v1.5
Helix v5.1 2005.08.09
The anchor used in this test is:
l3enc 0.99a as low anchor
If you want to read more before starting, the address of the test is http://www.listening-tests.info/mp3-128-1/ (http://www.listening-tests.info/mp3-128-1/). If you want to start right away, have a look at the readme (http://www.listening-tests.info/mp3-128-1/readme.htm). Please note that you do NOT have to test all samples - it is a great help even if you test one or two!
Info: The people who did not start testing already but who would like to participate, it would be very nice if you could focus especially on samples: 5, 6, 8, 9, 10, 11, 13, 14. Sample 1 already has 25 valid results, so please do not test it (of course, if you already did, you can submit your results for it, but if you just started, maybe invest time and effort in testing the aforementioned samples first).
Update: The test is scheduled to end on November 22nd, 2008.
Warning: There seem to be problems with ABC/HR when JRE 1.6 is installed. If you are affected by hangs, crashes or any other difficulties, please download and install JRE 1.5 Update 15 from http://java.sun.com/products/archive/j2se/5.0_15/index.html (http://java.sun.com/products/archive/j2se/5.0_15/index.html). You do NOT have to uninstall your existing copy of JRE 1.6 since both versions can be installed at the same time. However, if you have JRE 1.5 and JRE 1.6 installed, please make sure you start ABC/HR with JRE 1.5 manually. Since the working directory has to be the location where abchr.jar is stored, the best thing you can do is to open a command prompt window, navigate to the location where you stored abchr.jar and the rest of the files and then call "C:\Program Files\Java\jre1.5.0_15\bin\java.exe -jar abchr.jar". Of course, this is only an example which you have to adapt according to where java.exe is stored on your PC.
Note: As Hydrogenaudio Forums members and I found out, there is a bug in the iTunes MP3 encoder that is present in all iTunes versions since 5.0.1.4 (including the tested version 8.0.1.11). The problem manifests itself only on multi-core CPUs (both PCs and Macs) and leads to a significantly lower bitrate of the encoded samples. Apple engineers confirmed our findings and assured me that the next iTunes release will contain a fix for this problem. Please notice that the only functioning work-around is to disable all but one core in your computer's BIOS or to start Windows with the /numproc=1 parameter. Setting the iTunes process affinity to one core does NOT help.
Happy testing!
I've got problems with "Save test results" and "Save session". ABC/HR locks up and the only way I can close it is using the task manager. JDK 1.6.0_10 is installed.
I've got problems with "Save test results" and "Save session". ABC/HR locks up and the only way I can close it is using the task manager. JDK 1.6.0_10 is installed.
Can it be connected with Google Chrome being open? I think Chrome's sandbox is to blame. Closing Chrome helped in my case.
EDIT: No, now it works even with Chrome being open, so this is not the case.
EDIT2: WinXP SP2 + JRE Version 6 Update 10 (build 1.6.0_10-b33) here.
FYI I have also had various problems with ABC/HR locking up.
Java Control Panel says "Version 6 Update 7 (build 1.6.0_07-b06)" (most recent update).
I can load in a config, but if I then try to load in another, or use any other menu item it seems, it hangs.
I've got problems with "Save test results" and "Save session". ABC/HR locks up and the only way I can close it is using the task manager. JDK 1.6.0_10 is installed.
Me too on VistaSP1 and jre1.6.0_04. I did 3 samples, one went OK, an other caused a hang when opened Save Results (and therefor had to do listening test again) and the third sample hang ABC/HR after the result file was written.
I am seeing the same issue.
I have XP and Java 6 update 7 (build 1.6.0_07-b06). It's updater does not offer to install "update 10".
When I create a new unencrypted session the save functions work normally.
Same problem here. Windows Xp and 1.6.0_10-rc-b28.
Update 10 is only a release candidate.
Same for me (Windows XP).
Can I post the news of this iTunes encoder bug on a news site like neowin?
Sounds quite embarrassing.
shakey_snake, you can post it as the message is public anyways, but please consider that the MP3 developers at Apple were very cooperative and confirmed / fixed the bug within 24 hours. The next iTunes version should have the fix included.
As for the ABC/HR problems - I am afraid that there isn't much I can do except to contact schnofler once more, but since he didn't reply to my last PMs, I don't know if he can help or if he's too busy whatsoever. I can also confirm the problem but it doesn't happen all the time.
The problem seems to be caused by JRE 1.6. JRE 1.5 (https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=jre-1.5.0_15-oth-JPR@CDS-CDS_Developer) works fine here.
I just recompiled ABC/HR using the latest JDK, JDOM and JGoodies but the problem persists. With the "old" version included in the test, I can open the configuration files but cannot save them and with my compile, I cannot open the configuration files so I created new ones, I can save them, but once I try to open a new configuration, it hangs.
So yeah, unless schnofler comes up with a great idea, please use JRE 1.5.
Although it's great that the encoders can adapt to certain flaws by raising the bit rate, and it's a clear advantage when seeking transparency in the best listening conditions, I think encoders that can maintain quality and also keep a more constant bit rate should be noted in a 128kbps test, because they are superior, if I'm aiming for 128kbps and I get 200+ I switch encoder.
Hehe just curious, you didn't list the encoders by any chance in your opinion of best to worst did you?
I think encoders that can maintain quality and also keep a more constant bit rate should be noted in a 128kbps test, because they are superior, if I'm aiming for 128kbps and I get 200+ I switch encoder.
If one wants, needs, or things that wants or needs an ABR encoder, there are options out there for this task. This test is not about this, like you knew, right?
Also, if a setting does continually use a (much?) higher than expected bitrate, you have two options: Choose a lower one, or choose a different method/encoder.
That is not the case here, as you also knew, right?
I think encoders that can maintain quality and also keep a more constant bit rate should be noted in a 128kbps test, because they are superior, if I'm aiming for 128kbps and I get 200+ I switch encoder.
Like you knew, right?
No idea what you mean by this jerk off.
If one wants, needs, or things that wants or needs an ABR encoder, there are options out there for this task.
Some encoders don't allow ABR.
if a setting does continually use a (much?) higher than expected bitrate, you have two options: Choose a lower one, or choose a different method/encoder.
If 70% of your files are at the target bit rate but the other 30% use 40% or so higher bit rate you can't just use a lower VBR parameter, it's impractical to be switching back and forth VBR parameters, having to change encoders would prove my point
Moderators: Please change "[JAZ]" his user name breaks the forum system for quoting "[" "]" should be illegal.
Easy westgrove. Let's not start the name calling on page one.
westgroveg, the settings and encoders chosen for this test are based on a large number of samples and they result in ~128 kbps. As others pointed out, if your goal is to reach a target bitrate for each sample individually rather than a quality level, use ABR or even CBR.
I went ahead and consolidated all the samples into a zip file, as having to download and extract multiple files is something of an annoyance. Feel free to download that here (http://www.ron-jones.net/files/samples_all.zip).
I can mirror the original sample files as well if you'd wish, Sebastian.
No idea what you mean by this jerk off.
I was implying that you should know this test is about VBR encoders, not about ABR encoders.
Some encoders don't allow ABR.
And do they allow VBR? I mean: it probably is of no concern for this test.
If 70% of your files are at the target bit rate but the other 30% use 40% or so higher bit rate you can't just use a lower VBR parameter, it's impractical to be switching back and forth VBR parameters, having to change encoders would prove my point
You didn't get it. There are easy to encode files, and difficult to encode files. If your collection has mostly difficult to encode content, you would set yourself to a lower setting and be fine (*if your main concern is bitrate*). Else, you would have to accept that they are difficult to encode.
Again, the goal of VBR is to keep the quality constant, not the bitrate. If you need an ABR setting, use an ABR setting. We are evaluating VBR settings/encoders in this test.
Moderators: Please change "[JAZ]" his user name breaks the forum system for quoting "[" "]" should be illegal.
Isn't that a bit harsh on your side? I am a user of this site since 2002. I've been using this nickname since 1999 (or even 1998, can't remember). I wouldn't mind changing it to something else (Since I don't use it any longer on new places), but directly asking moderators to change it on my behalf because it annoys you when quoting me... It's not exactly keen, you know.
Also, recently this problem was brought up, and it is fixed in a more recent version of this board software, but the administrators won't update it yet (since it has many custom changes that have to be adapted)
Hi,
Why V5.7?
I've read it but I can't remember where nor what it said...
Nice work Sebastian et al
I think encoders that can maintain quality and also keep a more constant bit rate should be noted in a 128kbps test, because they are superior, if I'm aiming for 128kbps and I get 200+ I switch encoder.
Like you knew, right?
No idea what you mean by this jerk off.
Maybe if you didn't flip the order of all the sentences around you wouldn't have so much trouble reading them?
Hi,
Why V5.7?
I've read it but I can't remember where nor what it said...
Nice work Sebastian et al
LAME 3.98 presets use a slightly higher bitrate compared to 3.97.
I think encoders that can maintain quality and also keep a more constant bit rate should be noted in a 128kbps test, because they are superior, if I'm aiming for 128kbps and I get 200+ I switch encoder.
Like you knew, right?
No idea what you mean by this jerk off.
Maybe if you didn't flip the order of all the sentences around you wouldn't have so much trouble reading them?
I didn't flip the order the sentence Micky, I left irrelevant information out
[JAZ] didn't mean to go a lil' crazy at you, the thing about your user name, it's just that people can't quote you without manually editing code, but the mods don't seem to mind, truth is the forum system shouldn't allow any user to system characters in their name.
Did you not read the post where we said that this exact issue has been addressed in a more recent version of this forum software? Please don't blame the "system" for this.
Back on-topic: I think I'll sit this test out because it's painfully obvious that I need more ABX training. I can only tell a difference on one sample, which is most likely the low anchor. Can't wait to read the results, though!
Sebastian, keep the good work.
A listening test like this is so huge to prepare and execute, I even faint when thinking of it.
Looking forward to see the results.
I wonder if LAME 3.98.2 will ever be threatened as the current recommended encoder!
Note: Unfortunately my ears will not reach the category of "gold" until I fix my Class II malloclusion, at least more 1 1/2 year to go (Currently a cross-bite is disturbing the eustachian tube wich disturbs hearing). So sorry for not participating in this test!
Back on-topic: I think I'll sit this test out because it's painfully obvious that I need more ABX training. I can only tell a difference on one sample, which is most likely the low anchor.
Did you use headphones? If yes, what model?
Sebastian, keep the good work.
A listening test like this is so huge to prepare and execute, I even faint when thinking of it.
Looking forward to see the results.
I'll second that. Hats off for Sebastian.
I wonder if LAME 3.98.2 will ever be threatened as the current recommended encoder!
I'm afraid I expect a statistical tie between LAME 3.97, 3.98, Fraunhofer, perhaps iTunes too. Heck, maybe even Helix may not escape the tie's statistical error margin. In other words: please contribute your results, people!
I've just a had areally quick listen to the samples and I'll do the test soon.
Wow from the quick test just one encoder is really horrbile bad. I suppose it's the low anchor. Is that l3enc low anchor actually broken or something? I've never heard anything as bad as that.
Wow from the quick test just one encoder is really horrbile bad. ...
Just one? At least on some of the samples there is another one (not necessarily always the same one) which is pretty nasty as well.
As for the other encoders I'm having a hard time to differentiate them as far as the samples are concerned I tried so far. Differences are so subtle that I'm pretty uncertain whether they're just imagination. So my result often is 5.0.
Oh yeah I'm sure they'll be other artifacts, like I said I haven't had a full listen yet. But there is one encoder there that is just so bad it really stands out on every sample, like you can notice it immediately even if you're not listening with any attention. That surprised me a bit.
But there is one encoder there that is just so bad it really stands out on every sample, like you can notice it immediately even if you're not listening with any attention. That surprised me a bit.
ReallyRareWares states that it's "The First Ever publicly available MP3 Software Encoder" (Date: 1994-03-16). Legendary quality, you see.
Oh yeah I'm sure they'll be other artifacts, like I said I haven't had a full listen yet. But there is one encoder there that is just so bad it really stands out on every sample, like you can notice it immediately even if you're not listening with any attention. That surprised me a bit.
Sounds like its l3enc, that encoder really sucks at 128; but impressive at 192 for its time. But hearing the l3enc encode of castnets really does hurt my ears .
First, I would like to thank Sebastian and Alex for their work in this test. Greatly appreciated.
This is the first time I do an ABC/HR test (though, I'm familiar with the ABX one).
Right now, I've just tested the sample01 and saved my results.
I just want to know if there's a way
I can see them?
In other words, is there a way I can read the content of the .erf file I've just created?
Anyhow, I plan on testing the 14 samples and will mail them asap.
I wonder if LAME 3.98.2 will ever be threatened as the current recommended encoder!
Me too! Even though version 3.97 is already a great one IMO.
First, I would like to thank Sebastian and Alex for their work in this test. Greatly appreciated.
This is the first time I do an ABC/HR test (though, I'm familiar with the ABX one).
Right now, I've just tested the sample01 and saved my results.
I just want to know if there's a way I can see them?
In other words, is there a way I can read the content of the .erf file I've just created?
Anyhow, I plan on testing the 14 samples and will mail them asap.
Thanks for your kind words.
Once the test is over, Sebastian will publish the key file and you can process your result files, so keep them saved.
One thing that has not been separately mentioned or discusssed now is the tester's name. You can add your nickname to "Show name in results file". I'd recommend adding it unless you really want to send anonymous results. It would help finding your results if you have questions about some particular samples after the test.
I'd like to also remind that if you need to have a break and close the ABC/HR program you can save the session and reopen it, but you can't start from the beginning (i.e. reopen the ecf file) and quickly apply the ratings for the already tested samples because each time the ecf file is opened the samples will be in a different order.
While testing it happens rather often that I find differences subtle, and I'm not sure whether they really exist. So I'd like to try to ABX the difference for the encoder just under consideration.
The ABX button is in the program's head part so it doesn't invite me to use it in the situation described.
Can I use it? In case I can, how can I make sure that I ABX the encoder under consideration?
Once the test is over, Sebastian will publish the key file and you can process your result files, so keep them saved.
Thanks for your clarification.
You can add your nickname to "Show name in results file". I'd recommend adding it [...]
OK, will do.
While testing it happens rather often that I find differences subtle, and I'm not sure whether they really exist. So I'd like to try to ABX the difference for the encoder just under consideration.
The ABX button is in the program's head part so it doesn't invite me to use it in the situation described.
Can I use it? In case I can, how can I make sure that I ABX the encoder under consideration?
You can easily select the sample:
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/abchrjavaabx.png)
After a succesful ABX test the original sample for the tested encoder will be locked to 5.0 in the main window.
You can easily select the sample: ...
Thanks a lot.
Just to be sure: 'sample' here refers not to the 14 listening test samples, but to the 6 mp3/original pairs under consideration with 1 listening test sample. Is this correct?
Just to be sure: 'sample' here refers not to the 14 listening test samples, but to the 6 mp3/original pairs under consideration with 1 listening test sample. Is this correct?
Yes. You are testing only the samples that are included in the opened test configuration file. The program does not access the 13 other test configuration files anyhow.
In my screenshot I was testing the configuration file #9. I was unsure about the contender/encoder #3. I didn't move the sliders in the main window before trying the ABX test.
After you have succesfully ABXed all contenders you can compare the encoders with each other and reconsider the final ratings. In case you can't ABX a particular encoder you should not move the sliders from 5.0.
Just to be sure: 'sample' here refers not to the 14 listening test samples, but to the 6 mp3/original pairs under consideration with 1 listening test sample. Is this correct?
Yes. You are testing only the samples that are included in the opened test configuration file. ... In case you can't ABX a particular encoder you shoud not move the sliders from 5.0.
Thanks a lot for making things totally clear. I'll throw away my results obtained so far (with 5.0 score most of the time), and start anew with ABXing taking heavy part in it.
As I had Java crashes when saving results or sessions, I wrote down the (slider) results on paper before saving, just in case I had a crash. I figured that in case of a crash I just could open again the ABC/HR Config of the same sample and save results again. Luckily only on the 5th sample I noticed that when you reload the ABC/HR Config with the same track the sample orders have changed so my results written on paper didn't correspond anymore.
So I downgraded the Java version and after 4 tests I noticed I forgot to fill in my name. But I have saved the sessions. So my question is if it's OK to just open/load the saved ABC/HR Session, put in my name and save the results again (and overwrite the former result file)?
Thx
So I downgraded the Java version and after 4 tests I noticed I forgot to fill in my name. But I have saved the sessions. So my question is if it's OK to just open/load the saved ABC/HR Session, put in my name and save the results again (and overwrite the former result file)?
As I said earlier, it is safe to save a session and reopen it. You can add your name and continue testing after reopening a saved session.
Thanks for the feedback you guys (Alex B and others) provide. I usually get home after 5 pm that is why I cannot reply to the questions myself.
First results arrived today!
Yay, first results for all samples received today. One encoder is quite surprising so far.
Yay, first results for all samples received today. One encoder is quite surprising so far.
Which one?
Patience, young padawan...
Patience, young padawan...
LOL
I've just posted my results and one encoder surprised me too (I mean for the potential third place)
I downloaded ABC/HR, I also downloaded a sample package "Sample07.zip". I ran Sample07.bat, then executed abchr.jar, clicked Open ABC/HR config, loaded Sample07.ecf.
The problem: Clicking play on any sample doesn't produce any sound.
Problem #2. This sample package happened to be the eig sample, which produces easily noticable artifacts. Just out of curiosity, I tried to encode the sample with FHG surround encoder "-br 0 -m 4 -q 1 -vbri -ofl" and compared my encoding to the test samples. It sounded different from all of them, and it shouldn't.
The problem: Clicking play on any sample doesn't produce any sound.
Options > Settings > Playback > Playback Device and then choose the right one for you.
EDIT:
I also downloaded a sample package "Sample07.zip [...] This sample package happened to be the eig sample.
Weird, mine is Suzanne Vega, not eig.
I downloaded ABC/HR, I also downloaded a sample package "Sample07.zip". I ran Sample07.bat, then executed abchr.jar, clicked Open ABC/HR config, loaded Sample07.ecf.
The problem: Clicking play on any sample doesn't produce any sound.
From the presentation page:
Please note: if you don't hear any sound when using ABC/HR, simply change the device from ABC/HR's settings. In some cases, ABC/HR might not default to the primary audio driver.
This sample package happened to be the eig sample
Impossible - eig isn't even used in this test.
My sample07 is Suzanne Vega (Tom's Diner) as well.
Just downloaded Sample07 from both mirrors and it's definitely Tom's Diner. As I mentioned already, eig isn't even featured in this test.
The samples included are (alphabetically sorted):
atrain
Castanets_Original
chariotsoffire
fatboy_30sec
finalfantasy
Hypnotize
inthenight
Layla
linchpin
macabre
sfbay
TomsDiner
velvet
Waiting
Edit: Wondering why IPB sent me 3 mails for the same reply from Bodhi.
Problem #2. This sample package happened to be the eig sample, which produces easily noticable artifacts. Just out of curiosity, I tried to encode the sample with FHG surround encoder "-br 0 -m 4 -q 1 -vbri -ofl" and compared my encoding to the test samples. It sounded different from all of them, and it shouldn't.
I tried to encode samples #2 and #12 myself. My files were bit-identical to downloaded ones. (By the way, l3enc encodes raw files, not wavs). And sample 7 is definitely "Tom's Diner", not "Everything is green".
My Sample 7 file is Tom's Diner anyway and not eig. I have to admit i fail to ABX Tom's Diner on the non low anchor encodes (l3ence is very easy to ABX though). And now for some reason after i tried to ABX Tom's Diner. i need to watch a episode of Seinfeld .
If one encoder SURPRISES you then this definitely should not be LAME, because if it were it, it wouldn't be a surprise, would it?
... I have to admit i fail to ABX Tom's Diner on the non low anchor encodes ...
I have to admit that there are many samples where only the very obvious low anchor has an issue for me. Shows that even 128 kbps mp3 usually is fine (to me), and it's personal properties to which kind of artifact we are sensitive.
I have to admit that there are many samples where only the very obvious low anchor has an issue for me. Shows that even 128 kbps mp3 usually is fine (to me).
Me too.
[...] and it's personal properties to which kind of artifact we are sensitive.
I’m more sensitive to warbling, hi-hat chirping and warped ride cymbal (I play the drums) and also distorted voices or trumpets but not so much to pre-echo.
P.S. : I really liked the Vangelis sample, interesting one IMO.
I have to admit i fail to ABX Tom's Diner on the non low anchor encodes (l3ence is very easy to ABX though).
That's surprising. There's another Tom's Diner sample that can be easily distinguished.
IMHO, it has obvious problems and ABXing is not necessary. The four other samples are closer to the original, but not fully transparent.
I wonder if you simply tried to hear other kinds of artifacts and didn't notice this problem because it is different.
Would you mind testing the two samples I uploaded here: http://www.hydrogenaudio.org/forums/index....showtopic=66815 (http://www.hydrogenaudio.org/forums/index.php?showtopic=66815)
I have to admit i fail to ABX Tom's Diner on the non low anchor encodes (l3ence is very easy to ABX though).
That's surprising. There's another Tom's Diner sample that can be easily distinguished.
IMHO, it has obvious problems and ABXing is not necessary. The four other samples are closer to the original, but not fully transparent.
I wonder if you simply tried to hear other kinds of artifacts and didn't notice this problem because it is different.
Would you mind testing the two samples I uploaded here: http://www.hydrogenaudio.org/forums/index....showtopic=66815 (http://www.hydrogenaudio.org/forums/index.php?showtopic=66815)
I have found some issues with that sample. The mp3 version has a skratchy artifact on the vocals, abit like fatboy at V5.
foo_abx 1.3.3 report
foobar2000 v0.9.6 beta 1
2008/10/27 10:30:58
File A: C:\Downloads\sample07_source.flac
File B: C:\Downloads\sample07_mp3.flac
10:30:58 : Test started.
10:31:08 : 01/01 50.0%
10:31:11 : 02/02 25.0%
10:31:16 : 03/03 12.5%
10:31:19 : 04/04 6.3%
10:31:22 : 05/05 3.1%
10:31:30 : 06/06 1.6%
10:31:37 : 07/07 0.8%
10:31:42 : 08/08 0.4%
10:31:48 : 09/09 0.2%
10:31:53 : 10/10 0.1%
10:32:00 : 11/11 0.0%
10:32:05 : 12/12 0.0%
10:32:11 : Test finished.
----------
Total: 12/12 (0.0%)
Just downloaded Sample07 from both mirrors and it's definitely Tom's Diner. As I mentioned already, eig isn't even featured in this test.
OK, sorry then, I must have gotten a wrong (old) sample... Thanks for the help.
I have found some issues with that sample. The mp3 version has a skratchy artifact on the vocals, abit like fatboy at V5.
The sample is a direct clip from one of the test samples (and it is not from the low anchor).
I have found some issues with that sample. The mp3 version has a skratchy artifact on the vocals, abit like fatboy at V5.
The sample is a direct clip from one of the test samples (and it is not from the low anchor).
I've just made an ABC/HR test on sample07 and I can only spot the one you're talking about right now (like /mnt, no problem ABX it) and the low anchor, but not the other four.
I have found some issues with that sample. The mp3 version has a skratchy artifact on the vocals, abit like fatboy at V5.
The sample is a direct clip from one of the test samples (and it is not from the low anchor).
That was embarrassing , i did not know what was really wrong with that sample, i knew its imfamus for tuning. When i was trying to ABX Tom's Diner last week; I thought all the non low anchor samples sounded ok.
Normally we should not discuss about the test samples before the test is over, but I think this is a special case. The problem is quite obvious and it's worrying if someone cannot spot it.
So please be careful when starting a new test sample. Try to clear your mind from the previously listened samples and the possibly different problems you may have spotted. It might be a good idea to have a short break and do something completely different before opening the next test configuration file.
In case I know what to listen for (and I'm sensitive for the kind of problem with the sample) in many cases it's rather easy to ABX the differences.
With an unknown sample it's a lot more difficult. It's also a question of how much time is used on the sample. I remember having considered the harp40_1 Helix ~200 kbps result perfect, whereas after another day of intensive listening tests with other encoders at even 320 kbps I could easily distinguish all the mp3 results from the original. We are able to learn. But is hard learning significant? We do not listen to music this way when enjoying it.
So I think if we have given care with the listening test (while avoiding spending hours on a sample) we should consider an encoder sample transparent if we can't hear a difference. We should accept the fact that this may be just our (maybe temporary) inability to hear the issue. We shouldn't feel challenged too much in finding differences. A little bit of a challenge is necessary to do a good job, but we shouldn't go very far. After all it's about perfectness of encoders under real life conditions, especially at 128 kbps.
Adding to halb's comments:
This may also be relevant to testing-bias. In tests like these, we focus on short samples, with each of them typically representing one single issue. But the purpose of doing that isn't knowing how transparent those test-samples sound like - the purpose is extrapolating the scores to generic encoder quality (so, music in general). As we know, tuning a lossy encoder also is a matter of balance - what improves one issue, may worsen other issues. It is therefore very important, that the test-samples cover a wide range of "scenarios" evenly. Else, the following could happen: testers focus on specific issues and get accustomed to noticing them. The encoder then gets tuned to improve those cases, while unintentionally creating new different problems... but since the testers are focussed on the biased sample pool, the regression stays unnoticed. As visible in this thread, this may happen even if the testers also listen to music with those regressions, but don't recognize them because of being focussed on looking for issues related to the biased sample pool.
Many of us already knew that a MP3 VBR test @ 128 kbps would be quite difficult for the testers. I'd say that all contenders are usable for casual listening. Spotting the small differencies and occasional obvious artifacts needs concentration.
Particularly in this test we should not too easily let the contenders to pass any of the tests "as transparent" if we want to get any meaningful test results. Otherwise the test would be unnecessary. We could just say that VBR MP3 @ 128 kbps is good enough for casual listening and there is no matter which encoder is used.
So I think if we have given care with the listening test (while avoiding spending hours on a sample) we should consider an encoder sample transparent if we can't hear a difference.
I think at least half an hour would be a reasonable time for each test configuration. That would give only 5 minutes on average for each encoder. It is useful to try several different short passages when trying to ABX them and that takes some time.
I spent about an hour with Tom's Diner. The four near transparent encoders were difficult to ABX. I have yet to decide if I give them all the same rating or if I try to put them in some order.
Particularly in this test we should not too easily let the contenders to pass any of the tests "as transparent" if we want to get any meaningful test results. Otherwise the test would be unnecessary. ... I think at least half an hour would be a reasonable time for each test configuration. ...
Sure we shouldn't let the contenders pass too easily, and about half an hour per sample with all 6 encoder results sounds reasonable to me. In case there are suspicious spots 5 minutes per encoder result quickly pass by. But I admit in those cases where I didn't even have a suspicion I didn't spend half an hour.
BTW I'm just through with the first 9 samples, and my primary concern is with having a really silent home when I'm testing and being not pressed to finish a test because of other things waiting for me. So usually I manage to go through 1 or 2 tests per day.
Patience, young padawan...
sample12: for a change a sample with 4 encoders having obvious issues. What a relief!
sample12: for a change a sample with 4 encoders having obvious issues. What a relief!
Thats fatboy, alot of codecs struggle at the vocals on this sample and very easy to ABX.
Just sent my results. I had a nice and sometimes hard time spotting problems of this nice selection of samples. Curious to see that a few samples were easy to ABX on speakers (no matter what encoder excluding low-anchor). But generally spoken using headphones is a must.
I want to thank those who made possible this mp3 listening test and especially Sebastian.
Having trouble running the test here. I've downloaded the zip with abc-hr and java (from the link in the readme). The directory structure is preserved, I can unarchive samples no problem using the appropriate .bat file. Then it breaks.
I get this at start up.
D:\Program Files\Java\jre1.5.0_15\bin>java -jar E:\ABC-HR\abchr.jar
java.io.FileNotFoundException: clicodecs.cfg (The system cannot find the file sp
ecified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at abchr.settings.DecoderSettings.<init>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at abchr.gui.ProjectFrame.<clinit>(Unknown Source)
at abchr.gui.Main.main(Unknown Source)
and this, when I press play.
Exception in thread "AWT-EventQueue-0" java.lang.IllegalArgumentException: Line
unsupported: interface SourceDataLine supporting format PCM_SIGNED 44100.0 Hz, 1
6 bit, stereo, 4 bytes/frame, little-endian, and buffers of 88200 to 88200 bytes
at com.sun.media.sound.PortMixer.getLine(Unknown Source)
at abchr.audio.PlaybackThread.play(Unknown Source)
at abchr.gui.PlayAction.actionPerformed(Unknown Source)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.AWTEventMulticaster.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
What should I do?
I think I know what the problem is... What happens if you do this:
cmd
E:
cd \ABC-HR
D:\Program Files\Java\jre1.5.0_15\bin\java.exe -jar abchr.jar
I think I know what the problem is... What happens if you do this:
cmd
E:
cd \ABC-HR
D:\Program Files\Java\jre1.5.0_15\bin\java.exe -jar abchr.jar
nice, that fixes the first problem (at start up), but the play problem is still there.
I don't know if this might help but I explain what I did. I deinstalled Java v1.6, rebooted PC and then installed Java v1.5 and rebooted again. The .jar files are associated on my Windows VistaSP1 so I just have to doubleclick on abchr.jar to get ABC/HR running. With Java v1.5 all went OK.
I think I know what the problem is... What happens if you do this:
cmd
E:
cd \ABC-HR
D:\Program Files\Java\jre1.5.0_15\bin\java.exe -jar abchr.jar
nice, that fixes the first problem (at start up), but the play problem is still there.
The second problem might be caused by the wrong soundcard being selected. Can you please open the ABC/HR options and verify that your soundcard is selected and not something else?
The second problem might be caused by the wrong soundcard being selected. Can you please open the ABC/HR options and verify that your soundcard is selected and not something else?
ok then, all fixed I forgot I actually have 2 (forgot the on-board one)
Am I the only one to not be able to find the low anchor with the sample 11, or is there a problem with the pack? (i got it from the .be mirror)
I remember that with one sample (it will be sample 11 then) low anchor didn't have an obvious issue and it took me quite a while to find a problem. Once found it was rather easy to ABX.
I remember that with one sample (it will be sample 11 then) low anchor didn't have an obvious issue and it took me quite a while to find a problem. Once found it was rather easy to ABX.
I did notice that the low anchor of sample 11 is not far away from some of the other samples, but the last sample sounds almost bad as the low anchor IMO.
' date='Oct 30 2008, 21:39' post='596565']
Am I the only one to not be able to find the low anchor with the sample 11, or is there a problem with the pack? (i got it from the .be mirror)
The vocals at 0:08 on the low anchor is a dead give away, IMO the low anchor version of sample 11 is very competitive.
Particularly in this test we should not too easily let the contenders to pass any of the tests "as transparent" if we want to get any meaningful test results. Otherwise the test would be unnecessary.
The problem I'm having is distinguishing any differences whatsoever (save for the low anchor), even with prolonged listening at higher-than-"casual" volumes. There's always that danger of not being too sure about a particular sample and wishing to avoid rating the reference, so I wind up just choosing an appropriate rating for the low anchor and leaving the rest of the set "blank".
The 64kbps multiformat test was difficult enough for me, so I'm pretty sure I'm going to sit this one out. Would it be helpful, though, to submit results where only the low anchor is rated, or is that counter-productive?
Would it be helpful, though, to submit results where only the low anchor is rated, or is that counter-productive?
If you really cannot hear any differences after concentrating a bit, you can of course submit the results where only the low anchor is rated.
... I did notice that the low anchor of sample 11 is not far away from some of the other samples, but the last sample sounds almost bad as the low anchor IMO. ... The vocals at 0:08 on the low anchor is a dead give away, IMO the low anchor version of sample 11 is very competitive.
You can't be sure which is the low anchor then. It is possible, though not very likely, that a contender is worse than the low anchor.
I just tested the sample 11. I could easily spot two contenders. The other four were difficult or transparent to me. (A tip to anyone struggling with this sample: try to ABX the first 1.5 seconds.)
Sebastian,
Have you got lots of results? Maybe you should extend the test a bit (once again). This time we also have the Java problem. Some testers couldn't start testing before the solution was found and an earlier version was installed.
Personally I am through 10 samples. Naturally I want to test all samples, but I am now leaving for a weekend vacation. I would need only an extra day or two, but you should not end the test too early if there is any chance of getting more results.
Judging by the number of results, an extension is almost certain (unless everyone is waiting until Sunday to send the results) so no worries.
[The problem I'm having is distinguishing any differences whatsoever (save for the low anchor), even with prolonged listening at higher-than-"casual" volumes. There's always that danger of not being too sure about a particular sample and wishing to avoid rating the reference, so I wind up just choosing an appropriate rating for the low anchor and leaving the rest of the set "blank
That's why there's an integrated ABX tool in abc/hr. I've used it to verify that i'm hearing what i'm hearing.
I.e.: i tend to switch rapidly between the samples, until i find a suspecting problem. If i'm unsure of what i hear, i launch the abx and select that sample vs original. I then try to pick the difference, and produce a successful ABX.
Then, i go back and move the slider of the one showing that problem.
Other times i just listen to the reference sample several times, before listening to the lossy sample, in the attemt to memorize the sound.
I need at least twice as many results as I have now in order to have statistically valid data. The average number of results per sample is 7 and I would like to have 20.
Sample01: 11
Sample02: 7
Sample03: 6
Sample04: 7
Sample05: 7
Sample06: 6
Sample07: 7
Sample08: 6
Sample09: 7
Sample10: 6
Sample11: 6
Sample12: 9
Sample13: 6
Sample14: 7
Edit: Had to throw away some results as they contained ranked references (with a 2.0 score) and in some cases even multiple ranked references with different scores. Wondering if I should disqualify all results from that user.
The test was extended to end on November 8th, 2008.
There's one statistic which hasn't been considered yet, in any listening test I know of.
It would be interesting to know if the ability to ABX different encoders depends on the quality of the equipment, and what the nature of that dependency is. There's a tremendous number of people claiming that the quality of their mp3's etc. is not satisfying enough for their pro-audio gear, so one would assume that there's a positive correlation.
But, OTOH, balanced equipment is precisely the basis for the psychoacoustic models, and, I guess, to some degree also the preferred choice for listening tests by the developers. Contrary to cheap phones, for example, which can arbitrarily invalidate masking assumptions. According to this, there might instead be a negative correlation, i.e. it could mean that you can't distinguish anything above, say, 128kbps using a high-end setup.
Even if people merely submitted personal ratings of their equipment from low-end to ultra-high-end, such an analysis should be perfectly possible.
I need at least twice as many results as I have now in order to have statistically valid data. The average number of results per sample is 7 and I would like to have 20.
These are disappointing numbers so far. I hope many people wait until just before the deadline expires just like in school or college
Edit: Had to throw away some results as they contained ranked references (with a 2.0 score) and in some cases even multiple ranked references with different scores. Wondering if I should disqualify all results from that user.
Only allowing to move a slider after a successful ABX would be a solution.
Disqualifying is a difficult question. Ideally this should have been decided beforehand. You could look if the other results look meaningful (e.g. low anchor lowest quality). But that would introduce bias and probably wouldn't be too scientific.
Is the feature that the reference slider is locked after ABX anywhere documented? I find it very helpful but didn't really notice it until I read about it in this thread.
http://ff123.net/64test/practice.html (http://ff123.net/64test/practice.html) doesn't mention it. IIRC this feature wasn't available back then.
Edit: Had to throw away some results as they contained ranked references (with a 2.0 score) and in some cases even multiple ranked references with different scores. Wondering if I should disqualify all results from that user.
Maybe (just a guess!) this user did the same thing as Alexxander:
As I had Java crashes when saving results or sessions, I wrote down the (slider) results on paper before saving, just in case I had a crash. I figured that in case of a crash I just could open again the ABC/HR Config of the same sample and save results again.
but didn't notice that
when you reload the ABC/HR Config with the same track the sample orders have changed so my results written on paper didn't correspond anymore.
In this case - yes, it is better to disqualify all results from him/her.
Unfortunately, I received only one single result since my last post. If anyone finished the test already and is waiting for the deadline to come to submit results, please don't and send them ASAP.
Unfortunately, I received only one single result since my last post. If anyone finished the test already and is waiting for the deadline to come to submit results, please don't and send them ASAP.
And it's not necessary to test all samples, allthough it's preferible. If you only can test a few samples just pick some randomly between Sample01 and Sample14. It's better to test 3 samples and send the results then trying to do all and end up sending none. Send what you have, you can always send more later (before deadline).
Don't hesitate to ask if you don't get the setup right or have doubts working with the ABC/HR for Java program.
Sent in my results today.
Really tough to distinguish. I'm mostly untrained in spotting artifacts. The low anchor was generally easy to spot, but beyond that i could only rarely find a problem in any of the other samples.
Was the listing test announcement submitted to doom9, Slashdot, Digg, Reddit, Engadget, etc? Maybe that would help with the low number of samples.
doom9 yes, all others no. Roberto all sent the announcement to the people that subscribed to his listening test newsletter a while back.
IMHO, just extend the test. Maybe for weeks. The results submitted so far don't become invalid.
The JRE-issue may be one important reason for the poor feedback. I had to set up Ubuntu on a VM to get it work, but unfortunally i have not enough time to participate in the coming 2 weeks.
doom9 yes, all others no. Roberto all sent the announcement to the people that subscribed to his listening test newsletter a while back.
If you extend it for a couple of weeks, i can submit it to the sites.
I could extend it for let's say a maximum of two weeks. Do you guys think that would be OK?
Yes, extend it 2 weeks more. If it will lead to getting results from a few more users it will have impact at the total endresult. Participants should be made clear that not all samples have to be tested, we all have little spare time. It's better to send one ore a few results than none.
Test extended until November 22nd.
Test extended until November 22nd.
submited to:
reddit http://www.reddit.com/r/Music/comments/7c8...enning_test_at/ (http://www.reddit.com/r/Music/comments/7c8hs/participate_new_mp3128_kbps_listenning_test_at/)
digg http://digg.com/music/NEW_MP3_128_kbps_lis...t_HydrogenAudio (http://digg.com/music/NEW_MP3_128_kbps_listenning_test_at_HydrogenAudio)
slashdot (hope it appears soon)
doom9
engadget
lifehacker
gizmodo
techcrunch
etc.
I "did my part" and just contributed a sample (and ABX'd all the encoders, woot). I had a hard time ranking the encoders though, and my ears were not at all happy after everything was done. I think I'll limit myself to one sample a day in the future.
Here's an update of the number of valid results I have so far:
Sample01: 14 (+3)
Sample02: 8 (+1)
Sample03: 9 (+3)
Sample04: 10 (+3)
Sample05: 8 (+1)
Sample06: 7 (+1)
Sample07: 9 (+2)
Sample08: 8 (+2)
Sample09: 9 (+2)
Sample10: 8 (+2)
Sample11: 8 (+2)
Sample12: 10 (+1)
Sample13: 8 (+2)
Sample14: 9 (+2)
The numbers in the brackets show the number of additional results received since extending the deadline. Thanks to everybody who participated already.
And yeah, so far, results are very suprising.
Well, kwanbis got the word out, and Hydrogenaudio is now experiencing a Slashdotting... My my.
Now that it has appeared on /.:
Most comments so far are complaining that it's too complicated to set up. This might be a reason for so few responses so far.
To quote anonymous coward:
You know what, I thought I'd be nice and give this a shot, but the amount of effort involved just isn't worth it. If it isn't 'click on this link, listen, rate', it's too much work. Download x, install x, email x - way, way, way too much work for what is being given in return.
And they even haven't tried conducting the test yet which is even more work...
I hope there are at least some people who take the effort.
These tests are unusable in Linux, with jre-1.4, -1.5, and -1.6. What's the point in using Java, then? It seems filenames are inconsistent (expecting Sample01/Sample01.wav instead of Sample01/sample01.wv). Fixing that, I see another error:
Error
Config file could not be loaded.
file is not a supported file type
That's 55MB you won't be getting back.
These tests are unusable in Linux, with jre-1.4, -1.5, and -1.6. What's the point in using Java, then? It seems filenames are inconsistent (expecting Sample01/Sample01.wav instead of Sample01/sample01.wv). Fixing that, I see another error:
Error
Config file could not be loaded.
file is not a supported file type
That's 55MB you won't be getting back.
I don't understand what the problem is here. What do you mean with filenames are inconsistent? You have 14 ZIP files that you can download individually - nobody forces you to download all 55 MB and this is stated in the readmes. The ZIP consist of 6 MP3 files that represent the samples and one lossless WavPack (wv) file that is the reference. If your archiver doesn't mess up, you should end up with files that have the proper casing which is also used in the batch files.
Then what do you mean with that error? Where does that error come from and when? If you load an ecf file? Did you follow the readme file at all? Does the folder structure look like in folder-setup.png?
By the way, the reason why the whole test is not made in Flash is that Flash would require you to be connected to the Internet all the time. The current solution allows you to download the files at work or at an Internet cafe if you don't have broadband at home and do the test on your PC while you are offline. The resulting encrypted result files have only a few KB which should be no problem to submit from home then.
I agree, the test is a little hard to figure out. You really have to read the "read me" manual very closely. Especially if English isn't your first language I suppose.
For example: The sentence "Navigate to the "bin" directory and run "DecodeXX.bat" trew me off. It took me a while to understand that you first have to decode the samples to wav-files. I have never done this kind of decoding. Not hard to do, I just haven't done it before.
Anyway, I never got any farther than loading the samples in the ABC/HR program. Nothing happened when clicking the start buttons, on any of the samples, that's where I gave up. Sorry.
Please note: if you don't hear any sound when using ABC/HR, simply change the device from ABC/HR's settings. In some cases, ABC/HR might not default to the primary audio driver.
I don't understand what the problem is here. What do you mean with filenames are inconsistent? You have 14 ZIP files that you can download individually - nobody forces you to download all 55 MB and this is stated in the readmes. The ZIP consist of 6 MP3 files that represent the samples and one lossless WavPack (wv) file that is the reference. If your archiver doesn't mess up, you should end up with files that have the proper casing which is also used in the batch files.
Then what do you mean with that error? Where does that error come from and when? If you load an ecf file? Did you follow the readme file at all? Does the folder structure look like in folder-setup.png?
By the way, the reason why the whole test is not made in Flash is that Flash would require you to be connected to the Internet all the time. The current solution allows you to download the files at work or at an Internet cafe if you don't have broadband at home and do the test on your PC while you are offline. The resulting encrypted result files have only a few KB which should be no problem to submit from home then.
So I didn't know that wv was WavPack's extension, and this led to confusion. This was the first error I got:
Config file could not be loaded.
File Sample01/Sample01.wav could not be opened.
My only thought here is to check the permissions for the directory, and it is indeed writable. I downloaded all the samples because I wanted to, not because I'm stupid. I followed the instructions to the letter.
Guys, please understand that I am organizing these tests based on some given software. If the software is buggy, I have two options: either don't organize the tests at all or - like in this case - organize the test and inform people about possible problems. If I had the time, I would code something like ABC/HR myself. schnofler is busy with other things unfortunately and doesn't have time to look into the problem. My Java knowledge is very limited and I have no idea what is going wrong. I tried to build ABC/HR against JDK 1.6 but it didn't help - the hangs got shifted to other areas.
As for the complaints for decoding and downloading several packages: I could also make everything bundled in one package, but then people will complain that they have to DL 50 MB even if they only want to test two or three samples. Also, I could also provided the samples decoded to WAV already, but then the average package size won't be 4 MB, but 40 MB or even more. I also stated in my previous post why I don't offer an online based test environment. Beside the argument that you have to be online all the time, you also have to develop something like this first and I don't know Flash / Action Script at all.
My only thought here is to check the permissions for the directory, and it is indeed writable. I downloaded all the samples because I wanted to, not because I'm stupid. I followed the instructions to the letter.
You have to open the ecf file not the wav file.
Once ABC/HR is open, click "Open ABC/HR Config..." and load the file "SampleXX.ecf" (again, "XX" being the number of the test you want to take - make sure you ran the batch file respective to that package before).
You have to open the ecf file not the wav file.
Once ABC/HR is open, click "Open ABC/HR Config..." and load the file "SampleXX.ecf" (again, "XX" being the number of the test you want to take - make sure you ran the batch file respective to that package before).
This is exactly what I did. The error message reads, 'Config file could not be loaded'. And so, I left it implied that I was opening a config file.
Edit: I skipped step 4, which began 'WINDOWS USERS'. There are two subsections, and I am stupid.
Ah, OK, sorry. Hmm... And the folder structure is like on the picture? That is, you have a folder where abchr.jar is located and in that folder you have a subfolder called Sample01 where the decoded WAV files are?
Well, kwanbis got the word out, and Hydrogenaudio is now experiencing a Slashdotting... My my.
I hope it is for the good
Maybe we should be updating the 1st post with all the findings to help newbies.
The /.ers are having a field day it seems. I am glad the word got out there, but the ... complaints. I just finished the test fine, but it seems people are having troubles with it. I do think it would be a good idea doing a form of Flash application if at all possible in the future, even though you have a valid point of not having to be online to do the test. Maybe an option ?
I did like this reply though:
While popular music is acceptable at 128kbps with recent encoders, certain niche music genres like spectralist music clearly suffer at low bitrates. With pieces like Per Norgard's Symphony No. 3 [amazon.com] or Grisey's Les espaces acoustiques you can easily hear the difference between 256kbps and the original CD-quality on even average headphones or speakers. Any music which depends on a greater portion of the natural overtone series than just the first handful of partials will need higher bitrate encoding.
Oh well. Oh, and this one:
Umm... 128 Kbps? Seriously? And no Ogg Vorbis, AAC etc... If you're bothering to set up a listening test, why limit yourself to 128 Kbps MP3?
Also, this should really be set up as a blind test, you get to listen to two clips, and have to choose which is better. The clips are randomized, of course... I could go on, but I'd just make myself sound even more arrogant.
Arrogant indeed.
Thanks for putting the test together!
sizetwo.
I already feel sorry for Polar and Roberto for hosting the samples. If you guys get any trouble, please let me know.
why don't you upload all to www.mediafire.com as a mirror?
Is RapidShare an option?
Nobody's blaming you Sebastian, you are doing a great job organizing everything. Some of us are just pointing out that there are some things, or steps, that are a bit unclear, and maybe even mis-explained. You really want to get to the listening part as soon as possible, because you know that part alone is going to take a whole lot of time and effort. And when you run into a bunch of small trials and obstacles on the way there, it pisses you off, and discourages you.
I would really like to see an online, pre-set, interactive test. You could probably make it look just like the ABC/HR-interface. That interface could probably be saved and used in another test, just having the samples replaced. Would that be possible? I don't know. I'm talking about a future test now, I'm well aware of the fact it's a little late for this particular test.
Anyway, I have now passed all the hinders (thanx Seb!) and will hopefully be able to conduct a few tests pretty soon.
I understand your complaints and I agree that for a newcomer, the testing process is somewhat complicated. I am looking forward to simplifying it for upcoming tests, either by making the documentation easier to read and follow or by developing / using other software.
Anyways, just wanted to let you know guys that after an average of 2000 hits per day today I ended up with 11666. At 22 o'clock German time I had over 5000 already. So thanks for spreading the news. I am also happy to let you know that I've already received several valid results and sample 1 is only one result away from the "magical" 20 results.
Is RapidShare an option?
RapidShare is an option for sure, problem is that is realy slooooooow for non-premium users.
I rahter recommend mediafire and megaupload.
@Sebastian
I really sympathise with you. There's a neat, tight project going on, but not quite enough numbers. So you publicise it, and in come the hordes, not having a clue what they're doing or why they're doing it, or why it has to be hard work to get results. And, yes, I'm sure there are glitches, but people who care will work through them because the results matter, or at least take the time to have specific questions. And if in doubt, blame the poor sod who's organising it and doing more work than everyone else.
In other words, if you don't like this, for God's sake don't work in higher education.
I am really grateful for your work, and persistence. The only reason I haven't contributed is that I am sure my ears couldn't distinguish between anything half-way satisfactory at ~128.
So hang in there. Or walk away from it all. Whichever feels right.
Some comments:
I found this test through slashdot.
I use linux with JRE 1.6. ABC/HR works flawlessly.
Many people are saying the test has recevied poor results due to difficulties with setup and the like. I disagree. This is really very easy if you just take the time to read the short instructions.
That said, I don't believe this test is for everyone. It is VERY difficult to pick out aural differences in the samples. I have tested 1, 2, 6 ,13, and 14, and have found only 4 or 5 samples that I can verify via an ABX test, not counting the low anchor. I would not want someone casually performing these tests, as it takes time and patience to truly do this right.
I hope to finish the entire set before the 22nd, but this stuff is grueling!
Thank you for contributing Peter. Yes, blind testing is hard. Who'd have thought 128kbps MP3 could sound this good, hey?
ok, I just spent about 10 minutes on each set, on average, with certain sets getting a LOT more time. I've found the anchor in every set, and defects in about half. It was practically impossible for me to discrern any differences in the louder samples.
I only found 5 or 6 really obvious defects throughout all 14 sets. In some sets, I would find some defects that were so minor, it took me dozens of plays to convince myself it was real, even though I could ABX them >90% of the time. Problem is, I'm worried there are other defects this minor that I just didn't have the time or discipline to catch. I would listen to 2 second excerpts over and over again to find them, and I can't do that with every 2-second slice of each sample of each set. I'm worried that my "stumbling" upon a tiny defect will make one encoder look worse than it really is since I didn't have the time to find similiar defects in the others.
Is the idea that this stuff should even out since everyone else is in the same boat? Or is every else taking this way t0o seriously like I am?
I don't want to drag this thread too off-topic, but am I missing something with Slashdot comments? I don't use the site regularly, and I find navigating the comments near impossible. I posted two comments eight hours ago (before bed) and it took me around fifteen minutes this morning to find one of them (still haven't found the other). What's with all this unhiding? It doesn't seem to work very well. Maybe I just don't get the logic behind it; it's very confusing for a simple HA user like myself.
Edit: Even though I'm not a big RSS user (5' 8") I think I'd appreciate an RSS feed as opposed to the page "layout".
I don't want to drag this thread too off-topic, but am I missing something with Slashdot comments? I don't use the site regularly, and I find navigating the comments near impossible. I posted two comments eight hours ago (before bed) and it took me around fifteen minutes this morning to find one of them (still haven't found the other). What's with all this unhiding? It doesn't seem to work very well. Maybe I just don't get the logic behind it; it's very confusing for a simple HA user like myself.
Slashdot uses a user moderation system. Posts are ranked from 1 to 5. You choose what "level" of posts to see. Selecting a higher number means you will only see posts that other users consider useful,select a lower number to see more posts. By default, new users' posts are given a 1. As other users see your posts, they may decide your comment is useful and mod you up. Over time, if you get a lot of positive feedback, your default may rise to a 2 or 3.
The idea is to prevent meaningless chatter and allow intelligent discourse to rise to the top, but honestly, slashdot isn't that cool any more. Its kind of sad.
Slashdot hasn't been cool since 2002 - merely serviceable. The comment system used to be barely comprehensible, but the AJAX work they've been doing has been horrendous from what I've seen today.
I'm more saddened by the lack of knowledge displayed by a lot of the commentators.
OK, many thanks for the info guys. I don't want this to go any further off-topic.
I've actually made it slightly more useful (on quick inspection) by registering and setting some options. At least it seems I don't have to spend ten minutes unhiding things now.
While I'm here: congratulations to Sebastian, and thanks to kwanbis for his help in stirring up a little more interest. I guess this whole thing is about altering people's misconceptions, and it seems like there's a few on Slashdot who could learn a few things.
I posted on PSW (http://recforums.prosoundweb.com/index.php/t/25781/16937/)and on Stereophile (http://forum.stereophile.com/forum/showflat.php?Cat=0&Number=52848&an=0&page=0#Post52848).
Is the idea that this stuff should even out since everyone else is in the same boat? Or is every else taking this way t0o seriously like I am?
The way I understand it is this: Since the sample order is random in the test you shouldn't be able to prefer certain codecs "by accident", so in the long run all of them will be tested equally well. The less careful the test is conducted by the participants, the more results it will take to arrive at statistically meaningful results though, especially for good codecs which will probably be crammed closer together near 5.0.
ok, I just spent about 10 minutes on each set, on average, with certain sets getting a LOT more time. I've found the anchor in every set, and defects in about half. It was practically impossible for me to discrern any differences in the louder samples.
I only found 5 or 6 really obvious defects throughout all 14 sets. In some sets, I would find some defects that were so minor, it took me dozens of plays to convince myself it was real, even though I could ABX them >90% of the time. Problem is, I'm worried there are other defects this minor that I just didn't have the time or discipline to catch. I would listen to 2 second excerpts over and over again to find them, and I can't do that with every 2-second slice of each sample of each set. I'm worried that my "stumbling" upon a tiny defect will make one encoder look worse than it really is since I didn't have the time to find similiar defects in the others.
Is the idea that this stuff should even out since everyone else is in the same boat? Or is every else taking this way t0o seriously like I am?
Exactly the same here, my method is to listen to the low anchor, find the part of the sample where it REALLY fails, then listen to that excerpt with the other codecs to find artifacts. They all tend to exhibit the same failure modes, just to differing degrees. Some of the louder samples, I agree, are utterly impossible to abx at all.
Sam
But don't forget to also listen the whole sample for all encoders. It might happen that some encoders have artifacts at places where the low anchor is not so bad.
Exactly the same here, my method is to listen to the low anchor, find the part of the sample where it REALLY fails, then listen to that excerpt with the other codecs to find artifacts. They all tend to exhibit the same failure modes, just to differing degrees.
I doubt it. Different codecs have different psymodels, and why should they produce artifacts in the same places?
Is RapidShare an option?
Sebastian, i uploaded all samples to megaupload (to my premium account): http://www.megaupload.com/?d=13B7NWEP (http://www.megaupload.com/?d=13B7NWEP)
Maybe you can post it as a link in the 1st post.
...Many people are saying the test has recevied poor results due to difficulties with setup and the like. I disagree. ....
I totally agree with you. Though I did have some problems setting up the test I think it at least on a Windows system and for people doing the test when reading this thread the setup shouldn't be a major problem. I also agree that it's the listening test itself which is the real challenge, and I had a hard time finding deviations from the original.
...Many people are saying the test has recevied poor results due to difficulties with setup and the like. I disagree. ....
I totally agree with you. Though I did have some problems setting up the test I think it at least on a Windows system and for people doing the test when reading this thread the setup shouldn't be a major problem. I also agree that it's the listening test itself which is the real challenge, and I had a hard time finding deviations from the original.
Uh say what?! How the heck can you say you disagree with people who say they find the test difficult to set up? If people find it difficult, then they do! Even Sebastian admitted there were some things which weren't totally clear.
Uh say what?! How the heck can you say you disagree with people who say they find the test difficult to set up? If people find it difficult, then they do! Even Sebastian admitted there were some things which weren't totally clear.
We are talking about slashdoters. They are (where?) supposed to be more or less knowledgeable guys.
Uh say what?! How the heck can you say you disagree with people who say they find the test difficult to set up? If people find it difficult, then they do! Even Sebastian admitted there were some things which weren't totally clear.
We are talking about slashdoters. They are (where?) supposed to be more or less knowledgeable guys.
Well, I was one of those guys who found the tutorial a bit awkward. And then when someone says he disagrees that I find it to be a hassle to go through, it kind of ticked me off. That is not debatable really.
Once done, it won't be a problem I guess, but for first timers, and for people who have English as the second (or third) language, you REALLY have to pay attention to what you are doing. So that will probably discourage quite a few people from even wanting to participate. I think we have seen that already (i.e. extended test)
Thanks a lot to the people who submitted. The people who did not start testing already but who would like to participate, it would be very nice if you could focus especially on samples: 5, 6, 8, 9, 10, 11, 13, 14.
Sample 1 already has 25 valid results, so please do not test it (of course, if you already did, you can submit your results for it, but if you just started, maybe invest time and effort in testing the aforementioned samples).
Thanks for providing the test. I've submitted only 2 samples. I found no trouble setting this up in Linux, besides needing to set my soundcard in settings.
The test itself was taking more time than I would have guessed. And then I skipped the difficult samples.
With headphones on I noticed some serious errors though that are easily detected by carefull listening. Only two codecs seem to do a real good job. My guess is that that's lame in both cases. But who knows what surprises this test will bring. With the new deadline, I'll see if I can do one or two samples extra.
Updated number of results per sample:
Sample01: 26
Sample02: 17
Sample03: 17
Sample04: 18
Sample05: 16
Sample06: 15
Sample07: 19
Sample08: 16
Sample09: 14
Sample10: 16
Sample11: 15
Sample12: 18
Sample13: 16
Sample14: 16
Hi, this is the first time I've done one of these - I'm going through every sample (I'm unemployed right now so have a lot of free time lol ) and am finding it very interesting yet challenging. I'm sure my mediocre setup isn't helping things (noisy onboard sound + Sennheiser CX300s) - there's been a few samples when I couldn't tell any difference between the non-l3enc samples.
Just a question regarding making comments - will you know what way our samples were numbered when you look at the results? I'm just wondering because I've said made some comments comparing samples.
If I mistakenly identify a reference file as the compressed one, will my results be still valid for the other (correctly identified) samples of the set?
Just a question regarding making comments - will you know what way our samples were numbered when you look at the results? I'm just wondering because I've said made some comments comparing samples.
Yes.
If I mistakenly identify a reference file as the compressed one, will my results be still valid for the other (correctly identified) samples of the set?
It depends on the number of ranked references. If you submit 5 results and 4 of them contain ranked references, I am afraid I will have to throw away all of them because I cannot be sure if the single one where you didn't rate the anchor wasn't only a coincidence.
The best thing you can do when you want to rate a sample is to perform an ABX test between the reference and the respective sample. If you get at least 7/8, ABC/HR will automatically disable the slider for the reference so you cannot rate it by mistake. If you cannot hear any difference in ABX, then it also doesn't make sense to rate that sample at all because ABC/HR already demonstrated you that the sample is transparent for you.
Thanks.
I've downloaded several of the samples. I can always spot the low anchor, but I'll be damned if I can hear a difference from the reference in any of the other encoders.
Could someone maybe tip me off as to which sample is easist to ABX, and what I should be listening for?
The differences can be found in more complex sounds or where different sounds overlap, especially when containing high frequencies. Try Sample05, between 2.0 and 16 secs there are several sounds with distortions.
I wonder if it's fair to give hints for a listening test
Better no, please.
Listen to the low anchor sample to spot artifacts' positions.
As the low anchor's artifacts often seemed to massively distort entire samples without regards to any particular sound or location, it didn't really help me much.
With HD-600 headphones and a headphone amp the difference on some of the castanet samples was so faint I often wasn't really sure the difference I was hearing was real or my imagination, but I could reliably ABX them (I think I scored somewhere around 23 out of 30 on average.)
Oh, and the l3enc version caused actual physical discomfort on some samples... MP3 has came a LONG way!
Wow, do you test each sample 30 times? I'm a lot lazier and don't go further than 8 times. It's just enough to pass the test with a margin of one error.
Since the final deadline is approaching, I would like to ask you to slowly think about submitting the results if you haven't already.
Any update to your Nov 11 results submission count?
I received some results, but I didn't check if they are valid or not.
I am not going to update the counters as it doesn't make any sense now anyways - the final deadline remains November 22nd because I don't have time after that date to process and prepare the results if I extend once more and also, having a test last several months is boring.
Hello,
Thanks for you work, but I have a problem.
I clicked on Save ABC/HR Session and the Java Application is not responding anymore...
What should I do?
Regards,
Naunaud.
from the first post :
Warning: There seem to be problems with ABC/HR when JRE 1.6 is installed. If you are affected by hangs, crashes or any other difficulties, please download and install JRE 1.5 Update 15 from http://java.sun.com/products/archive/j2se/5.0_15/index.html (http://java.sun.com/products/archive/j2se/5.0_15/index.html). You do NOT have to uninstall your existing copy of JRE 1.6 since both versions can be installed at the same time. However, if you have JRE 1.5 and JRE 1.6 installed, please make sure you start ABC/HR with JRE 1.5 manually. Since the working directory has to be the location where abchr.jar is stored, the best thing you can do is to open a command prompt window, navigate to the location where you stored abchr.jar and the rest of the files and then call "C:\Program Files\Java\jre1.5.0_15\bin\java.exe -jar abchr.jar". Of course, this is only an example which you have to adapt according to where java.exe is stored on your PC.
Group: Members
Posts: 1
Joined: 5-June 05
Member No.: 22536
Wow, you joined almost 3 years ago, it is your first post!
Yaaaay!
from the first post :
Warning: There seem to be problems with ABC/HR when JRE 1.6 is installed. If you are affected by hangs, crashes or any other difficulties, please download and install JRE 1.5 Update 15 from http://java.sun.com/products/archive/j2se/5.0_15/index.html (http://java.sun.com/products/archive/j2se/5.0_15/index.html). You do NOT have to uninstall your existing copy of JRE 1.6 since both versions can be installed at the same time. However, if you have JRE 1.5 and JRE 1.6 installed, please make sure you start ABC/HR with JRE 1.5 manually. Since the working directory has to be the location where abchr.jar is stored, the best thing you can do is to open a command prompt window, navigate to the location where you stored abchr.jar and the rest of the files and then call "C:\Program Files\Java\jre1.5.0_15\bin\java.exe -jar abchr.jar". Of course, this is only an example which you have to adapt according to where java.exe is stored on your PC.
I'll try that (I should have read carefully the first post...)
Group: Members
Posts: 1
Joined: 5-June 05
Member No.: 22536
Wow, you joined almost 3 years ago, it is your first post!
Well, you can learn without posting
This is a great forum, I try to not flood with useless comments
The test will end tomorrow evening (German time), so the ones still testing or who didn't post their results yet, please hurry up!
Hello,
I'm testing what I can. I started yesterday. Since I'm not in toutch with Hydrogenaudio anymore, I just saw that the test had begun.
And you know what ? This stupid ABC/HR, though reinstalled, kept the open/save default directories of the ancient tests. So yesterday, I ABXed samples 5 and 6 from the multiformat test from two years ago !!! (http://3141592.pio2001.online.fr/smileys/mad.gif)
But tonight I got the right 8 and 9 samples.
I'll try to test about two more samples tonight, but I have to prepare a real-life ABX test between amplifiers and interconnects that takes place tomorrow in France, thanks to Grand X, who owns the listening room and gear, and Ohl, who owns a hardware ABX switch...
Expect my submission in several hours.
Results from you would be really awesome Pio!
Still waiting for Alex B's results. Any maybe Guru.
Results from you would be really awesome Pio!
Still waiting for Alex B's results. Any maybe Guru.
You'll have some results tomorrow - probably not the full set. I promise :-)
Still waiting for Alex B's results...
I finally got the two remaining samples tested and sent my results.
Mine were also sent few hours ago.
Can't wait to see the results...
[expectation mode] @guruboolez [/expectation mode]
I am sorry, but the results will be published in about two days. You can still post results until then. It seems that some people had some problems saving and I would like to give them the chance to submit since they tested already but sent me wrong files. Pio also said that he might be able to test a few files more today.
I am not updating the listening test site as this short extension isn't worth mentioning.
Edit: First "correction" just arrived. Somebody sent the ECF files by mistake instead of the ERF ones.
Can we expect an "extra" listening test with the same samples involving Nero/AAC, aoTuV 5.5 and MPC/SV8, or is this kind of thing a little bit out of place these days?
Well, this was definitely the last test at 128 kbps, that is for sure.
The next test will focus on AAC encoders at 80 kbps and the winner of that test will be featured in a multiformat listening test at the same bitrate where Vorbis will participate as well. If Musepack developers think that their encoder is competitive at 80 kbps, I can think about including it, but last time I checked, MPC was good only for > 128 kbps.
Correct me if I'm wrong, but I thought there was some degree of agreement to testing the 96k-ish range next. It's been stated numerous times before that 128k has long bumped into the boundaries of the practically, statistically viably and publicly testable (and I expect this test to largely confirm that), and that the real competition now lies in the near-transparent bitrates, which have hardly ever been thoroughly publicly put to the test. Fwiw, I'm more than willing to host the test samples again.
On second thought, maybe we should be giving Sebastian a break, and let him properly finish this test's results, which I'm sure we're all looking forward to.
isn't it "technically" wrong to call "128 kbps test" when the encoders are basically using VBR? Because for sure there will be higher bitrates than 128.
isn't it "technically" wrong to call "128 kbps test" when the encoders are basically using VBR? Because for sure there will be higher bitrates than 128.
In a very theoretical sense the answer may be 'yes' and in fact VBR brings a slight problem for chosing encoder parameters in order to give a fair comparison to the contenders.
In a practical sense these problems are solved by considering average VBR bitrate for a large variety of tracks (not those investigated within the test).
isn't it "technically" wrong to call "128 kbps test" when the encoders are basically using VBR? Because for sure there will be higher bitrates than 128.
... and lower. The overall average bitrate of the now tested encoders & settings was 127.8 kbps in my bitrate tests.
Did you read the pretest thread at all? I posted the test results several times, for instance:
http://www.hydrogenaudio.org/forums/index....st&p=593735 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=57322&view=findpost&p=593735)
ok so let's get to the RESULTS!
OK, the test is over. Please do NOT submit any more results as they won't be processed. I am starting to sort files and should have the overall graph ready in a few hours. The results page with the results per sample should be ready in the next days.
Topic title edited
OK, first step done. I managed to save all attachments from your mails, decrypted them and sorted them accordingly. I am very happy to tell you that all samples have more than 20 results. Actually, we have an average of over 27 results per sample with sample 1 being the most tested one with a total of 39 results!
Therefore, I would really like to thank everyone a lot for participating and spreading the news!
Now getting to the analytical part.
I am very happy to tell you that all samples have more than 20 results. Actually, we have an average of over 27 results per sample with sample 1 being the most tested one with a total of 39 results!
Very good news indeed!
Now getting to the analytical part.
Good luck with that
Partial results are here: http://www.hydrogenaudio.org/forums/index....showtopic=67529 (http://www.hydrogenaudio.org/forums/index.php?showtopic=67529) and are waiting for approval. This thread can be closed now. Please continue discussion about results in the dedicated thread.