HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: rjamorim on 2003-09-10 23:40:38

Title: 64kbps listening test
Post by: rjamorim on 2003-09-10 23:40:38
Hello.

I'd like to invite all HydrogenAudio members to participate in my 64kbps public listening test, that starts now.

Please head to my listening tests page, instructions are there.
http://audio.ciara.us/test/ (http://audio.ciara.us/test/)

Test ends on September 21st.

Thanks for your attention.

Best regards;

Roberto.

Update: A Java version of ABC-HR has been released. Now, users of MacOS, Linux, Solaris, etc. can also participate.

Further details are available at the presentation page.

Regards;

Roberto.
Title: 64kbps listening test
Post by: Audible! on 2003-09-10 23:46:05
"Redirection limit for this URL exceeded...."  are there obligatory cookies involved?
Title: 64kbps listening test
Post by: mdmuir on 2003-09-10 23:50:38
Quote
"Redirection limit for this URL exceeded...."  are there obligatory cookies involved?

I had the same error as well-in Mozilla. IE just wouldn't load the page-no error given.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-10 23:51:06
Quote
"Redirection limit for this URL exceeded...."  are there obligatory cookies involved?

That's WEIRD!

Can you try now, starting at the test home page?

@verloren: Are you around? :B
Title: 64kbps listening test
Post by: mdmuir on 2003-09-10 23:53:54
Goin to the test home page, and then to the 64kbps presentation worked!
Title: 64kbps listening test
Post by: Digga on 2003-09-10 23:54:57
Quote
IE just wouldn't load the page-no error given.

had the same problem, works now, just try again

*edit* all right, I was a little too late  *edit*
Title: 64kbps listening test
Post by: rjamorim on 2003-09-10 23:55:19
Quote
Goin to the test home page, and then to the 64kbps presentation worked!

Ah, much better. Thanks for checking it out

Still, something very weird is going on. Probably the server is misconfigured.
Title: 64kbps listening test
Post by: mdmuir on 2003-09-10 23:56:41
No problem Roberto,

Anything to help a dedicated hard working man such as yourself!
Title: 64kbps listening test
Post by: Audible! on 2003-09-11 00:06:51
Working now....that's some quick work! 
Title: 64kbps listening test
Post by: lexor on 2003-09-11 02:30:56
just wondering, are you using that new compile of oggenc john33 posted recently? not that I really worried, just wondering
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-11 02:38:47
Just finished downloading the first sample set.... 12MB.... big on a 56k 

One question, though: do we post test-results here?

Thanks. Looking foward to seeing what 64kbps format I should use for a little flash player if and when they support it.

EDIT: Disregard that first question for anyone who may have noticed it.....
Title: 64kbps listening test
Post by: rjamorim on 2003-09-11 03:12:27
Quote
just wondering, are you using that new compile of oggenc john33 posted recently? not that I really worried, just wondering

Yes, it is.

Unfortunately, I couldn't wait anymore for Vorbis 1.0.1. The Xiph schedules are not reliable enough, I was fearing I would have to conduce this test in december if I waited for it (I already wasted one week waiting). :B

Quote
One question, though: do we post test-results here?


No! Please! Don't even comment about your results here.

The instructions are in the readme. Send them to my e-mail.
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-11 03:36:34
Quote
Quote
One question, though: do we post test-results here?


No! Please! Don't even comment about your results here.

The instructions are in the readme. Send them to my e-mail.

Heehee.... missed that part 
Title: 64kbps listening test
Post by: joey_m on 2003-09-11 15:57:37
D'oh!

I was  trying to find the real audio codec, until I realized the sample files were already encoded for the most part... <sigh>

I'll do my best to help out, downloading the samples now.


Cheers, Joey.
Title: 64kbps listening test
Post by: tigre on 2003-09-11 16:16:35
Great work, Roberto! It's really interesting to *hear* the differences between codecs I would have never bothered to test otherwise. I hope many people will participate.
Title: 64kbps listening test
Post by: bond on 2003-09-11 16:24:40
rjamorim,
thanks for all the work from my side too!

it's really interesting how absolutely different the codecs can sound...
Title: 64kbps listening test
Post by: ViPER1313 on 2003-09-11 20:51:52
OOPS!!!!
Title: 64kbps listening test
Post by: bond on 2003-09-11 20:55:31
 @ ViPER1313

better not post your results here
Title: 64kbps listening test
Post by: askoff on 2003-09-11 20:56:39
EDIT: So once again i'm too late... But i gues that's not so important afterall.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-11 21:11:41
Hehe

Well, sorry about that "rule", but that's needed to avoid people "comparing results".

Something like "oops, Guruboolez rated sample 4 2.2, I rated it 4.5. He has much better hearing, so I must be doing something wrong. Let me edit this result file..."

Also, by comparing results people might find out which codec is which. That is no good :B

Thanks for your comprehension
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-11 21:29:29
Quote
Also, by comparing results people might find out which codec is which. That is no good :B

Hmmm.... hate to tell you this, but knowing the codecs involved, I can pick at least two of them out 

Of course I'm completely objective
Title: 64kbps listening test
Post by: rjamorim on 2003-09-11 21:35:10
Quote
Hmmm.... hate to tell you this, but knowing the codecs involved, I can pick at least two of them out  

Of course I'm completely objective

Ah, of course, some of them are impossible to hide.

Quote
[00:37:49] <ff123> oh damn!  played the very first file for 1 second.  Found blade


But at least, People aren't announcing out loud "1 is XXX, 2 is YYY..."
Title: 64kbps listening test
Post by: BadHorsie on 2003-09-12 01:05:14
ok guys ... i help you to abx 64kbs files if you help me abx eac vs. cdparanoia rips 

BadHorsie
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-12 01:08:17
Quote
ok guys ... i help you to abx 64kbs files if you help me abx eac vs. cdparanoia rips 

Uhhhh....... to...... WAV files (as PCM blah blah)??? I think that's a losing battle.
Title: 64kbps listening test
Post by: BadHorsie on 2003-09-12 01:13:40
would you please do a quick abx test?

BadHorsie
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-12 01:22:57
Quote
would you please do a quick abx test?

BadHorsie

..... to wav files, right? I'm all for helping you out but 1.) Wavs, even losslessly compressed, are HUGE, and 2.) I don't think I'd be able to tell the difference. They are.... um.... the same, and all (unless you have a bad CD drive, etc.)

Unless of course you're talking about a compressed format.
Title: 64kbps listening test
Post by: BadHorsie on 2003-09-12 02:17:47
try to rip a song a few times and do a md5sum. you will be suprised.
try to rip a cd which are pre-emphased with eac and cdparanoia. you will be suprised.

as far as i know had no one ever abx'ed eac vs. cdparanoia rips. i think it could not be bad to do it once.

BadHorsie
Title: 64kbps listening test
Post by: rjamorim on 2003-09-12 02:34:48
I don't see much point in ABXing EAC vs. CDex. If a CD is clean they will sound identical. If it's scratched or dirty, EAC will fail on some parts, CDex will fail on others. It's unlikely that only one of them will fail, unless it's a slight error to which one of them is prepared and the other isn't.

MD5sums and binary comparations also don't make sense at all. All you need is an offset difference of one bit for the MD5s to be completely different.

Heck, you can even get different sums if you rip the same track twice with any of these rippers. Even if the CD is perfectly clean.

Pre-emphasis means nothing either. Just because a ripper does de-emphasis while ripping, doesn't mean you can't run a de-emphasis filter on the ripped wave file after ripping. (You can use Sox (http://sox.sourceforge.net/) for that)

Besides, since you migrated completely to Linux, there's not much of a choice. Either be happy with cdparanoia or go back to Windows.

Regards;

Me.
Title: 64kbps listening test
Post by: phong on 2003-09-12 04:51:03
Shockingly enough, apparently EAC can be made to work under Wine, though I haven't had any success with it.  Anyhow, I've got a script that rips CDs twice with cdparanoia to be secure that it does it right.  So far, with the exception of some significantly damaged CDs the MD5 sum comes out the same ever time.  Some drives might be less reliable (I've only tried the three I've got in my computer - an ultraplex, a plexwriter and a Nakamichi 5 disc changer).
Title: 64kbps listening test
Post by: 2Bdecided on 2003-09-12 12:38:07
You don't ABX lossless audio - you compare bits, and that's that.

Time align, inverse mix paste in CEP is fine, though there are other methods.

If the two are not identical, then one is at fault. What it sounds like does not matter - lossless (e.g. ripping CD to .wav) should be a perfect copy of the original. If it's not, there's a fault.


Back to the original topic: How many samples do you want us to try rjamorim?

Everyone: try this test, it's quite good fun.

Cheers,
David.
Title: 64kbps listening test
Post by: Mac on 2003-09-12 13:51:20
I was going to say, please keep this thread about the 64k test..  this is something actually important


To Roberto, could you please plan a 32kb test next, at 64kb I'm finding that I can spot about half of the codecs..

As 2b sais, this test is a lot more enjoyable, it's a bit less demoralising now I can pick out at least 2 codecs every time
Title: 64kbps listening test
Post by: bond on 2003-09-12 16:30:26
anyone here to announce the test at slashdot?
Title: 64kbps listening test
Post by: rjamorim on 2003-09-12 17:49:39
Quote
Back to the original topic: How many samples do you want us to try rjamorim?

As many as you have time, patience and bandwidth to try.

If you can test all of them, that's perfect. But if you can only test one, that's already of much help to me.

BTW, talking about listening tests, would you be interested in my help for that MP2 listening test you were planning? If you want, we can discuss this over e-mail (http://pessoal.onda.com.br/rjamorim/mail.gif).

Quote
To Roberto, could you please plan a 32kb test next, at 64kb I'm finding that I can spot about half of the codecs..


Well, that's a possibility, of course. But I have other priorities first. Maybe I can conduce something like that early 2004

Quote
anyone here to announce the test at slashdot?


Thanks for reminding, that would indeed be welcome. Also, announcing at Kuro5hin and <shivers> r.a.o would be interesting.

Regards;

Roberto.
Title: 64kbps listening test
Post by: lexor on 2003-09-13 15:18:32
do you want us to submit the news to comunity sites we are registered with? like DeskMod? guys over there are getting pretty bored  (now that they can't submit their app skins/visualizations/wallpapers etc...)
Title: 64kbps listening test
Post by: bond on 2003-09-13 15:44:14
why not?

the more people who join the test, the more work for rjamorim 
Title: 64kbps listening test
Post by: rjamorim on 2003-09-13 15:54:46
Yes, the more it is announced, the better for all of us

Quote
the more people who join the test, the more work for rjamorim


Evil yu0 :B
Title: 64kbps listening test
Post by: [proxima] on 2003-09-13 16:40:38
I'm listening a sample and i occured with a possible bug in ABC-HR (this is not the 1st time). The samples are not played anymore, only the first part.
I've already done the comments for the samples and i don't want to restart the ABC-HR session again... any chance to recover the situation ? I'm afraid i should restart abc-hr 
Title: 64kbps listening test
Post by: ancl on 2003-09-13 16:55:53
Quote
,Sep 13 2003, 05:40 PM] I'm listening a sample and i occured with a possible bug in ABC-HR (this is not the 1st time). The samples are not played anymore, only the first part.

I've never encounted that error. Are you sure that you have not changed which part of the track that should be listened at? (The bar at the bottom of the screen...)

(Edit:Spelling)
Title: 64kbps listening test
Post by: [proxima] on 2003-09-13 17:04:46
Quote
I've never encounted that error. Are you sure that you have not changed which part of the track that should be listened at? (The bar at the bottom of the screen...)

Thanks. You're right, i've played a little with the bar and the problem is now resolved. I don't know if there was a bug, maybe i'm just too tired... for today i think i'll stop doing listening test 
Title: 64kbps listening test
Post by: nwn on 2003-09-13 22:29:47
Guru posted on other forum and I decide to come on the evaluation process

but .. as i try many investigations ..the fact is that i don't hear any sound from ABC ..

so let me explain config ..
what is the package i choose same issue so fix on the Sample01.zip

i decompress the file and all files are in the subdir \Sample01\
sure i've run the Sample01.bat (and not exe) .. and Sample01 files are converting in wav extension

all these wave are correct cause i can hear them in foobar.. good player isn'it ..

i launch abx and load config01 and see infos and buttons active (blue etc ..)

but which button and package (i try 5) no sound at all ..

observ .. portion selction bar is active but 0 / 0 0 sec displayed ...

here is the config01.txt
TestName = BigYellow Listening Test

Original = .\Sample01\BigYellow.wav
Sample1  = .\Sample01\BigYellow_1.wav
Sample2  = .\Sample01\BigYellow_2.wav
Sample3  = .\Sample01\BigYellow_3.wav
Sample4  = .\Sample01\BigYellow_4.wav
Sample5  = .\Sample01\BigYellow_5.wav
Sample6  = .\Sample01\BigYellow_6.wav
Sample7  = .\Sample01\BigYellow_7.wav
Sample8  = .\Sample01\BigYellow_8.wav

Offset_Orig =  14
Offset1    =  0
Offset2    =  14
Offset3    =  14
Offset4    =  18
Offset5    =  14
Offset6    =  48
Offset7    =  48
Offset8    =  14

___________________________________________________
subfolder

Directory of C:\Downloaded\Test_64_abc-hr\Sample01
bigyellow.wav                                                                                                  bigyellow_1.wav                                                                                                bigyellow_2.wav                                                                                                bigyellow_3.wav                                                                                                bigyellow_4.wav                                                                                                bigyellow_5.wav                                                                                                bigyellow_6.wav                                                                                                bigyellow_7.wav                                                                                                bigyellow_8.wav                                                                                               
9 file(s) found
Total file size 35 815 584 bytes
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-13 22:36:46
Quote
observ .. portion selction bar is active but 0 / 0 0 sec displayed ...

lol  I think you've deselected what ABC-HR sould play 

Just select the entire sample (highlight the entire bar so that it's shaded), and then hit a play button. The bar can be shortened(/deselected) so that only particular parts of the file are played.
Title: 64kbps listening test
Post by: nwn on 2003-09-13 22:40:05
i'm newbie for ever but not stupide .. no matter
it's a joke ..

the all bar is grayed

i read the posts....

yes i understand your analysis
thanks to help


Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-13 22:42:48
Quote
i'm newbie for ever but not stupide .. no matter

Never said you were. But you did say 0/0 seconds, and that means nothing's selected, at least on my copy...

EDIT: Do you have a screenshot of abc-hr you can post?
Title: 64kbps listening test
Post by: ff123 on 2003-09-13 23:49:24
From the screenshot it looks like everything's been loaded up properly, except that the portion select bar says 0.0 on both ends (it should say 30.0 sec at the right end for the "gone" sample).

I'm not sure what went wrong, but you could try setting it up manually by going to the File -> Setup Test menu and entering the files and the offsets by hand (if you loaded the config file properly, they should all be there already, though).

ff123
Title: 64kbps listening test
Post by: nwn on 2003-09-14 00:50:34
hi fff123 happy for your post ....

yes, i was surprise for "0" value too ...  and conclude this is the problem ..

i've already look for ...  and change manually path .. cause i thought taht ".\" was maybe not good .. ("..\" was maybe better)

it's what for i published the config01.txt .. cause in fact .. I dont't know if offset values are the rights one ....

but as i change manually the setup no more success ...

the config01.txt is not changed by manual setting?
as i  press the Ok buttun soft ask for saving result test txt ....
but the config01.txt is not changed

version  0.9b 30/8/02

i looked for your site before .. see the explanation about error ...
"AAC decoder, FAAD (dated before about the early part of 2003) had a bug"
and didn't find any new version

okay .. that's all ..

no ...
all wave seems to be Pcm but lenght time are 24 sec except one is 25 sec ...
and curious suscpect ....  some files haven't the same bitrate and bits

Sample01\BigYellow_4.wav
----------
540911 samples @ 22050Hz
bitrate = 705
samplerate = 22050
channels = 2
codec = PCM
bitspersample = 16
----------
540911 samples @ 22050Hz

Sample01\BigYellow_7.wav
bitrate = 1024
samplerate = 32000
channels = 2
codec = PCM
bitspersample = 16
----------
783360 samples @ 32000Hz

all others are

bitrate = 1411
samplerate = 44100
channels = 2
codec = PCM
bitspersample = 16
----------
1143808 samples @ 44100Hz

is there a problem with codec ... or calculate process ... i saw wich codec are generating this files ...

that's all for now ...
what can i do ..
Title: 64kbps listening test
Post by: ff123 on 2003-09-14 01:01:08
Quote
i've already look for ...  and change manually path .. cause i thought taht ".\" was maybe not good .. ("..\" was maybe better)

Try using the browse button to select the file.

Quote
it's what for i published the config01.txt .. cause in fact .. I dont't know if offset values are the rights one ....


Try setting up the test manually one sample at a time to see where the error occurs.  I would use an offset of 0 for the samples until you get all 8 files loaded, then start changing the offsets one by one.

Quote
but as i change manually the setup no more success ...

the config01.txt is not changed by manual setting?
as i  press the Ok buttun soft ask for saving result test txt ....
but the config01.txt is not changed


Setting up the test manually does not change the config file.  The results file is completely separate from the config file.  The results file stores your comments and your ratings.

Quote
version  0.9b 30/8/02

i looked for your site before .. see the explanation about error ...
"AAC decoder, FAAD (dated before about the early part of 2003) had a bug"
and didn't find any new version


No new version was needed to fix this error -- FAAD was not reporting the WAV properties correctly, and the error was fixed with a recent version of FAAD2.

ff123
Title: 64kbps listening test
Post by: nwn on 2003-09-14 01:16:35
okay i did exactly what you said ...

i already use the browse button and lear insert value ...

but i just restart the process ...
neither the right time value changed ....
always "0" ....

ok ..

i must to try with another pack from another test ...
where can i found a pack which you are certain works ...



no move any file .... respect original tree and file position
hi 2bdecided
Title: 64kbps listening test
Post by: 2Bdecided on 2003-09-14 01:17:59
This is probably a stupid question, but you haven't moved abchr.exe have you? e.g. onto the desktop etc?

Cheers,
David.
Title: 64kbps listening test
Post by: ff123 on 2003-09-14 01:27:15
Quote
okay i did exactly what you said ...

i already use the browse button and lear insert value ...

but i just restart the process ...
neither the right time value changed ....
always "0" ....

ok ..

i must to try with another pack from another test ...
where can i found a pack which you are certain works ...

Just to be clear:

You tried just the reference and one test file to start with?  You can make the test file also point to the reference file (just in case one of the decoded test files are messed up, for some reason or another).  Leave the offset at 0.

After you hit ok, the first group out of the eight will become active.  The portion select bar should also have 30.0 sec on the right.

I'm unable to replicate this problem with the "gone" package, by the way.

ff123
Title: 64kbps listening test
Post by: ff123 on 2003-09-14 01:29:50
I wonder if there's some problem with the different sample rates?

That's another reason to point to just the reference file, which is sampled at 44.1 kHz.

ff123
Title: 64kbps listening test
Post by: nwn on 2003-09-14 01:40:00
null
Title: 64kbps listening test
Post by: ff123 on 2003-09-14 03:27:46
Quote
YES Exactly ...
i also try to launch soft and whithout load config ... select direct file/setup test
browse for the orignal file and the first wave without changing offset

exact blue icon activate ...

no change on time ...

adjust offset :
Offset_Orig =  14
Offset1     =  0

and no more change ...

I test from scratch with the same file for orig and first ....
offset 0 .. then 14
no more result....

ok on the batch process ...
wav04 is Lame
wav07 is faad

for Lame i've 8 lame.Exe and 5 different versions of lame on some different dir .... and the binone in more .. with a prefetch lame ... must fix that ...
the lame one in bin is v3.903mmx
is there a path problem ..

Faad two file same size 13K don't know release no seen under dos ..

nwn,

I'm having trouble understanding exactly what is and isn't working in your last message.  Please clarify.  You may have to supply pictures of either the setup window or a copy of whatever config file it is you're using.

ff123
Title: 64kbps listening test
Post by: rjamorim on 2003-09-14 06:42:58
nwn;

Comme une exception, je crois que tu peux écrire en français et ignorer, seulement cette fois, la règle de "seulement anglais".

Après, moi ou Pio2001 peut traduire ton post par l'anglais.

---------
For those that want to understand WTF I wrote:
--
As an exception, I think you can write in French and ignore, only this time, the rule of "only english".

After, me or Pio2001 can translate your post to English.
---------

Roberto.


OBS: Please noone come whining rubbing rule 10 on my nose. This is a special case. I hope you understand.
Title: 64kbps listening test
Post by: wkwai on 2003-09-14 11:05:13
I noticed that people posted audio clips which are too loud.. You should not use loud clips for subjective listening tests. 
This has something to do with the masking effects. For loud tones, the spreading functions for the masker is not as steep as those of quieter tones.
You would probably noticed that you can hear more "hidden tones" when the volume of your player is lower than when it is higher.
This effect is modelled in the ISO Psychoacoustic Model I.
Title: 64kbps listening test
Post by: nwn on 2003-09-14 11:35:05
okay ...

I did exactly what fff123 said ..

first i decide to start from scratch again ..
re-download from http://audio.samharris.us/test/abc-hr_bin.zip (http://audio.samharris.us/test/abc-hr_bin.zip)
- decompress in C:\Downloaded\abc-hr_bin
- run abc.Exe
- i didn' load any config
- go to "file/setup test"
- browse "orig wave" and first "wave" to the same file

"BigYellow.wav"
bitrate = 1411
samplerate = 44100
channels = 2
codec = PCM
bitspersample = 16
----------
1080185 samples @ 44100Hz

- no change offset
- click ok ...
- "blue" arrow appears .. ok
- bar displayed in gray .. but OO (start)  ---- OO (stop)sec
SO what button I click ...NO SOUND ....

- go to setup test
- change offset to "Offset_Orig =  14"
- press Ok
- same result ........

I decided to test with another pack ..
downloaded "64kbsGroupTest" with NewKid
Same effects ...

same display than i post


-----------------------------------

when i looked on the file infos ...
i saw than bitrate and bitspersample different for
wav04 is Lame encoded
wav07 is faad encoded

so .. i looked for lame.exe and faad.Exe
- lame.exe in \bin dir is v3.903mmx (but i have some others lame.exe in other dir)
- faad.Exe in \bin dir is 134k size

________________________________________________

okay ..
- i didn't move any file or dir ...
- i loaded all the packs and run the batch for each
- same results for each pack
- no other error message ...

is this post more explicit and in a better english?

what do you want
- i do
- screen capture
- i verify
Title: 64kbps listening test
Post by: tigre on 2003-09-14 12:40:17
Quote
I noticed that people posted audio clips which are too loud.. You should not use loud clips for subjective listening tests.

There are only loud clips in > 90% of mainstream music of the last 10 years (mastered at high volume). A test that uses clips that have nothing to do with real life wouldn't be worth much, would it?

Quote
This has something to do with the masking effects. For loud tones, the spreading functions for the masker is not as steep as those of quieter tones.
You would probably noticed that you can hear more "hidden tones" when the volume of your player is lower than when it is higher.

I don't understand where's the point. Audio players that deserve this name have features to change volume (= e.g. volume slider). Noone forces you to do the test at a certain volume.

As you probably have noticed, the test has already been started an there was a pre-test thread open for discussion for some weeks (including the clips). It might be a good idea to mention your concerns earlier next time.
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-14 14:35:48
Quote
I'm unable to replicate this problem with the "gone" package, by the way.

ff123

Me either. They've all worked so far.
Title: 64kbps listening test
Post by: nwn on 2003-09-14 15:41:28
  me too unable to reproduce on others computers ....

in fact my reputation must be saved ... 

i decide a great .. decision .. to test on one of my five other computers in home ..

on two others PC  .. after install ... ALL is working fine  ....

so this is my stupid pc station that have some issue ..
multi-layered install/desinstall for more than two years ... on xp ... standard issue ..

in fact no icon sound wants to be displayed on taskbar and my volume key on Ms MM Keyboard are producing noting ..

it's some sample of my dirt configuration ..

no problem . as my hard disk is going to have TEC issue by november .. (smart monitoring) i will reinstall a new one  .. but no ibm 60/75 series of course ..

sorry ... stop search to resolve this issue
this is probably non chance
Title: 64kbps listening test
Post by: 2Bdecided on 2003-09-15 09:38:22
Glad you found the problem.


Everyone: go on, try this test - it's fun!

Cheers,
David.
Title: 64kbps listening test
Post by: Atlantis on 2003-09-15 11:33:30
Quote
Everyone: go on, try this test - it's fun!

I completely agree.
That was my first listening test at this bitrate, and while some samples are extremely easy to abx / test, a pair of them are really tricky (what a headache!  )

I can't wait for the final results (and if they are like mine... big surprises  )!
Title: 64kbps listening test
Post by: joey_m on 2003-09-15 19:06:27
Quote
Everyone: go on, try this test - it's fun!

... or frustrating!    Maybe because of my sub-par gear or ears (probably both!), but some samples are giving me quite a hard time!


Cheers, Joey.
Title: 64kbps listening test
Post by: Atlantis on 2003-09-15 20:50:21
Quote
Quote
Everyone: go on, try this test - it's fun!

... or frustrating!    Maybe because of my sub-par gear or ears (probably both!), but some samples are giving me quite a hard time!


Cheers, Joey.

Nah 

That's the fun!
Title: 64kbps listening test
Post by: Gambit on 2003-09-16 15:55:23
Roberto: While doing the test I've noticed that the files have different sample rates.
Of course, that's because each encoder chooses the appropriate sample rate at this bitrate.
But wouldn't it be better if all the samples would be resampled to 48kHz after decoding? To avoid resampling issues etc...
Title: 64kbps listening test
Post by: AstralStorm on 2003-09-16 16:39:37
I think we're testing real-life situations, SB Live/AC97 playback too.

/EDIT\
Yeah, I had some hard samples too, mainly because these are no problem samples,
LAME AP128 (I think these were compressed with it) doesn't damage them too badly
and I did the test at 0300 UTC. (it's night here then)

Now ABXed all my 5.0 samples and it wasn't very hard.
\EDIT/
Title: 64kbps listening test
Post by: 2Bdecided on 2003-09-16 16:50:23
Quote
I think we're testing real-life situations, SB Live/AC97 playback too.

Yes, but what about cards which sound very different at different sample rates, or glitch when changing sample rates? It makes ABX a bit pointless! And could make the test quite unfair.


Luckily the Audiophile 2496 changes between 22.05, 32 and 44.1 seamlessly, whereas it mutes and fades in when switching to 96k (and back).

Cheers,
David.
Title: 64kbps listening test
Post by: AstralStorm on 2003-09-16 17:06:44
Quote
Yes, but what about cards which sound very different at different sample rates, or glitch when changing sample rates? It makes ABX a bit pointless! And could make the test quite unfair.

Only issue is with 32kHz, as some people might not detect lowpass. (16kHz)

But what about cards supporting only 44,1kHz, like some AC97's?

Anyway, you shouldn't need to ABX these samples anyway - they're damn easy (maybe except few samples nr 5 and these are 44,1kHz anyway)
Title: 64kbps listening test
Post by: rjamorim on 2003-09-16 17:13:46
Quote
Roberto: While doing the test I've noticed that the files have different sample rates.
Of course, that's because each encoder chooses the appropriate sample rate at this bitrate.
But wouldn't it be better if all the samples would be resampled to 48kHz after decoding? To avoid resampling issues etc...

Well, I already discussed that with Darryl, because tigre also brought this issue some days ago. There are some points we came up with:

- In a sound card that resamples everything to 48, all samples will be treated equal - I.E, all will be upsampled since there are no 48kHz samples

- Test would need to be restarted! I can't mix together the results of a non-resampled test with the results of a resampled one. People would hate me for that.

- Someone might complain that "no, my favorite codec WAS performing well, but then that stupid Roberto added resampling, and everybody knows resampling reduces the quality". Specially if this test is announced at /. :B

- On some very old cards (like the SB16 I owned for years and now is owned by my mother), if you feed them 48kHz signal the quality gets terrible unless you change the resampling quality setting at the control panel (the default is lowest quality). I remember all the hissing when I tried playing my first DVD here. >_<
So, it's also hard to decide upon a standard sampling rate for all samples.

So, if people think this is a major issue, that's OK. I'll restart the test. But there's no way I can change the settings now, halfway through it.

Thanks for bringing this issue.

Regards;

Roberto.
Title: 64kbps listening test
Post by: 2Bdecided on 2003-09-16 17:18:01
We don't discuss whether they're easy or not until the test is over.

But look further up this thread at some of the comments...


I had a sound card that _didn't_ resample - it just inserted duplicate or zero samples. It sounded horrible. Should I have graded 22kHz sampled material using this card? Actually, you should have heard 11kHz sampled material!

But that was a pro, the DAL CardD - one of the best at the time.

Cheers,
David.
Title: 64kbps listening test
Post by: 2Bdecided on 2003-09-16 17:21:00
No no no - don't re-start the test!

Just remember it for next time (  ), and mention it in the results. It probably doesn't matter that much, if at all - just being rigorous.


btw, it would be natural to resample everything back to the original sample rate, e.g. 44.1 in this case. Otherwise you're comparing sound cards and windows resampling and codecs, rather than just codecs!

Cheers,
David.
Title: 64kbps listening test
Post by: Gambit on 2003-09-17 09:16:04
I also don't think that there is a need to re-start the test. I just brought it up as an idea to be considered for future tests.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-17 16:50:44
Quote
I also don't think that there is a need to re-start the test. I just brought it up as an idea to be considered for future tests.

OK, thanks.

I'll make sure I'll mention it at the results page as well.
Title: 64kbps listening test
Post by: schnofler on 2003-09-17 18:36:35
Hi,
this is my first post to this forum, and I don't know whether that is the right topic to post this to, but I guess you might be interested in this so I just post it here.

So, let's get to the point. I programmed a Java version of ff123's ABC/HR application so all those Linux users out there can participate in listening tests in the future (well, actually I just had nothing better to do, but that doesn't sound so philanthropic  )
It's still a very early alpha (I started development five days ago), but it's already pretty functional. I designed it to be as compatible as possible to ff123's Windows application. It reads the same config-files and writes the same result files, and overall my Java version is pretty much a straight rip-off of the original. It also has some additional features like optional disabling of the fast-switching feature, looping and saving sessions in progress.

Now two issues arise: First, I'd like to ask ff123, whether you mind me publishing such a blatant copy of your work, and, if you don't object, what exactly I'd have to do so I don't violate this Lesser GPL under which you distribute your application (I don't have a lot of experience on these things, and I didn't feel like working through those ten pages of legal mumbo-jumbo in the license.txt).
And secondly, I don't have a website or webspace myself, so if some of you are interested in my version, it would be nice if one of you could make it available to the public (the program is only about 100KB in size).

Thanks in advance, and also many thanks to rjamorim for organizing this great listening test.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-17 19:10:34
Quote
Hi,
this is my first post to this forum, and I don't know whether that is the right topic to post this to, but I guess you might be interested in this so I just post it here.

So, let's get to the point. I programmed a Java version of ff123's ABC/HR application so all those Linux users out there can participate in listening tests in the future (well, actually I just had nothing better to do, but that doesn't sound so philanthropic   )
It's still a very early alpha (I started development five days ago), but it's already pretty functional. I designed it to be as compatible as possible to ff123's Windows application. It reads the same config-files and writes the same result files, and overall my Java version is pretty much a straight rip-off of the original. It also has some additional features like optional disabling of the fast-switching feature, looping and saving sessions in progress.

Now two issues arise: First, I'd like to ask ff123, whether you mind me publishing such a blatant copy of your work, and, if you don't object, what exactly I'd have to do so I don't violate this Lesser GPL under which you distribute your application (I don't have a lot of experience on these things, and I didn't feel like working through those ten pages of legal mumbo-jumbo in the license.txt).
And secondly, I don't have a website or webspace myself, so if some of you are interested in my version, it would be nice if one of you could make it available to the public (the program is only about 100KB in size).

Thanks in advance, and also many thanks to rjamorim for organizing this great listening test.

Woot!

Your tool is a Godsend

About licensing: trying to make a loong license short, the LGPL states that if you change LGPLd code, you must make the changes available somehow. That's all.

It doesn't force you to open the sources of all the work related to it, like the GPL.

And I would be very glad to host it at RareWares (check the bottom of the post). ff123 probably would also be happy to host in his ABC/HR page.

Can you mail it to me?


I'll test it ASAP and, if there are no issues/bugs, I'll use it as an option for Linux/Mac/whatever users.

Best regards;

Roberto.

One question: Out of curiosity, what runtime engine version is needed to run it?
Title: 64kbps listening test
Post by: schnofler on 2003-09-17 20:20:51
OK, I mailed the program to you. I doubt there won't be any bugs, though, so if you find some, I'll be glad to eliminate them 
The JDK I used is 1.4.2, but generally it should run on 1.4, maybe even 1.3 (Sun did quite a number of bugfixes to the Java Sound API since then, however).
Title: 64kbps listening test
Post by: ff123 on 2003-09-18 00:12:13
Quote
So, let's get to the point. I programmed a Java version of ff123's ABC/HR application so all those Linux users out there can participate in listening tests in the future (well, actually I just had nothing better to do, but that doesn't sound so philanthropic   )
It's still a very early alpha (I started development five days ago), but it's already pretty functional. I designed it to be as compatible as possible to ff123's Windows application. It reads the same config-files and writes the same result files, and overall my Java version is pretty much a straight rip-off of the original. It also has some additional features like optional disabling of the fast-switching feature, looping and saving sessions in progress.

That's great!

If Roberto hosts the program on his site, I'll just create a link on my page to point to it.

I'm in the process of writing code to "obscure" the config and results files, so when I get that worked out, I'll let you know.  But since the code will be freely available, it shouldn't be a problem to duplicate.

Looks like I'll also have to add in the looping capability and disabling fast switching as well, so that the applications stay as similar as possible.

Could you email me a copy as well?  (my email is in the "about" box of abc/hr)

ff123
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 00:32:32
Quote
Could you email me a copy as well?  (my email is in the "about" box of abc/hr)

I already uploaded it to RareWares:
http://rarewares.hydrogenaudio.org/files/a.../abchr-java.zip (http://rarewares.hydrogenaudio.org/files/abchr-java.zip)
(Didn't announce it yet)

Worked like a charm on my Win2k

I'll test it later on Linux. Then, I will reupload the abc-hr_bin.zip packages with it and new instructions.

Again, thanks a lot, schnofler.

Regards;

Roberto.
Title: 64kbps listening test
Post by: schnofler on 2003-09-18 01:27:17
Quote
I'm in the process of writing code to "obscure" the config and results files, so when I get that worked out, I'll let you know. But since the code will be freely available, it shouldn't be a problem to duplicate.


I thought about that, too. Encryption of the results files would be a nice feature. I don't know a lot about the subject, but using some asymmetric encryption, where the public key is stored in the config file and the application uses this key to encrypt the results should work fine. The test conductor could then use his private key to decrypt the results files. Hmm, I hope the Java SDK contains some easy to use RSA encryption stuff 

Quote
Looks like I'll also have to add in the looping capability and disabling fast switching as well, so that the applications stay as similar as possible.


Heh, sorry, didn't mean to push you 

Thanks for hosting, Roberto. And glad you could get it to work.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 01:34:46
Quote
I thought about that, too. Encryption of the results files would be a nice feature. I don't know a lot about the subject, but using some asymmetric encryption, where the public key is stored in the config file and the application uses this key to encrypt the results should work fine. The test conductor could then use his private key to decrypt the results files. Hmm, I hope the Java SDK contains some easy to use RSA encryption stuff

You can have an idea of what have been/is being discussed about encryption usage in ABC/HR here:
http://www.hydrogenaudio.org/forums/index....howtopic=11691& (http://www.hydrogenaudio.org/forums/index.php?showtopic=11691&)
Title: 64kbps listening test
Post by: phong on 2003-09-18 01:41:00
Blah, so much for my Python one.  :-)  Haven't had time to work on it anyway I suppose.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 02:28:07
Damn. Something didn't work when I tried it on Linux.

But I blame it on my Linux, that is a mess (abusage of apt-get, it seems), and my lack of decent knowledge on this OS.

If anyone is interested, this is the error I got when I ran java -jar abchr.jar

Code: [Select]
Exception in thread "main" java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:112)
    at java.util.jar.JarFile.<init>(JarFile.java:117)
    at java.util.jar.JarFile.<init>(JarFile.java:55)


I'll upload it with the abchr_bin packages anyway. Hopefully other people out there will be luckier than me.

@danchr: I demand that you try it on MacOS X. Now you have no more excuses to avoid participating.
Title: 64kbps listening test
Post by: ff123 on 2003-09-18 03:19:03
Downloaded the app, and it looks really nice.  That was really quick coding!

A few things I would like to test, or at least know if they have been tested:

1.  Randomization of the samples, both upon loading the files for the first time and when re-starting the test.

2. ABX randomization

3. Accuracy of the results file

4. Do the offsets work properly?

ff123
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 03:30:46
Uploaded his latest version with better error handling. Address is the same.
Title: 64kbps listening test
Post by: danchr on 2003-09-18 06:10:07
Quote
@danchr: I demand that you try it on MacOS X. Now you have no more excuses to avoid participating.

Do not underestimate the powers of laziness. And studies.

I'll look into it later  (It's 7 AM)
Title: 64kbps listening test
Post by: schnofler on 2003-09-18 10:51:41
Quote
Damn. Something didn't work when I tried it on Linux.

But I blame it on my Linux, that is a mess (abusage of apt-get, it seems), and my lack of decent knowledge on this OS.

If anyone is interested, this is the error I got when I ran java -jar abchr.jar

Code: [Select]
 Exception in thread "main" java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:112)
at java.util.jar.JarFile.<init>(JarFile.java:117)
at java.util.jar.JarFile.<init>(JarFile.java:55)


Well, that looks like Java couldn't even open the jar file. I have no idea what might be causing that, but you could avoid the problem by unzipping the jar file yourself (jar files are just zip files in disguise). If you change to the directory where you unzipped it you should be able to start the program with "java -cp . abchr.Main".

Quote
@danchr: I demand that you try it on MacOS X. Now you have no more excuses to avoid participating.


Actually, I seriously doubt, that it will run on MacOS. That's because of the architecture of the Java Sound API. For one thing, I don't know if there's even an implementation of that API for MacOS (Sun provides Windows and Linux versions only, afaik). And additionally, different implementations can have quite different behaviour. For example, in Sun's reference implementation the only way to output sound is with Sun's software mixer. I can access other mixers (supposedly hardware mixers, since they were named after my sound card), but they seem to be of no use (no output lines can be opened).
In theory, my program does not need Sun's software mixer, but if it doesn't find that mixer, it just grabs the first one available. And if that one is not as generously designed as Sun's mixer (which has almost no constraints on audio formats and buffer size), it is very likely that the program won't be able to play sound.
But nonetheless, if the MacOS implementation of this API is quite similar to Sun's, the program might run without problems, so I really don't want to scare anyone off. Any error (or success) reports would be helpful.

Quote
Downloaded the app, and it looks really nice. That was really quick coding!


Thanks 

Quote
A few things I would like to test, or at least know if they have been tested:

1. Randomization of the samples, both upon loading the files for the first time and when re-starting the test.

2. ABX randomization

3. Accuracy of the results file

4. Do the offsets work properly?


Of course I tested all of this, but simply because of time constraints, I can't be 100% sure that it all works. As for randomization, the samples are randomized when you load a config file, I'm not sure what you mean by "re-starting" (They don't get randomized again when you save a session and load it later, but of course they do get randomized if you load a config file you had loaded before). However, I can remember that there was some debate about the quality of Java's random number generator. I don't know whether that's a thing of the past, so if anyone wants to conduct some randomness-tests, let me know the results 
I'm pretty sure that offsets are working, and I did some serious testing with the results files, but still, any help testing the program is highly appreciated.
Title: 64kbps listening test
Post by: appleguru on 2003-09-18 12:13:50
Ok. i'm running OS X. As of yet, I'm having limited success getting this to work. But that doesn't mean it shouldn't  OK. So, I d/led the 1st sample files (Big yellow) and extracted them as nesscessary. I can't however, open the Sample01.bat file in the ./Bin/ folder. Any tips? What exactly does that file do? Skipping that step, and going into the App, and trying to open the config file, I get a "File.\Sample01\BigYellow.wav could not be found".. So I figure that that is what the .bat files does... Ok... Now...

I open QT, and export a wav file from the QT AAC version of the file, and add that to the directory. The error still occurs. Now I look at it closer.. And I think I see the problem.

Windows uses backslashes (\) for file paths... OS X uses forward slahes  (/)... Soo... Any help?
Title: 64kbps listening test
Post by: appleguru on 2003-09-18 12:29:43
Upon further inspection, the BAT file does exactly that, it converts all the files to WAV. Ok, so this is a DOS Batch file.. so, if the osx binaries were compiled/included for flac, faad, LAME, oggenc (Maybe oggdec?), and the Batch files converted to say shell scripts, then this could still work. Or, if someone had enough Web bandwidth to host all "Uncompressed" files... Just a few thoughts...
Title: 64kbps listening test
Post by: c_haese on 2003-09-18 13:45:49
Quote
Windows uses backslashes (\) for file paths... OS X uses forward slahes  (/)... Soo... Any help?

The same problem occurs on Linux, by the way.

Anyway, to add to the list of working configurations, besides the backslash problem it works well on my setup with RedHat Linux 7.3 and JRE 1.4.1. Kinda makes my own not-yet-released work-in-progress wxWindows/SDL version look like a pile of manure

Cheerio,

Carsten Haese
Ogg Traffic Editor, Xiph.org Foundation
Title: 64kbps listening test
Post by: Continuum on 2003-09-18 18:24:36
Great program! I really like the new keyboard layout in ABX-mode. (w00t)

Some things I noticed:
- You can create comments without changing any rating (not so in ff123's version). They are not saved though, which could lead to some confusion.
- The cursor does not change above the edges of the selected range.
- Default result filename is the name of the config file.

Quote
However, I can remember that there was some debate about the quality of Java's random number generator. I don't know whether that's a thing of the past, so if anyone wants to conduct some randomness-tests, let me know the results

ff123 had a program that could analyze random sequences for their randomness.

Here is a first test I quickly performed:
0 of 1, p = 1.0
0 of 2, p = 1.0
0 of 3, p = 1.0
0 of 4, p = 1.0
0 of 5, p = 1.0
0 of 6, p = 1.0
0 of 7, p = 1.0
0 of 8, p = 1.0
1 of 9, p = 0.998
2 of 10, p = 0.989
3 of 11, p = 0.967
3 of 12, p = 0.98
3 of 13, p = 0.988
4 of 14, p = 0.971
5 of 15, p = 0.94
5 of 16, p = 0.961
5 of 17, p = 0.975
5 of 18, p = 0.984
5 of 19, p = 0.99
5 of 20, p = 0.994

I'm no expert on this, but this doesn't look very random to me .
Title: 64kbps listening test
Post by: caligae on 2003-09-18 20:20:55
Runs fine under FreeBSD-CURRENT using native Java Development Kit 1.4.1. At least it plays sounds and works basically, haven't stress-tested it yet. Just very slow, well it's Java after all...

The '\' vs '/' could be easily resolved using this simple script:
Code: [Select]
#!/bin/sh
for file in config*txt; do sed -i.bak -e 's|\\|/|g' ${file}; done


The encoding script would be a bit harder. Maybe it would be best to use wine so that we can be sure that the output is the same.
Title: 64kbps listening test
Post by: ff123 on 2003-09-18 21:00:35
Quote
ff123 had a program that could analyze random sequences for their randomness.

Here is a first test I quickly performed:
0 of 1, p = 1.0
0 of 2, p = 1.0
0 of 3, p = 1.0
0 of 4, p = 1.0
0 of 5, p = 1.0
0 of 6, p = 1.0
0 of 7, p = 1.0
0 of 8, p = 1.0
1 of 9, p = 0.998
2 of 10, p = 0.989
3 of 11, p = 0.967
3 of 12, p = 0.98
3 of 13, p = 0.988
4 of 14, p = 0.971
5 of 15, p = 0.94
5 of 16, p = 0.961
5 of 17, p = 0.975
5 of 18, p = 0.984
5 of 19, p = 0.99
5 of 20, p = 0.994

I'm no expert on this, but this doesn't look very random to me .

I used a web-based test for runs to test abchr, I think this one:

http://www.ubmail.ubalt.edu/~harsham/Busin.../Randomness.htm (http://www.ubmail.ubalt.edu/~harsham/Business-stat/otherapplets/Randomness.htm)

For Continuum's data, I put a '0' in the web-form if the answer was incorrect, and a '1' if the answer was correct.  I get a p-value of 0.014, which is moderate evidence against randomness, according to that calculator.  Probably more data (how about 80 trials, for example) would give a better picture.

ff123
Title: 64kbps listening test
Post by: Continuum on 2003-09-18 21:34:43
I made two more tests, this time with 80 trials. The first one did look quite random to me, not the other:
7 of 10, p = 0.171
15 of 20, p = 0.02
20 of 30, p = 0.049
25 of 40, p = 0.076
29 of 50, p = 0.161
37 of 60, p = 0.046
41 of 70, p = 0.094
47 of 80, p = 0.072

I entered this in the Web-script. Results:
Number of Runs ( R ): 40
P-value: 0.47916
Conclusion: Little or no real evidences against randomness

I should have stopped at "14 of 18, p = 0.015".
Title: 64kbps listening test
Post by: schnofler on 2003-09-18 23:03:07
First, thanks to everyone for the testing. Nice to know that it runs under Linux, generally.

Quote
Some things I noticed:
- You can create comments without changing any rating (not so in ff123's version). They are not saved though, which could lead to some confusion.
- The cursor does not change above the edges of the selected range.
- Default result filename is the name of the config file.

OK, I resolved all these issues. Any suggestions for a default filename for the results files?

Quote
Runs fine under FreeBSD-CURRENT using native Java Development Kit 1.4.1. At least it plays sounds and works basically, haven't stress-tested it yet. Just very slow, well it's Java after all...

As for the speed, Sun claim they have made performance improvements for Java Sound in 1.4.2. Actually I was rather surprised how well it performed. As you say, it's Java, so of course it eats a lot more memory and is less responsive than ff123's Windows native version. I almost expected fast switching to be totally useless in Java due to latency, but actually the Java Sound API handles this quite well (on my machine, at least).

I also checked on the randomness issues. Actually Sun use a rather common algorithm (linear congruential) for its random number generator. I don't really think they could have messed this up badly, and anyway, I wouldn't know how to improve on the JDK implementation. Suggestions are welcome, of course.
Title: 64kbps listening test
Post by: appleguru on 2003-09-18 23:15:15
I was thinking.. if we could BitTorrent the uncompressed test files (Converted to wav) then we could all use them right? Or do the .bat scripts randomize their order as well?This would be far easier than getting binaries together for all OS's...
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 23:24:29
Quote
The '\' vs '/' could be easily resolved using this simple script:
Code: [Select]
#!/bin/sh
for file in config*txt; do sed -i.bak -e 's|\\|/|g' ${file}; done


The encoding script would be a bit harder. Maybe it would be best to use wine so that we can be sure that the output is the same.

In my next test (whenever that might happen), I plan to work on alternative packages for people running *nix implementations (Linux, Darwin, Solaris, BSD...). That would consist of:

-Bash scripts instead of .bat files
-abchr.jar instead of abchr.exe
-No precompiled binaries of codecs, I would only hope people have mpg123, ogg123, flac, oggenc... on %PATH%.
-tar.gz instead of .zip 

In this test I have just thrown together the .jar in the abc-hr_bin package and hoped people would read the .bat file and sort out some way of doing the processing themselves. On next test it'll be a better thought process :B

Another thing that would be interesting would be that ff123 and schnofler agree on some sort of standard directory separator for the config files, so that there are no more issues of / and \. Of course, that is not really needed if we have one abc-hr package for each OS flavour.

Comments?

Regards;

Roberto.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 23:30:27
Quote
I was thinking.. if we could BitTorrent the uncompressed test files (Converted to wav) then we could all use them right? Or do the .bat scripts randomize their order as well?This would be far easier than getting binaries together for all OS's...

Bandwidth is not the problem anymore. Spoon has been kind enough to offer me 200Gb of bandwidth, and if that's not enough, Dibrom offered me about 500Gb :B

What keeps me from releasing all samples in lossless is that it would make things too hard for Dial-Up people. A sample package could go well up to 30Mb (even zip-compressed). Multiply it by 12, and you get around 350Mb for a person to download. On 56kbps, it would take well over 30 hours to download everything.

And no, the bat scripts aren't randomizing the order. If they were, there would be no way I could later tell which codec is which. And it would also making processing teh results a nightmare.
Title: 64kbps listening test
Post by: AstralStorm on 2003-09-18 23:32:47
Let's just write a 'configure' script and hope that people have automake.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-18 23:36:31
Quote
Let's just write a 'configure' script and hope that people have automake.

Hehe. You can try that yourself, I don't know that much about scripting.

BTW, I created the batch files following your templates, but couldn't get them to properly be converted to exe. One of the bat2exe tools I tried stopped the process at the middle of the batch, the other generated a GPF on ff123's PC. :B

Could you tell me what tool were you using? Maybe you can also try using it on this test's batches?

Thank-you.

Regards;

Roberto.
Title: 64kbps listening test
Post by: Jon Ingram on 2003-09-19 00:00:06
(thanks for giving me a reason to use HA again  ).

I don't have time to test all 12 samples. From the results you've got so far, are there any particular samples you'd prefer me to concentrate on? I can probably get 4-6 done before Sunday.
Title: 64kbps listening test
Post by: appleguru on 2003-09-19 00:00:21
Well, I'm really anxious to try this test. I happen to have broadband, and there's a good number of people that do to. If bandwidth isn't an issue, I think this would be a great way to get more people involved.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 00:05:58
Quote
(thanks for giving me a reason to use HA again  ).

I don't have time to test all 12 samples. From the results you've got so far, are there any particular samples you'd prefer me to concentrate on? I can probably get 4-6 done before Sunday.

Not really, whatever sample suits your musical tastes better, maybe

You can find more info about the style of each sample here:
http://www.hydrogenaudio.org/forums/index....showtopic=12358 (http://www.hydrogenaudio.org/forums/index.php?showtopic=12358)
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 00:10:21
Quote
Well, I'm really anxious to try this test. I happen to have broadband, and there's a good number of people that do to. If bandwidth isn't an issue, I think this would be a great way to get more people involved.

OK. I'll create sample packages of decoded WAVs, and upload them overnight.

It's a little late to do that, but if it can get more participants... why not?

But I'll definitely use RAR and not ZIP, because RAR compresses audio much better.

Fortunately, there are free command line RAR decompressors for nearly every platform (http://www.rarlab.com/rar_add.htm).

Regards;

Roberto.
Title: 64kbps listening test
Post by: pseudoacoustic on 2003-09-19 00:36:17
Quote
But I'll definitely use RAR and not ZIP, because RAR compresses audio much better.

Make sure you enable 'audio' compression (or 'multimedia' on older versions (there is a seperate 'true color' scheme for uncompressed bitmaps and videos)), and disable the others.

Also, see if increasing the prediction order and MB to use (all in the compression options button of the WinRAR program) increases compression ratios; it can in many cases, but does take much more memory to compress.
Title: 64kbps listening test
Post by: phong on 2003-09-19 03:31:55
Is there any reason not to use FLAC?  It's about as commonly available as RAR.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 03:48:41
Quote
Is there any reason not to use FLAC?  It's about as commonly available as RAR.

Well, I don't know if it's as commonly available on MacOS

Besides, RAR has the added feature of compressing solidly.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 05:45:56
Quote
Make sure you enable 'audio' compression (or 'multimedia' on older versions (there is a seperate 'true color' scheme for uncompressed bitmaps and videos)), and disable the others.

If I disable the others, the compressed files actually get bigger.

Quote
Also, see if increasing the prediction order and MB to use (all in the compression options button of the WinRAR program) increases compression ratios; it can in many cases, but does take much more memory to compress.


I think prediction is only used in Text encoding.

It's probably used by default on Audio encoding, because that's the logic way.

Thanks for the hints anyway.

Regards;

Roberto.
Title: 64kbps listening test
Post by: appleguru on 2003-09-19 05:52:07
BTW, On OS X, FLAC (Its called MacFLAC, latest version, 2.1.2) is availble with a nice GUI as well as Command Line binaries. It works quite Well. Rar is just as good tho
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 06:35:55
Quote
BTW, On OS X, FLAC (Its called MacFLAC, latest version, 2.1.2) is availble with a nice GUI as well as Command Line binaries. It works quite Well. Rar is just as good tho

Will, since I'm going for the whole enchilada, I might as well distribute the samples only packed.

I'm already uploading, it will take about 7 hours to finish the process. :B

And each package is average 25Mb.

I'll only announce these packages here. The test is messy enough as it is, I don't want to make things even worse (some people might think that the old packages were somehow wrong and they now should test with the new packages... you get the picture)

And, unfortunately, I can't postpone the test ending. I'll have to travel next week, so this Sunday is indeed the deadline.

The new packages will be announced tomorrow, as soon as I wake up (considering there'll be no issues while uploading). So, please check this thread in about 9 hours

Regards;

Roberto.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 07:42:27
OK, all packages have been uploaded now:

http://rarewares.hydrogenaudio.org/samples...es/Sample01.rar (http://rarewares.hydrogenaudio.org/samples/Sample01.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample02.rar (http://rarewares.hydrogenaudio.org/samples/Sample02.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample03.rar (http://rarewares.hydrogenaudio.org/samples/Sample03.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample04.rar (http://rarewares.hydrogenaudio.org/samples/Sample04.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample05.rar (http://rarewares.hydrogenaudio.org/samples/Sample05.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample06.rar (http://rarewares.hydrogenaudio.org/samples/Sample06.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample07.rar (http://rarewares.hydrogenaudio.org/samples/Sample07.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample08.rar (http://rarewares.hydrogenaudio.org/samples/Sample08.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample09.rar (http://rarewares.hydrogenaudio.org/samples/Sample09.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample10.rar (http://rarewares.hydrogenaudio.org/samples/Sample10.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample11.rar (http://rarewares.hydrogenaudio.org/samples/Sample11.rar)
http://rarewares.hydrogenaudio.org/samples...es/Sample12.rar (http://rarewares.hydrogenaudio.org/samples/Sample12.rar)

Remember that the folders must be laid out this way:
http://pessoal.onda.com.br/rjamorim/folder-setup.png (http://pessoal.onda.com.br/rjamorim/folder-setup.png)

So decompress the sample packages to folders with the same names as the packages.

Enjoy, and thanks for participating!

Regards;

Roberto.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-19 16:24:58
Updated the former post with all the package addresses.
Title: 64kbps listening test
Post by: Mac on 2003-09-19 17:50:00
Test completed 

I took 10 of the samples, as two of the music styles just plain offended me!  Thanks to Roberto I have a new found shame in my own hearing
Title: 64kbps listening test
Post by: AstralStorm on 2003-09-19 20:14:22
Quote
BTW, I created the batch files following your templates, but couldn't get them to properly be converted to exe. One of the bat2exe tools I tried stopped the process at the middle of the batch, the other generated a GPF on ff123's PC. :B

Could you tell me what tool were you using? Maybe you can also try using it on this test's batches?

The link: http://www.computerhope.com/download/utili...ity/bat2exe.com (http://www.computerhope.com/download/utility/bat2exe.com)
Actually it makes .COM files, but it doesn't matter as they work.
Tried with your new batches - they're all right except the cleanup. (weird)

I'll try another utility, maybe it will be better...
Title: 64kbps listening test
Post by: Jon Ingram on 2003-09-20 23:10:30
Thanks for a wide selection of tracks (although there wasn't enough vocal material to my mind) -- I enjoyed testing my ears out! Any ideas on when we'll get the results?
Title: 64kbps listening test
Post by: Mac on 2003-09-20 23:30:06
Knowing Roberto it will be only a few hours into the 22nd of September
Title: 64kbps listening test
Post by: rjamorim on 2003-09-20 23:42:37
Quote
Knowing Roberto it will be only a few hours into the 22nd of September

My former record, on the 128kbps test, was 3 hours after the closure. I plan to break it this time.

(By having as much as possible ready at the closure time)
Title: 64kbps listening test
Post by: ScorLibran on 2003-09-21 03:47:55
Finally...my work lets up a little and I can dedicate my PC for some testing...and with a day left to spare for this listening test!

I've downloaded the test start package, and I'm downloading two samples now.  I hate dial-up!  Anyway, I'll do the best I can to test as many samples as possible by midnight tomorrow.  Sorry for starting so late.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-21 03:58:31
@ScorLibran: no problem.


Well, the test will be closed in exactly 24 hours. After brazilian midnight (4AM GMT, 8PM California), I will accept no more result submissions and will start calculating the results.

For those that just noticed this thread, will have the day free tomorrow and want to help at least a little...

For some very strange reason, there are quite less results for samples 09 and 11, comparing to what I got for the other samples. Although what I have is most probably enough to obtain statistically significant values, the more results I have for each sample, the better.

So, if you didn't participate yet, or participated but didn't pick those samples, I invite you to test one of them, or both, and send the results to my e-mail address (available at the readme) before the mentioned deadline.

Thank-you very much.

Best regards;

Roberto.
Title: 64kbps listening test
Post by: Garf on 2003-09-21 07:56:11
Wait for me!
Title: 64kbps listening test
Post by: Garf on 2003-09-21 09:42:53
Quote
For some very strange reason, there are quite less results for samples 09 and 11, comparing to what I got for the other samples

Nothing strange, IMHO these are by far the easiest samples on the encoder and the hardest ones on the listeners. I think most people gave up because they couldn't identify more than one or two (or none) of the encoded clips.

Results sent.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-21 15:59:36
Quote
Nothing strange, IMHO these are by far the easiest samples on the encoder and the hardest ones on the listeners. I think most people gave up because they couldn't identify more than one or two (or none) of the encoded clips.

Makes sense. Still, it's good to know that some easy samples came out of this test, it makes things more or less equilibrated with harder samples (Waiting, EnolaGay).

Quote
Results sent.


Thanks a lot.
Title: 64kbps listening test
Post by: AstralStorm on 2003-09-21 16:27:07
Could you please enable some automated notification that the results arrived?
I've lost one file in 128k test due to my ISP.

BTW, did my results arrive?
Title: 64kbps listening test
Post by: tigre on 2003-09-21 16:48:05
Quote
Could you please enable some automated notification that the results arrived?
I've lost one file in 128k test due to my ISP.

BTW, did my results arrive?

Similar experiences/concerns/question here. Another possibility would be publishing a list (without results) of all participants that have submitted results so far ... - if it's not too much of an effort, of course. So everyone who's not on the list can send the results again.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-21 17:08:12
Quote
Similar experiences/concerns/question here. Another possibility would be publishing a list (without results) of all participants that have submitted results so far ... - if it's not too much of an effort, of course. So everyone who's not on the list can send the results again.

Yes, probably the list would be a good idea. Although it might raise some concerns about privacy... (for those that chose not to have their names mentioned on the test results)

The notification function is also possible, if I create another yahoo mail account. Then I can set an appropiate auto-reply.
That would not be a good idea with my current account because it's what I use for all putposes. My friends would be thrilled to get auto-replys "your results have been received" when they send me e-mail :B

@AstralStorm: yes, I got your results.
Title: 64kbps listening test
Post by: ScorLibran on 2003-09-22 02:44:54
Results sent!

And the result I'll remember the longest...Grado SR-80s hurt my ears!  That's it...I'm ordering Sennheisers tomorrow!  Not blaming Grado...they make great sounding cans.  My ears are just an odd size/shape, and they don't like the supra-aural Grados.  I just need something that doesn't press on them so much.

With the test, I could only do five out of twelve samples.  My dead-slow dial-up connection is downloading two more samples right now, one's at 23% and the other's at 29%.  They've been going for over an hour.  I wouldn't make it under the deadline waiting for them, so I sent what I was able to finish so far.  Maybe by the time of the next test I'll have DSL or something.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-22 02:56:23
One hour to go. If you haven't sent your results yet, I invite you to do so now.


@ScorLibran: Thanks a lot. Next test will hopefully feature smaller sample packages.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-22 04:06:35
Test is now CLOSED!

No result submissions will be accpeted from now on.

If people already want to start looking at their results and find out how they ranked each codec, here's the super-secret code that nearly everyone already broke:

1-MP3pro
2-Real Audio Gecko
3-WMA Std
4-FhG MP3
5-Lame MP3
6-HE AAC
7-LC AAC
8-Vorbis

Results out in a few hours.

Best regards, and huge thanks to everyone that participated.

Roberto.
Title: 64kbps listening test
Post by: phong on 2003-09-22 04:28:21
[edit]Removed per rjamorim's post.[/edit]
Title: 64kbps listening test
Post by: rjamorim on 2003-09-22 04:46:39
Quote
A different set of samples could see them in a much different order.

The order gets even more different if we use a different set of ears
Just wait for, EG, Guruboolez' results.

Anyway, I would like to ask that people don't comment on the test results here, and wait for the results announcement thread, that should be opened in a few hours. That way, comments don't get spread over several threads.

Thanks for your comprehension

Regards;

Roberto.
Title: 64kbps listening test
Post by: kl33per on 2003-09-22 04:47:54
Removed as per Roberto's above post.
Title: 64kbps listening test
Post by: ScorLibran on 2003-09-22 05:01:00
I just want to comment a moment on the "ease-of-use" of the test materials.  Roberto (and anyone else who helped prepare all of this) are to be commended for the ease with which the test could be executed by us participants.  This was my first involvement in one of the public listening tests, so maybe they've always been this way, but I'm new to all of this.

I was just worried I'd have to do a lot of encoding, test configuration, etc.  It was nice to simply download the start package and samples, then run a batch file and go.  Seamless configuration...an important feature for a newbie like me.   

Thanks Roberto!
Title: 64kbps listening test
Post by: phong on 2003-09-22 05:07:15
I'll have to agree with ScorLibran.  Both this test and the 128k test were quite easy to participate in thanks to Roberto's efforts.  Just compare that to the effort required to get a video to play under windows that happens to use the latest and greatest version of whatever-codec-is-cool-this-week.  Thanks!
Title: 64kbps listening test
Post by: guruboolez on 2003-09-22 09:01:36
Quote
Quote
Knowing Roberto it will be only a few hours into the 22nd of September

My former record, on the 128kbps test, was 3 hours after the closure. I plan to break it this time.

(By having as much as possible ready at the closure time)

No speed record this time : four hours since test closure :/ (I can return to bed )
Title: 64kbps listening test
Post by: bond on 2003-09-22 09:07:33
wake up again, the results are here 
Title: 64kbps listening test
Post by: rjamorim on 2003-09-22 09:16:27
Enough of this thread.

Move on here:
http://www.hydrogenaudio.org/show.php/showtopic/13464 (http://www.hydrogenaudio.org/show.php/showtopic/13464)



@Guru: Sorry, there were some issues to deal with.
Title: 64kbps listening test
Post by: Gecko on 2003-09-22 11:15:09
An idea to even further simplify things:

You could include wget in the distribution and make batch files that download the sample packages. There could be a "doall.bat" file that downloads all packages and runs the encoding/decoding, but I guess you will create more traffic that way, while not every downloaded sample is put to use.
Title: 64kbps listening test
Post by: rjamorim on 2003-09-23 05:40:23
Quote
An idea to even further simplify things:

You could include wget in the distribution and make batch files that download the sample packages. There could be a "doall.bat" file that downloads all packages and runs the encoding/decoding, but I guess you will create more traffic that way, while not every downloaded sample is put to use.

Well, there are lots of issues with that.

First, once the script starts running, you can't stop it anymore. If you do, there's no way to continue from where you left off.

Second, people that are on dial up and want to test only one or two samples would get pissed, because you can only start processing after you finished downloading.

Dial-up adds another issue that is, if the connection goes down at some time, you must restart from zero - again, because you can't restart the batch from where you left off.

And testing such a batch script would be a pain

Thanks for your suggestion but I believe that it's not really feasible

Best regards;

Roberto.
Title: 64kbps listening test
Post by: Gecko on 2003-09-23 10:13:46
Quote
First, once the script starts running, you can't stop it anymore. If you do, there's no way to continue from where you left off.

I believe you can ctrl-c out of it. Also wget -c or --continue should make it able to resume the script at any point. Allready downloaded files remain untouched and incomplete ones will be resumed. If all of the encoders/decoders can be set to overwrite the output no matter what, then I think all should be fine. If that is not the case, you could delete all the files that were encoded/decoded before restarting. Granted this would take some more time.
Quote
Second, people that are on dial up and want to test only one or two samples would get pissed, because you can only start processing after you finished downloading.

Agreed. My primary idea was to put the wget stuff in the beginning of each sampleXX.bat. So if a user wants to test sample04 he just has to doubleclick sample04.bat and it will be downloaded for him, extracted to the correct location and all of the encoding/decoding would be done. I thought that the "doall.bat" would only be interesting for broadband users. You could put a warning in the start, something like: "You are about to download 120mb... continue? [Y/N]". My main concern about that is still that you'd be wasting bandwidth with users who will just download everything.
Quote
And testing such a batch script would be a pain

You have a volunteer!
Quote
Thanks for your suggestion but I believe that it's not really feasible

Oh well.  It's not much an issue anyway. I think you can just "wget -i Readme.txt" and it will get every file that is referenced to. Add "-A zip" and it will only download zip files.