Hydrogenaudio Forums

Hydrogenaudio Forum => Listening Tests => Topic started by: IgorC on 20 March, 2011, 08:32:56 PM

Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 20 March, 2011, 08:32:56 PM
There has been significant advancement since the last 64kbps test. The Nero HE-AAC and Aoyumi Vorbis encoders improved, Apple added HE-AAC capability, and we have a new low delay codec claiming high quality: CELT.

The following codecs are presented in this test:
Nero 1.5.4 HE-AAC
Apple QuickTime 7.6.9 HE-AAC
Ogg Vorbis AoTuV 6.02 Beta
CELT 0.11.2

Public Multiformat Listening Test @ ~64 kbps [March 2011] (http://listening-tests.hydrogenaudio.org/igorc/Public%20Multiformat%20Listening%20Test%20@%2064kbps.htm)

Every single result is helpful.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: hödyr on 20 March, 2011, 08:48:21 PM
Hahaha. Very interesting. Looking forward to take the challenge .
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: romor on 20 March, 2011, 09:20:51 PM
I hope to see interesting results, too

Yesterday, I was processing large amount of recorded speech in 128K CBR stereo (L=R) MP3s to AAC through iZotope Nectar, for use on portable device
I run couple of tests and settled at "neroAacEnc -hev2" as most desirable. Using "-br 48000" or 56000, produced unwanted "smearing" artifacts on some accented words, while forcing "-hev2" (although unrecommended) went fine. Reason may be in processing transcoding chain, but I thought to mention it

Also, I was wondering what's going on with Speex? Is there relation with CELT developement, as is with CELP? And 1.2rc1 is dated in 2008
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Nothbeve on 21 March, 2011, 03:20:11 AM
can i report my listening test of those samples?
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 21 March, 2011, 03:58:34 AM
can i report my listening test of those samples?


That's the idea, see "How do I take the test?" on the webpage.

If it's about discussing the kind of artifacts you heard and so on: it's usually better to postpone that after the test completes, to prevent biasing people who still want to take the test.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: ExUser on 21 March, 2011, 05:42:28 AM
Horrible timing for me, it's semester-end at school. IDK how many other HAers are still in the academic world, but I'll bet there are a few.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: DonP on 21 March, 2011, 06:46:36 AM
Also it's almost tradition to open discussion about future public test but this time there weren't too much time for it and  I sincerely hope that you, people,  will understand.


Something inherent in the test that there isn't time?

Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 21 March, 2011, 06:56:41 AM
Note: Important workaround was made. It is not uncommon to see that some soundcards have isues with not 48 kHz sample rates. Aslo until moment CELT is optimized for 48 kHz while other codecs are well tested at 44.1 kHz. To avoid these limitations all decoded files (Vorbis and Nero, Apple HE-AAC) were resampled with very highly transparent resampler SOX VHQ while CELT files were directly encoded from the resampled source. The resampling hadn't impact on final results.

Also it's almost tradition to open discussion about future public test but this time there weren't too much time for it and I sincerely hope that you, people, will understand.
The most part of samples come from previous public tests to avoid any kind of bias but also some new samples have been included.


Donp,
I have very limited spare time now. It's like  two-three weeks. There is no chance to discuss all conditions and perform the test itself during such short time.
I will be glad if anyone else will organize tests in future
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: jmvalin on 21 March, 2011, 11:58:46 AM
Donp,
I have very limited spare time now. It's like  two-three weeks. There is no chance to discuss all conditions and perform the test itself during such short time.
I will be glad if anyone else will organize tests in future


Thanks very much IgorC and Garf for organizing this.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: misuzu on 21 March, 2011, 01:32:17 PM
Nobody use the lower bit to store music

I think 112k or 128k (not 140k) is useable
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: markanini on 21 March, 2011, 01:33:34 PM
Donp,
I have very limited spare time now. It's like  two-three weeks. There is no chance to discuss all conditions and perform the test itself during such short time.
I will be glad if anyone else will organize tests in future


Thanks very much IgorC and Garf for organizing this.

+1

I've done samples 1-7 so far. Looking forward to completing all tests during the week.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: markanini on 21 March, 2011, 01:41:21 PM
Nobody use the lower bit to store music

I think 112k or 128k (not 140k) is useable

Streaming audio would perhaps be the one of the more common uses. With mp3 codecs at such bit rates quality is quite atrocious but with modern codecs like AAC-HE it can be perfectly acceptable. 64kbps is the new 128kbps! 
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 21 March, 2011, 01:49:25 PM
Nobody use the lower bit to store music

I think 112k or 128k (not 140k) is useable

Streaming audio would perhaps be the one of the more common uses.


Yes. Also, testing at 128kbps doesn't make sense anymore - just look at the result of the last mp3 test...
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Jillian on 21 March, 2011, 02:02:29 PM
I want to ask everyone. Before the test finish, which codec will win? Of course you may have to take a guess/bet or based on your experience/personal preference.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: vinnie97 on 21 March, 2011, 02:35:43 PM
Great!  I was just thinking, with all the development happening with the codecs, it was past due for another such test. ;-)  So Apple is now in the race, eh?  Ugh.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Sebastian Mares on 21 March, 2011, 04:05:17 PM
Yay, new listening test!  Thanks for organizing guys!. Hope Nero wins.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: /mnt on 21 March, 2011, 06:04:58 PM
Cool, a new public test, shame that the AAC test did not take off.

Hopefully I should find some free time to try out some samples this weekend.

Horrible timing for me, it's semester-end at school. IDK how many other HAers are still in the academic world, but I'll bet there are a few.

Same proplem with me aswell, am doing my first year of Computer Science at Uni and gotta get assigments completed before easter holidays. Also doing a group with students does not help at all esp that they decided to start last minute for our Problem Solving presentation.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 21 March, 2011, 09:17:34 PM
Yay, new listening test!  Thanks for organizing guys!. Hope Nero wins.

Thanks to you for previous tests that help us  a lot to prepare this test.

Cool, a new public test, shame that the AAC test did not take off.

 

Same proplem with me aswell, am doing my first year of Computer Science at Uni and gotta get assigments completed before easter holidays. Also doing a group with students does not help at all esp that they decided to start last minute for our Problem Solving presentation.


I understand you, guys. I was in the same situation during last public MP3 128 test. I couldn't participate. It's not easy study and later do some blind test activity. Too much activity for brain. But hopefully it's not that hard at ~64 kbps.

/mnt,  personally your results will be very interesting for me.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 22 March, 2011, 07:16:43 AM
The list of the samples:

1.poets of the fall
2. bachpsichord
3.spmg54_1 (german speech)
4. tomsdiner
5. kraftwerk
6.macabre
7. bibilolo
8.Stravinskii Capriccio
9. fly_to_fly
10. fatboy
11. les voleurs
12. eig
13. enter sandman
14. big yellow
15. davinci
16. Castanets
17. Linchpin
18. waiting
19. Bittersweet           
20. Dimmu borgir orchestra
21. bleed
22. Sangue de bairro
23. Against
24. stars
25. worlds apart
26. take your finger frin my head
27. i want to break free
28. atrain
29. bonhemian rhapsody
30. on the roof with Quasimodo

The most part of the samples are from previous public tests.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: lessthanjoey on 23 March, 2011, 01:35:21 AM
I've been lurking awhile but this is my first post - it might seem like a strange thread to post in but I would like to participate in this and I'm having some trouble.

Specifically, ABC/HR doesn't run correctly on my HTPC where I do most of my listening. I've tried a few variants of Java (including one which runs ABC/HR just fine on my laptop) and all hang with a not responding on startup (it creates the small window but the interior never gets rendered). Both machines (laptop and HTPC) are Win 7 Pro, laptop works fine with Java 6 Update 24, HTPC doesn't. I've tried 32-bit, 64-bit, no difference.

Any suggestions?

EDIT: Turns out it was an issue with remote desktop. No problems running with a keyboard attached to the machine directly. Strange.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Fool_on_the_hill on 23 March, 2011, 06:15:36 AM
Hello! Can't run abchr.jar:
Could not find the main class:
c:\Users\1\Downloads\ABC-HR\abchr.jar. Program will exit now.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 23 March, 2011, 08:48:10 AM
Hello! Can't run abchr.jar:
Could not find the main class:
c:\Users\1\Downloads\ABC-HR\abchr.jar. Program will exit now.


Possible issues I can think of:
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 23 March, 2011, 09:01:10 AM
Hello! Can't run abchr.jar:
Could not find the main class:
c:\Users\1\Downloads\ABC-HR\abchr.jar. Program will exit now.

More probably old version of Java.
You can update Java or replace .jar files from this package http://www.hydrogenaudio.org/forums/index....st&p=701994 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77994&view=findpost&p=701994)
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Fool_on_the_hill on 23 March, 2011, 12:04:43 PM
Java 6 update 24
Windows 7 64-bit with latest updates

The archive is unzipped.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 23 March, 2011, 02:40:10 PM
Have you tried to replace abchr.jar by the same file from here http://www.hydrogenaudio.org/forums/index....st&p=701994 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77994&view=findpost&p=701994)?
You should also add two other .jar files as well. (jdom.jar, looks-1.3.1.jar)

Hope that fix it.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: C.R.Helmrich on 23 March, 2011, 04:26:28 PM
I want to ask everyone. Before the test finish, which codec will win? Of course you may have to take a guess/bet or based on your experience/personal preference.

I have certain vague expectations about which codec will win, but I won't tell you (and I ask everyone else with knowledge about the codecs under test not to do so either) since this could influence the test.

A (IMHO) more important question: Igor, are all session results excluded for which some hidden reference is graded with less than 5.0? Maybe you should mention the post-screening procedure so that listeners behave carefully enough.

And about the resampling: Except for CELT, was the input or the output of the codecs resampled to 48 kHz?

Chris
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 23 March, 2011, 04:41:25 PM
Hello, Chris.

A (IMHO) more important question: Igor, are all session results excluded for which some hidden reference is graded with less than 5.0? Maybe you should mention the post-screening procedure so that listeners behave carefully enough.

You're right.
Here are the rules:
Code: [Select]
If at least one of the following errors takes place then the result will be INVALID.
1. graded the reference lower than 5.0
2. didn't grade the low anchor.

If one listener will submit INVALID result then he will be informed and will have one more and unique posibility to submit result for one particular sample.
Now if the listener will submit 3 or more invalid results then only ABX results will be accepted from him/her or directly rejected in abussive cases.


And about the resampling: Except for CELT, was the input or the output of the codecs resampled to 48 kHz?

Chris

The output of decoded (vorbis, aac) files was resampled.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Fool_on_the_hill on 24 March, 2011, 06:31:37 AM
Tried to abchr.jar (+jdom.jar, looks-1.3.1.jar) from here http://www.hydrogenaudio.org/forums/index....st&p=701994 (http://www.hydrogenaudio.org/forums/index....st&p=701994)?
The same result.  Strange...
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: LithosZA on 24 March, 2011, 09:17:02 AM
Quote
Tried to abchr.jar (+jdom.jar, looks-1.3.1.jar) from here http://www.hydrogenaudio.org/forums/index....st&p=701994 (http://www.hydrogenaudio.org/forums/index....st&p=701994)?
The same result. Strange...


You could try uninstalling the 64bit version of JRE/JDK and try using the 32bit version of the java runtime instead.
Don't think it will make a difference, but it is worth a try.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Sebastian Mares on 24 March, 2011, 11:26:34 AM
Your JRE is probably too new.  Had this problem in my tests, too.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 24 March, 2011, 11:48:45 AM
Thank you, Sebastian.

There are two versions of ABC-HR for Java 5 and 6. http://www.hydrogenaudio.org/forums/index....st&p=683924 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77573&view=findpost&p=683924)

Fool_on_the_hill, you can try it

Code: [Select]
The Java 5 build should also run fine on Java 6 machines. In both cases just double-click the jar to start.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: C.R.Helmrich on 24 March, 2011, 05:34:55 PM
Some relevant knowledge collected for the 128-kb test that didn't take place (yet...):

Instruction sheet for audio codec listening tests (high quality) (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77272&view=findpost&p=708363)

(German: eine deutsche Version der Anleitung liegt hier (http://www.ecodis.de/audio/anleitung_hoch.html)).

The best codecs in this 64-kb test can still be considered high-quality, so the description is applicable.

By the way, two posts above mine in that thread is a Mac binary of faad. In case any Mac user needs it. [Edit: unfortunately, not any more  ]

Chris
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Alex B on 25 March, 2011, 05:17:18 AM
Thanks for organizing this test.


To help the testers who do not have access to a Windows computer I created FLAC versions of the decoded samples and put them online on Dropbox.

I decided to pack the sample folders into six zip files (a compromise between storing them individually or in a single package). Each zip file contains five sample folders:

http://dl.dropbox.com/u/7690288/samples01-05.zip (http://dl.dropbox.com/u/7690288/samples01-05.zip) (48.6 MB)
http://dl.dropbox.com/u/7690288/samples06-10.zip (http://dl.dropbox.com/u/7690288/samples06-10.zip) (67.3 MB)
http://dl.dropbox.com/u/7690288/samples11-15.zip (http://dl.dropbox.com/u/7690288/samples11-15.zip) (67.6 MB)
http://dl.dropbox.com/u/7690288/samples16-20.zip (http://dl.dropbox.com/u/7690288/samples16-20.zip) (52.7 MB)
http://dl.dropbox.com/u/7690288/samples21-25.zip (http://dl.dropbox.com/u/7690288/samples21-25.zip) (61.6 MB)
http://dl.dropbox.com/u/7690288/samples25-30.zip (http://dl.dropbox.com/u/7690288/samples25-30.zip) (67.7 MB)

I compressed the original, decoded 48 kHz WAVE test samples with FLAC 1.2.1 (-8) and verified the resulting FLAC files with foobar2000's bit compare tool.

The sample folders are included in the zip files. You must unzip the folders and decompress the FLAC files back to WAVE before they can be used in the test. The new filename extension must be .wav instead of .flac, otherwise the filenames must be unaltered. The folders must be placed in the ABC-HR folder:

ABC-HR > Sample01 > the six uncompressed .wav sample files
ABC-HR > Sample02 > the six uncompressed .wav sample files
ABC-HR > Sample03 > the six uncompressed .wav sample files
etc.

The 30 sample folders (= 180 WAVE files) will need about 712 MB of disk space.

You can have the FLAC files in the same folders if you don't want to delete them. They will just occupy some additional disk space (about 366 MB for all 30x6 samples).


Please don't abuse the bandwidth that is available for this Dropbox account. It makes no sense to download these versions if you can use the smaller original package (or packages) and let the bat file (or files) automatically process the test samples.

EDIT

If anyone else wants to host these FLAC versions I can provide also the individual sample packages and all samples in a single package (PM me).
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 25 March, 2011, 06:08:13 AM
Thank you, AlexB
That is exactly what people asking with Linux and OS X.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 25 March, 2011, 12:25:33 PM
Here is more clear interpretation of rules:

Quote
Participants who don't want to worry too much about the grading rules
can simply ignore them.  Listeners should do their best to rank the samples and
be careful to identify the hidden references. Listeners should ABX tests
they are at all unsure.

1) If the low anchor is not graded, or if any hidden reference is graded
below 4.5-5(*1a) (see App.) the result is INVALID.

*1a It will be discussed with people who have experience of conducting public tests when final results will be ready.

2) For each sample with a ranked reference or an ungraded low anchor the
listener will have a single chance to submit a replacement test run for
that sample. The replacement test must cover all codecs, not just the
codecs with the ranked reference. (This also covers cases where the
reference is ranked but still at or above 4.5)

3) If a listener submits more than 2/15 (4 for 30 samples submitted) INVALID
results then only ABX results will be accepted, or the listener will be excluded
completely in cases of apparently abusive behavior.


App. These rules aren't extremely strict in order to allow for simple human
error while still excluding careless participants.

A stricter procedure to exclude all ranked references risks a systemic
bias against any codec which are very good on a few samples and thus
subject to more reference confusion by causing those samples to be excluded
and weighing the test towards other samples.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Alex B on 25 March, 2011, 12:39:19 PM
I just noticed that this test is not in the forum news.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 25 March, 2011, 09:26:05 PM
I just noticed that this test is not in the forum news.

Oh, it will be great to have it there
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: ExUser on 25 March, 2011, 10:37:10 PM
Create a post in the News Submissions forum and I'll see what I can do.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Alex B on 26 March, 2011, 05:24:32 AM
Actually the first post in this thread is the news announcement.

During the past tests we usually had 1) a pretest discussion in "Listening Tests", 2) a news thread in which the open test was announced and commented and finally 3) a news thread in which the results were published and commented.

For example, the previous Multiformat 64 kbps test:

1) pretest discussion: http://www.hydrogenaudio.org/forums/index....showtopic=53134 (http://www.hydrogenaudio.org/forums/index.php?showtopic=53134)
2) announcement: http://www.hydrogenaudio.org/forums/index....showtopic=56397 (http://www.hydrogenaudio.org/forums/index.php?showtopic=56397)
3) results: http://www.hydrogenaudio.org/forums/index....showtopic=56851 (http://www.hydrogenaudio.org/forums/index.php?showtopic=56851)

This time we did not have a pretest discussion (at least not in a single dedicated thread). This thread is already the news & comments thread.


To: Admins/Mods

- Could you simply create a short (perhaps slightly edited) news article from the first post that would point to this thread?
or
- Could you move this thread to "Validated News", create a short (perhaps slightly edited) news article from the first post that would point to this thread and perhaps leave here, in "Listening Tests", a short note about the moved thread and a link to the new location?
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 26 March, 2011, 11:29:19 AM
AlexB has a point. We need the announcement of this test on News Section. This way more members will participate.

Some members have already sent their results. 
it's on approximately half way to get enough results.
Hopefully people will have time to conduct  test during this weekend.

Hydrogenaudio community, please, participate in test! 
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 27 March, 2011, 12:18:26 PM
BTW,  If you don't want to download separate packages there is All-In-One package (ABC-HR and all samples)
It should be more easy.

ABC-HR_bin_and_samples (http://listening-tests.hydrogenaudio.org/igorc/ABC-HR_bin_and_samples.zip) (All-In-One)
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: LithosZA on 27 March, 2011, 04:31:01 PM
The test was really fun. I was supprised how nearly transparent some of these samples were at 64Kbit/s.
I would encourage that more people take this test.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Fool_on_the_hill on 28 March, 2011, 08:24:27 AM
Thank you, Sebastian.

There are two versions of ABC-HR for Java 5 and 6. http://www.hydrogenaudio.org/forums/index....st&p=683924 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77573&view=findpost&p=683924)

Fool_on_the_hill, you can try it

Code: [Select]
The Java 5 build should also run fine on Java 6 machines. In both cases just double-click the jar to start.


still the same error
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 28 March, 2011, 01:46:42 PM
still the same error


It's a sort of particular incompatibility. I have the same Windows 7 x64 and Java 5/6 packages work fine on it. Also have checked on Window XP machine. No problem. 
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 29 March, 2011, 02:24:27 PM
Some people wait for the last day to send their results. That's ok. 
But it might be the case that someone can forget to send the results until 30th of March.
So if You have already even a few results send them soon.

Of course, the last moment results will be accepted during 30th and 31st (~01.00 of GMT -3.00) of March.

As far as I can see all participants have received the answer. Thank You  to All who participate.

Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Sebastian Mares on 29 March, 2011, 02:41:07 PM
I usually got a decent number of results on the last day from people who wanted to test more but didn't make it in time.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: markanini on 29 March, 2011, 05:56:50 PM
I've had fun doing the tests but please make the samples shorter next time so that the participant can more easily establish points in the audio to focus on and avoid fatigue. For example sample 15 starts of with a female voice and then switches to a male voice, forces you to acquire a split personality
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: lessthanjoey on 30 March, 2011, 12:44:37 AM
I'm extremely impressed by all four (non low anchor) codecs. Two in particular appear essentially transparent to my ear across the entire set of samples. I can only hope that those two are consistent and it's not a cycling of which is best in each case
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: romor on 30 March, 2011, 01:45:34 AM
Two in particular appear essentially transparent to my ear across the entire set of samples. I can only hope that those two are consistent and it's not a cycling of which is best in each case
Notices the same - I downloaded bundle out of curiosity and 2 codecs perform superbly on couple of samples I tested

2m to midnight - is there minimum amount of samples that can be reported? Like 1/3 of all or 10 for reckless ones? 

edit: I realised now (readme.txt) that it's not necessary to test all samples, sorry for wasting space
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 30 March, 2011, 02:40:24 PM
The test is extended until the 10th of April.
We still need more results for some particular samples.
I will post the numbers of those samples in a few minutes.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Alex B on 30 March, 2011, 03:01:17 PM
The test is extended until the 10th of April.
We still need more results for some particular samples.

Thanks, I was just about to request for an extension.

I noticed this thread only 5 days ago (didn't expect a new test) and did 5 samples on last Sunday. This week has been busy for me and I may not have any spare time and energy left for testing more samples before the next weekend.

I will post the numbers of those samples in a few minutes.

I was also going ask for a list of the samples that do not have enough results yet.


Since the test has now been extended, you could copy your first post in this thread, add a link that points to this thread, add a note about the new deadline, and post it to the News Submissions forum: http://www.hydrogenaudio.org/forums/index.php?showforum=22 (http://www.hydrogenaudio.org/forums/index.php?showforum=22)

For some reason no admin or mod has directly announced the news, even though we requested that earlier. The "news submission" route is quite unnecessary for a HA listening test. Who ever organized the hosting of the samples (was it Garf?) could have posted the news in the very beginning.

EDIT

If (hopefully when) the news post is validated and published on the front page this thread could be linked as the comments thread and moved to the Validated News forum as I requested earlier: http://www.hydrogenaudio.org/forums/index....st&p=749581 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=87576&view=findpost&p=749581)
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 30 March, 2011, 03:38:08 PM
Thanks, I was just about to request for an extension.

Don't be angry, but from Day 0 it was clear that test will be extended.    Previously such test had to be extended twice http://listening-tests.hydrogenaudio.org/sebastian/mf-64-1/ (http://listening-tests.hydrogenaudio.org/sebastian/mf-64-1/)
There are a lot of people who still have not finished to submit their results and they are important.


I was also going ask for a list of the samples that do not have enough results yet.

We need more results for samples 11 and 14-30 (14,15....29,30) and results for other samples are welcome as well.

Since the test has now been extended, you could copy your first post in this thread, add a link that points to this thread, add a note about the new deadline, and post it to the News Submissions forum: http://www.hydrogenaudio.org/forums/index.php?showforum=22 (http://www.hydrogenaudio.org/forums/index.php?showforum=22)

Done (http://www.hydrogenaudio.org/forums/index.php?showtopic=87785&pid=750182&st=0&#entry750182)
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 30 March, 2011, 03:52:48 PM
Since the test has now been extended, you could copy your first post in this thread, add a link that points to this thread, add a note about the new deadline, and post it to the News Submissions forum: http://www.hydrogenaudio.org/forums/index.php?showforum=22 (http://www.hydrogenaudio.org/forums/index.php?showforum=22)

Done (http://www.hydrogenaudio.org/forums/index.php?showtopic=87785&pid=750182&st=0&#entry750182)


This was completely useless for the news page so I just cannibalized it.

Quote
For some reason no admin or mod has directly announced the news, even though we requested that earlier. The "news submission" route is quite unnecessary for a HA listening test.


No admin or mod approved anything because nothing suitable was posted to the news submissions forum.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Alex B on 30 March, 2011, 04:00:33 PM
Thanks.

I think IgorC did a small mistake when he did not post the first post of this thread to the "News Submissions" like Sebastian used to do. (For example: http://www.hydrogenaudio.org/forums/index....howtopic=56397) (http://www.hydrogenaudio.org/forums/index.php?showtopic=56397)).
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 30 March, 2011, 04:02:28 PM
Thanks.

I think IgorC did a small mistake when he did not post the first post of this thread to the "News Submissions" like Sebastian used to do. (For example:
http://www.hydrogenaudio.org/forums/index....howtopic=56397) (http://www.hydrogenaudio.org/forums/index.php?showtopic=56397)).


I would have been rejected anyway (or just sat there) because the format of the post was totally unsuitable.

Edit: I've edited it into submission, but I had to move part of IgorC's comment into his next post. Merged both threads. Should be fine now.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 30 March, 2011, 04:17:47 PM
Thank you, Garf.
Sorry for the mess.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Alex B on 30 March, 2011, 05:18:07 PM
Of course you could have followed the rules at http://www.hydrogenaudio.org/forums/index....showtopic=20857 (http://www.hydrogenaudio.org/forums/index.php?showtopic=20857).

I.e. posted only the bare minimum, essential "news" info in the first post and...
... All other additional information like screenshots, full changelogs, personal opinions/comments may be added in a 2nd post, provided that they are really important.

However, this a collaborative effort. IgorC is not alone. Others who are more experienced can and should help (as Garf did now).

I just did a quick calculation of the work effort that is requested from the testers. Assuming that we would get 20 results for each sample and each codec would need 2-3 minutes of testing time, making 10-15 minutes for each sample, 30 samples would need 30 x 20 x 12.5 minutes = 125 hours of intensive testing work. On average each tester would spend 6 hours and 15 minutes for testing all 30 samples. Actually, I don't know if 2-3 minutes per codec is even a realistic estimate. In my experience at least some samples/codecs that are difficult to test need more time.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Dakeryas on 31 March, 2011, 08:38:42 AM
I've done the following "Once ABC/HR is open, click "Open ABC/HR Config..." and load the file "SampleXX.ecf"" after having unzipped all samples according to the .png and having launched the DecodeAll.bat.

I can change the playback range, add comments, but no sound would play when clicking on either "Play sample (q)" or "Play sample (w)" or "Play reference (e)". Any ideas?
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 31 March, 2011, 08:40:15 AM
I've done the following "Once ABC/HR is open, click "Open ABC/HR Config..." and load the file "SampleXX.ecf"" after having unzipped all samples according to the .png and having launched the DecodeAll.bat.

I can change the playback range, add comments, but no sound would play when clicking on either "Play sample (q)" or "Play sample (w)" or "Play reference (e)". Any ideas?


Check the settings to see which output audio device is selected.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: ExUser on 31 March, 2011, 10:57:25 AM
The test is extended until the 10th of April.
We still need more results for some particular samples.
I will post the numbers of those samples in a few minutes.
Awesome. This should give me a window of a day or two to tear through some samples. Thanks for the extension.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: mixminus1 on 31 March, 2011, 11:40:58 PM
I'm in the process of plowing through all 30, and should be done by tomorrow afternoon.

HTH!
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Dakeryas on 02 April, 2011, 02:14:15 PM
I've done the following "Once ABC/HR is open, click "Open ABC/HR Config..." and load the file "SampleXX.ecf"" after having unzipped all samples according to the .png and having launched the DecodeAll.bat.

I can change the playback range, add comments, but no sound would play when clicking on either "Play sample (q)" or "Play sample (w)" or "Play reference (e)". Any ideas?


Check the settings to see which output audio device is selected.



Solved because the microphone was indeed the default output, many thanks !

I'll test a little as from now on!
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 06 April, 2011, 04:15:17 PM
Thank you to everyone for their results.
All participants have received the answer.
If someone haven't receive it then post here and also can send me  PM or email.

Some participants initially had problems with ABC-HR  Java because previously their ABX tool was other (like foobar ABX, ff123's)
I will ask if someone  already has a few results send them as soon as possible to avoid the loss of their time. This way there is time to analyse them and report to listener if everything is good.

More results are needed for samples 9,11,13,14 and 16-30 (16,17,...,29,30). Especially for 9,11,27,29.
Though results for other samples are welcome as well if the listener has enough time.


Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 07 April, 2011, 10:03:54 PM
The test will end on 10th of April  (at 11:00 of GMT -3:00). Definitely.
Finally there will be the results. 



Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 09 April, 2011, 10:47:34 PM
Reminder.
The test will end in less than 12 hours.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: enry2k on 10 April, 2011, 03:14:35 AM
One of the samples is the Susan Vega's Tom's diner song, the track originally used by the mpeg researchers to tune up the MP3 format:-)
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: jmvalin on 10 April, 2011, 08:21:28 AM
One of the samples is the Susan Vega's Tom's diner song, the track originally used by the mpeg researchers to tune up the MP3 format:-)


I think that applies to *all* audio codecs, really.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 10 April, 2011, 10:02:28 AM
The test is finished.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: sauvage78 on 10 April, 2011, 12:58:00 PM
IgorC:
Whatever the result, I am grateful that you helped jmvalin with CELT when so far I miserably failed to find the motivation to help him. 
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 10 April, 2011, 01:30:20 PM
Thank you
Sure, no problem 

The main purpose of this test was directed not only for us (users) who are interested to know what codec to use in some particular scenarios but also to help to developers of AAC, CELT and Vorbis to improve their encoders.


All results have been already processed.
Though the approval of other organizator is still required.
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: mixminus1 on 10 April, 2011, 02:41:08 PM
IgorC:
Whatever the result, I am grateful that you helped jmvalin with CELT

So am I!

All I'll say about CELT right now is:  "Impressive.  Most impressive."
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: Garf on 11 April, 2011, 06:23:24 PM
I just confirmed IgorC the results look OK and sent some additional analysis.

Thanks to everyone that submitted results. There were over 500 valid ones! Thanks to this, the results of the test are highly significant, and will allow for some clear conclusions. Listening tests like this are not possible without volunteers that put in some of their free time, so really, this is your accomplishment. Also thanks to IgorC for conducting the test and setting up everything.

Now that I've seen the results I must say, they aren't even remotely close to what I expected! A lot has changed in 4 years...

So IgorC, hurry up on those webpages to satisfy everybodies curiosity
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 11 April, 2011, 07:44:30 PM
thank you, Garf

and thank to all participants
http://www.hydrogenaudio.org/forums/index....showtopic=88023 (http://www.hydrogenaudio.org/forums/index.php?showtopic=88023)
Title: Public Multiformat Listening Test @ ~64 kbps [March/April 2011]
Post by: IgorC on 11 April, 2011, 11:44:55 PM
I would ask to Admins to close this topic.
The test is end.

Thank you.