Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Dial-up bitrate listening test - Finished! (Read 29441 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Dial-up bitrate listening test - Finished!

Reply #25 –
Yes just thank you !
I guess you'll probably still be working on Rarewares

I expected Vorbis to do a little better. But I'm glad Nero (HE-AAC + PS) does so well. We need support in embedded devices now.

Dial-up bitrate listening test - Finished!

Reply #26 –
At last !
Thanks, Roberto !
---
Yeah, seems that SBR is good idea when bitrate is low...

Dial-up bitrate listening test - Finished!

Reply #27 –
Another Thanks from me, Roberto! You always did a great job with the tests - I hope there will be somebody who will continue...

Dial-up bitrate listening test - Finished!

Reply #28 –
Thank you Roberto, for all your listening tests, not just this one. They have all been very informative (to me). Good luck on your future.

I've been more of a lurker than a poster on these forums, but have learnt (and am still learning) a great deal from the collective knowledge of all the 'regulars' who frequent these forums.

Thank you!

Dial-up bitrate listening test - Finished!

Reply #29 –
Quote
I saw it. But even so, >900 results is a large number. So I'm surprised the margins are still 0.4 or so points wide.

For comparison, the previous test had no more than 300 results per sample and the margins were as small as 0.2!

I'd expect a nonparametric test to give larger margins, but this is extreme. Also, the requirement for a normal distribution should be much easier satisfied with this low bitrate than with the previous high bitrate tests, so the choice to use parametric before and nonparametric now looks very weird to me.
[a href="index.php?act=findpost&pid=228946"][{POST_SNAPBACK}][/a]

The margins are so wide beacuse results vary much between samples.

-Eugene
The  greatest  programming  project of all took six days;  on the seventh  day  the  programmer  rested.  We've been trying to debug the !@#$%&* thing ever since. Moral: design before you implement.

Dial-up bitrate listening test - Finished!

Reply #30 –
Quote
Hello.

I'd like to announce the results of the Dial-up bitrates (32kbps) listening test
Hi Rjamorim,

nice test, intersting results, great job.

One question. Do you have an excel-file with all results. I'd like to make my own statistic analysis.

thanks

Dial-up bitrate listening test - Finished!

Reply #31 –
w00t, awesome.

Dial-up bitrate listening test - Finished!

Reply #32 –
Quote
I guess you'll probably still be working on Rarewares
[{POST_SNAPBACK}][/a]


Sure. And even if I stopped, I believe John33 and xmixahlx would keep doing a good job maintaining it.

Quote
I expected Vorbis to do a little better.


Yeah, I hope Monty looks into enabling 32kHz encoding at 32kbps.

Quote
One question. Do you have an excel-file with all results. I'd like to make my own statistic analysis.


Sure. The Excel file is here:
[a href="http://www.rarewares.org/rja/32kbps.zip]http://www.rarewares.org/rja/32kbps.zip[/url]

I also included the result tables generated by chunky, so that you can feed them to Friedman using different parameters.

Please disregard the comments in the spreadsheet. They haven't been updated since the MP3 test.

Regards;

Roberto.

Dial-up bitrate listening test - Finished!

Reply #33 –
In order to reduce a slashdot effect I wgeted http://www.rjamorim.com/test/32kbps/results.html (--mirror --no-parent etc.) Now I am ready to zip it and share it via the ed2k net. Others might make a torrent or use another p2p application that has secure hashes (i.e. not KaZaA). Is that okay, or would I be infringing your copyright, rjamorim? Or the listening test participants' copyright? (Their comments are included in the zip as well.)

BTW crawling the results, I noticed that elmar3rd's results are not included, because the "#" isn't escaped in the href.

[span style='font-size:8pt;line-height:100%']EDIT: punctuation[/span]

Dial-up bitrate listening test - Finished!

Reply #34 –
Quote
In order to reduce a slashdot effect I wgeted http://www.rjamorim.com/test/32kbps/results.html (--mirror --no-parent etc.) Now I am ready to zip it and share it via the ed2k net. Others might make a torrent or use another p2p application that has secure hashes (i.e. not KaZaA). [{POST_SNAPBACK}][/a]


Well, that's a welcome initiative, but how it is going to help save bandwidth?

Maybe include the ed2k link in the Slashdot article, if it's ever posted?

If someone creates a torrent, I can put it at RW's tracker.

Quote
Is that okay, or would I be infringing your copyright, rjamorim?


Nah, all those pages are free for anyone to copy and mirror if they want.

I actually created a big mirror [a href="http://www.rjamorim.com/test/test.rar]archive[/url] of the listening tests page (except the samples folder), so that people can keep it and mirror elsewhere if rjamorim.com ever goes down.

Quote
Or the listening test participants' copyright? (Their comments are included in the zip as well.)


They are informed in the readme that all results are going to be published, so I believe they can't (and wouldn't) complain.

Quote
BTW crawling the results, I noticed that elmar3rd's results are not included, because the "#" isn't escaped in the href.


I'll ask Mac to fix the php indexer. You can also get the .rar mirror file to obtain his results.

Thank-you.

Dial-up bitrate listening test - Finished!

Reply #35 –
This line needs updating in the text of the explanation:

"For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence."

ff123

I wonder why Nero didn't do much better than mp3pro?  Are the advantages of aac over mp3 not as big at this bitrate?

Dial-up bitrate listening test - Finished!

Reply #36 –
Quote
"For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence."
[a href="index.php?act=findpost&pid=229169"][{POST_SNAPBACK}][/a]


Oops. Thank-you very much for pointing that out. Just fixed it.

Dial-up bitrate listening test - Finished!

Reply #37 –
roberto, thanks a lot for all your hard work conducing the listening tests! they were really more than very important in bringing light into the codec jungle


my comments on the results:
ahead: plz start investing much more time on working on your LC aac encoder!
I know, that I know nothing (Socrates)

Dial-up bitrate listening test - Finished!

Reply #38 –
Quote
I wonder why Nero didn't do much better than mp3pro?  Are the advantages of aac over mp3 not as big at this bitrate?
[a href="index.php?act=findpost&pid=229169"][{POST_SNAPBACK}][/a]


If you look you'll see all other codecs are close together too (hence my pendanticism about the analysis).

It's more like SBR being so big an advantage that it swaps out the other differences.

Dial-up bitrate listening test - Finished!

Reply #39 –
Quote
I wonder why Nero didn't do much better than mp3pro?  Are the advantages of aac over mp3 not as big at this bitrate?
[a href="index.php?act=findpost&pid=229169"][{POST_SNAPBACK}][/a]

There's imo still much more potential for the ND to develope at 32kbps compared to mp3pro, remember that this was the first semi-public version of this low-bitrate implementation which isn't actually even released yet officially but only for this test.
Juha Laaksonheimo

Dial-up bitrate listening test - Finished!

Reply #40 –
I'd like to point out that Nero Digital technology will certainly improve - as well as SBR in the future - at this moment, it has shown a great potential and definitely more space to become even better.

3.3 is not a bad score for 32 kb/s and beta technology

I would like to thank everybody who took part in this test.

Dial-up bitrate listening test - Finished!

Reply #41 –
Quote
I wonder why Nero didn't do much better than mp3pro? Are the advantages of aac over mp3 not as big at this bitrate?

I have the feeling that the AAC-LC part of the Ahead encoder is "far" from beeing optimal.
It does not mean that it is bad, but means that there is still an high progress margin.

note: from an external point of view, I think that it would have been a bad choice to spend a lot of time fine tuning the LC part if that would imply spending less time on SBR and PS.

Dial-up bitrate listening test - Finished!

Reply #42 –
Quote
I have the feeling that the AAC-LC part of the Ahead encoder is "far" from beeing optimal.
It does not mean that it is bad, but means that there is still an high progress margin.

note: from an external point of view, I think that it would have been a bad choice to spend a lot of time fine tuning the LC part if that would imply spending less time on SBR and PS.
[a href="index.php?act=findpost&pid=229337"][{POST_SNAPBACK}][/a]


I feel it too.
I compare mid-low bitrate of  Nero VBR AAC-LC with those of Quicktime (CBR), (for using on my mobile phone, which does not support AAC-HE) I think there are a lot of room for improvement.

When Quicktime offers its VBR encoder, Ahead will get serious competition on mid-to-high bitrate. Or Nero decide to avoide competition and focus only on low bitrate side?

Dial-up bitrate listening test - Finished!

Reply #43 –
Based on it's performance in the 128kbps test, I didn't expect Vorbis to be tied with the back of the pack in this one.  Just goes to show that formats perform very differently at different bitrates when compared to other formats.

Thanks for all the work you've done on this test, Roberto, and for the work you've done on all the others.  You've been an invaluable part of the audio encoding community with these selfless contributions of time and effort.

Best of luck to you in your future aspirations. 

Dial-up bitrate listening test - Finished!

Reply #44 –
Hi Roberto,
Did you get a chance to to a total music only results chart?  If so where is it posted? 

Thanks,
Birch

Dial-up bitrate listening test - Finished!

Reply #45 –
Quote
Hi Roberto,
Did you get a chance to to a total music only results chart?  If so where is it posted?


i would be interested in those too. also the results with anova analysis (for comparability purposes with the other tests) would be nice.

Dial-up bitrate listening test - Finished!

Reply #46 –
Surprisingly enough, I'm on another trip :B

I'll try to produce these graphs next monday.

Dial-up bitrate listening test - Finished!

Reply #47 –
A late thank you to Roberto as well!  BTW, excuse the noob question but how does one view the .ecf files created in the test? I'd like to review my results and I guess they need to be decrypted in some fashion.

Dial-up bitrate listening test - Finished!

Reply #48 –
Quote
how does one view the .ecf files created in the test? I'd like to review my results and I guess they need to be decrypted in some fashion.
[{POST_SNAPBACK}][/a]


On ABC-HR Java, go to Tools -> Process test results.

Once there, you can load the .ecf files and decrypt them with the admin key:
[a href="http://www.rjamorim.com/test/32kbps/comments/32kbps.key]http://www.rjamorim.com/test/32kbps/comments/32kbps.key[/url]

You can also convert them from XML to txt. txt is easier for reading, and XML is easier for parsing.

Regards;

Roberto.

Dial-up bitrate listening test - Finished!

Reply #49 –
Again, thanks. I found that I most consistently rated WMA and mp3pro the highest followed by nero and qdesign.  I think the grainyness of SBR prevented me from voting for mp3pro and nero as the clear winners.