Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal 64kbps listening test; multi-format (Read 18453 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal 64kbps listening test; multi-format

Introduction
I recently completed a listening test of several encoders for portable use.  The direction of the test was a low bitrate, for fitting a lot of music on a smaller flash player.

Bitrate
Most flash player families have several models with capacity stepping up by a factor of 2, e.g. 512Mb/1Gb/2Gb.  I was interested to see what drop in quality I'd have to live with if I bought the next size down but wanted to store the same length of music.  I currently use LAME V5 mp3s for portable use which, so far, has given me an average bitrate of 131kbps.  The average bitrate for this test was therefore 64kbps.

I didn't include LAME V5 in the test as I was already quite familiar with its quality; the average score would have been above 4.5.


Encoders
A quick flick through a catalogue showed that mp3-, AAC-, Vorbis-, WMA- and ATRAC-capable devices are all commonly available,
so this is what I tested.  I didn't do any pre-testing to find the best encoder for each type, just used what seemed to be popular or recommended after a brief search.

ATRAC3plus:  The latest version of Sonicstage (Sony's music manager and encoder) that's available for download is version 4, but I got version 3.4 with a walkman less than a month ago.  I decided to test 3.4, as it's you get when you part with money.

MP3:  this wasn't really a contender, just included to satisfy my own curiosity.  I think that Gabriel recently recommended V9 as a low anchor for another test, and that's what I used (LAME 3.98 alpha 6, V9 vbrnew).

Vorbis:  Lancer (22 July 2006)  -q 0 gave me 64kbps on the first try.

AAC:  Nero's command line encoder - 26 May 2006 (but downloaded sometime during July).  I used -q 0.24 to get just under 64kbps, but all the resulting files were AAC (SBR) according to Foobar.  I understand that these are not compatible with all players, so I also used -q 0.14 with LC profile enforced.

WMA:  I got Windows Media Audio 9.1 (WMA9STD) from the Microsoft website in August.  The quality-based VBR setting Q50 gave me an average bitrate of 71kbps, which I considered too high for a contender in a 64kbps test.  Instead, I used the average bitrate setting -a_mode 3 -a_setting 64_44_2.


Summary of the encoder settings:
ATRAC3plus: Sonicstage 3.4 64kbps
MP3: LAME 3.98a6 -V 9 --vbrnew
Vorbis: Lancer (22 July 2006) -q 0
AAC: NeroAAC (26 May 2006) -q 0.24
AAC LC: NeroAAC (26 May 2006) -q 0.14 -lc
WMA: WMA 9.1 Standard -a_mode 3 -a_setting 64_44_2

MP3, AAC and Vorbis were encoded using Foobar's converter.  The Vorbis files were then converted back to WAVs because I couldn't get them to play in ABC/hr for Java.

WMA was encoded using Windows Media Encoding Script, then back to WAV using Foobar.

ATRAC3plus was encoded using Sonicstage and then back to WAV also with Sonicstage.  I had to encode these one song at a time, otherwise I got a fraction of a second from the previous song at the start of the file.  This was to ensure that the test remained blind.


Songs
The songs were generally some form of guitar rock, with only a couple of exceptions.  The full list is below:

Eskimo Joe - Liar
Jebediah - La Di Da Da
Lash - Better Than You
Millencolin - Right About Now
MxPx - Tomorrow's Another Day
Nancy Vandal - Boy Troubles
Ninety Pound Wuss - It Seems So Far Away
NOFX - Dinosaurs Will Die
Pennywise - My Own Country
Placebo - The Bitter End
Refused - New Noise
Shihad - Stranger
Snapcase - Caboose
The Avengers - I Believe In Me
The Hard-Ons - First Cut Is The Weakest
The Hives - Declare Guerre Nucleaire
The Living End - Trapped
The Real McKenzies - Wild Cattieyote
The Saints - Ghost Ships
The Smashing Pumpkins - Mellon Collie

Resulting Bitrates
The average bitrates for each encoder were:

ATRAC: 64kbps
LAME: 68kbps
Lancer: 63kbps
NeroAAC LC: 62kbps
NeroAAC: 63kbps
WMA9: 64kbps


Analysis - ff123 analyses
On ff123's recommendation, I used the Tukey HSD analysis to determine the winners and losers.  The 10% and 5% analyses are below:

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 20
Critical significance:  0.10
Tukey's HSD:  0.521

Means:

NAAC    Lancer  ATRAC    NAAC-LC  WMA9    LAME   
  3.38    3.22    2.83    2.40    2.18    1.58 

-------------------------- Difference Matrix --------------------------

        Lancer  ATRAC    NAAC-LC  WMA9    LAME   
NAAC      0.165    0.550*  0.985*  1.200*  1.805*
Lancer              0.385    0.820*  1.035*  1.640*
ATRAC                        0.435    0.650*  1.255*
NAAC-LC                              0.215    0.820*
WMA9                                          0.605*
-----------------------------------------------------------------------

NAAC is better than ATRAC, NAAC-LC, WMA9, LAME
Lancer is better than NAAC-LC, WMA9, LAME
ATRAC is better than WMA9, LAME
NAAC-LC is better than LAME
WMA9 is better than LAME



FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 20
Critical significance:  0.05
Tukey's HSD:  0.576

Means:

NAAC    Lancer  ATRAC    NAAC-LC  WMA9    LAME   
  3.38    3.22    2.83    2.40    2.18    1.58 

-------------------------- Difference Matrix --------------------------

        Lancer  ATRAC    NAAC-LC  WMA9    LAME   
NAAC      0.165    0.550    0.985*  1.200*  1.805*
Lancer              0.385    0.820*  1.035*  1.640*
ATRAC                        0.435    0.650*  1.255*
NAAC-LC                              0.215    0.820*
WMA9                                          0.605*
-----------------------------------------------------------------------

NAAC is better than NAAC-LC, WMA9, LAME
Lancer is better than NAAC-LC, WMA9, LAME
ATRAC is better than WMA9, LAME
NAAC-LC is better than LAME
WMA9 is better than LAME


Analysis - graphs
I used this webpage to generate the usual graphs (95% confidence intervals and box plots):
http://www.physics.csbsju.edu/stats/anova.html

I don't have my own space to put these online, so I had to put them in the uploads section.
http://www.hydrogenaudio.org/forums/index....showtopic=48444

Conclusion
After all that, I guess there were no surprises overall.  There were some surprises if you look at the songs individually though:

1) I only gave two scores of 5, both of them to Nero with LC enforced.  For the same songs, Nero without LC enforced scored 4.7 and 2.2.

2) Lancer's highest score was 3.7, while even LAME scored one 4.0 (other maximum scores: ATRAC 4.7, Nero-LC 5.0, Nero 4.7, WMA9 4.5).

3) My favourite song of the bunch, New Noise, got the lowest average score.  I wonder if it is a real problem for the encoders, or if I just marked them more harshly than usual.

Thanks to everyone who has worked on the encoders, the abc/hr programme and who helped with the write-up.
Stephen

Personal 64kbps listening test; multi-format

Reply #1
I must leave now  , but this test really looks great!
Thank you very much for sharing this !

Personal 64kbps listening test; multi-format

Reply #2
I assume NAAC is HE-AAC then? I wish there are more device support for HE-AAC, outside of cellphones.
twitter.com/pika2000

Personal 64kbps listening test; multi-format

Reply #3
Thanks for the test and for the big efforts to make it online!

It would be great if this could be put somewhere online as well.

Personal 64kbps listening test; multi-format

Reply #4
Nice.  Does the Lancer vorbis include the latest aoTuV low bitrate improvements?

ff123

Personal 64kbps listening test; multi-format

Reply #5
2) Lancer's highest score was 3.7, while even LAME scored one 4.0 (other maximum scores: ATRAC 4.7, Nero-LC 5.0, Nero 4.7, WMA9 4.5).
Despite this, on all occasion Lancer is better than LAME, as per your test.

Personal 64kbps listening test; multi-format

Reply #6
Quick replies:

Pika2000:  You're right, NAAC (my code for Nero AAC) refers to the HE-AAC files.

ff123:  I'm not sure what Lancer is based on to be honest, but (from memory) I think it used AoTuV 4.51.  I tried looking up the Blacksword/Lancer webpage, only to find that my July 22 version is now six versions out of date...

Stephen

Personal 64kbps listening test; multi-format

Reply #7
it shouldn't really matter than much. AFAIK, aoTuV release 1 just contains 1 or more bugfixes and the lancer builds optimize the speed(as far as i'm aware of).

also, thank you for posting the results. very helpful to me right now


Personal 64kbps listening test; multi-format

Reply #9
Interesting test, thank you!

Personal 64kbps listening test; multi-format

Reply #10
Nice test, thank you!

Any chance to see the raw data?

By the looks of it, Vorbis (Lancer) is providing the most constant quality. I guess this is what halb27 is looking for.

 

Personal 64kbps listening test; multi-format

Reply #11
I've added an excel spreadsheet (zipped) with the results to the uploads thread, link in first post of this thread - see under the "Analysis - Graphs" section.