Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: iPod compatible formats listening evaluation (Read 46032 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

iPod compatible formats listening evaluation

[!--sizeo:10--][span style=\"font-size:12pt;line-height:100%\"][!--/sizeo--]IMPORTANT WARNINGS[/size]

• this test is unfinished
• this test is about “classical music” samples, nothing else
• choosen VBR settings are pertinent and fair with a given set of samples or musical genre (in fact, classical music) but are unlikely to work with different musical genres
• one person and only one contributed to this test. For collective listening tests, go on Roberto Amorim and Sebastian Mares websites


[!--sizeo:10--][span style=\"font-size:12pt;line-height:100%\"][!--/sizeo--]INTRODUCTION[/size]

My latest fad is to ask Santa Clauss one of those big iPod (160 GB) and then fit my whole musical collection and access to every CD of mine everywhere and without any computer help. My plan is to have everything at ~130 kbps which should be enough in most situation.
My last evaluation at this bitrate is two years old now. There wasn't any revolution in the meantime but all tested encoders are now outdated. That's why I decided to evaluate current encoders a second time and to restrict the test to iPod compatible formats: MP3 and AAC. Excellent formats such as Vorbis or WMAPro are for this sole reason NOT included in this 2007 evaluation.

(Un)fortunately new encoders were released or announced before the test completion (which needs almost 2 full weeks to be performed): a new LAME beta version, a new QuickTime AAC on Macintosh system. I wouldn't spend so many hours to test the 80 remaining samples with encoders that are not up-to-date anymore. I therefore cancelled the test. Some results are nevertheless very interesting - that's why I decided to post them on the forum.

[!--sizeo:10--][span style=\"font-size:12pt;line-height:100%\"][!--/sizeo--]TEST SETTINGS[/size]

As previous tests of mine, 150 classical music samples are supposed to feed my curiosity. I tested the half.
Hardware settings are: Terratec DMX6Fire 24/96 soundcard, Onkyo R-A5 Amplifier, Beyerdynamic DT-531 headphones. Testing rythm: ~10 samples per day.

[!--sizeo:10--][span style=\"font-size:12pt;line-height:100%\"][!--/sizeo--]CHOICE OF ENCODERS AND POOLS[/size]

I decided to not close the door to any modern encoders. That's why Fraunhofer MP3 iTunes MP3, Helix [Real] MP3, Coding Technologies [Winamp] AAC encoders are competing here. But all couldn't be seriously compared each others and accurately ranked. Not 150 times at least. A pre-selection was needed and that's why I started the test with smaller MP3 and AAC pools (25 samples in each).

---- In MP3 pool:
  Fraunhofer v.1.4 (encoder date: 2007.05.18 ; package from 2007.07.11): VBR -m3 -q1
  Helix MP3 v.5.1: VBR -V65
  iTunes v.7.4.3.1 MP3: VBR 128 highest
  LAME v.3.98 beta 5: VBR -V5



No real surprises here. LAME is better than any other competitors with a confidence >95% and will therefore compete with AAC in final test. iTunes and Fhg performances are similar and aren't globally bad; but both are more unstable (or less robust against encoding difficulties) than LAME. Helix is considerably worse and finishes last.


---- In AAC pool

I exempted iTunes from the selections due to its excellent past performances. It will be opposed in the global evaluation with the winner of this pool. I know two serious challengers: Coding Technologies and Nero Digital. The latter is represented in the pool by two releases: the last one and the previous one.  I'm very suspicious about the last Nero's encoder since I discovered problems with it. This test and the direct comparison with the previous Nero Digital release is a good occasion to confirm or infirm my suspicions.

  Coding Technologies[encoder built in Winamp 5.5 beta, bit-identical output to final version]: CBR 128
  Nero Digital v.1.0.7.0 (February 2007): VBR -q 0,45
  Nero Digital v.1.1.3.4 (Aug 2007): VBR -q 0,44 (higher bitrate and better performance than -q0.45…)



This test confirms the existence of a quality gap between Nero Digital 1.1.34.2 and the previous version. This gap is really huge for my taste (and with classical music I recall). Something bad happens with last version and I hope it will fixed. Coding Technologies AAC's performances are rather good, especially for a CBR encoder. Nero 1.0.7.0 wins and will be opposed to iTunes AAC and LAME.

The final test will therefore be: iTunes AAC + Nero Digital 1.0.7.0 + LAME 3.98b5 + 2 anchors



[!--sizeo:10--][span style=\"font-size:12pt;line-height:100%\"][!--/sizeo--]CHOICE OF ANCHORS[/size]

For once I decided to be more practical than “scientifical” in the choice of anchors and take some risks.

As low anchor[/color] I used 96 kbps AAC. It's a (small) risk because this setting might give similar results to MP3 at 130 kbps. It's unlikely but not impossible. I was also very curious to check how would perform ~100 kbps by itself, and when compared to MP3. If low anchor appear to be “good enough” why not using it to fill a digital jukebox? iTunes can't be used here because last version forces 32000 Hz sampling rate (too easily noticeable for my taste on most situation). Nero Digital is configurable and offers VBR 96 kbps encodings. For obvious reasons I avoided last Nero implementation.

As high anchor[/color] I used 160 kbps MP3 (LAME -V4). The risk is higher IMO and I wouldn't be really surprised if any AAC implementation would reach or surpass even with a penalty of 30 kbps. But if MP3@160 would outperfom AAC@130 I would seriously consider the most compatible and universal format as final encoding choice for my future chritmas present.
• Nero Digital 1.0.7.0 -lc -q0,25
• LAME 3.95b5 -V4


The following evaluation can be read as:
- Nero AAC vs iTunes AAC vs LAME MP3
- Nero AAC at 100 kbps vs Nero AAC at 130 kbps
- LAME -V5 vs LAME -V4
- 100 kbps AAC vs 130 kbps MP3
- 130 kbps AAC vs 160 kbps MP3




[!--sizeo:6--][span style=\"font-size:24pt;line-height:100%\"][!--/sizeo--]RESULTS[/size]

Note: only plots are posted here. As unfinished test I won't spend too much time to format all results individually as I did in previous tests (like here and here).


[!--sizeo:5--][span style=\"font-size:18pt;line-height:100%\"][!--/sizeo--]I. CLASSICAL: 5 electronic/artificial samples[/size]



5 samples aren't enough to get statistically pertinent conclusions. With confidence we can say that LAME at 160 kbps is better than Nero AAC at 100 kbps (what a surprise...). I can also add that none of current AAC implementation is able to give good anything better than average results on these critical samples. I regret aoTuV Vorbis performance which was able to match (and even surpass) LAME -V2 at 128 kbps (nominal) (see here to get 2 years old plot).


[!--sizeo:5--][span style=\"font-size:18pt;line-height:100%\"][!--/sizeo--]IIa. CLASSICAL: 30 orchestral & chamber samples on modern instruments[/size]



With this group of 30 samples we have what people would consider as “classical” music: chamber and orchestral various compositions on modern instruments.
And here the first big change since my last evaluation: while iTunes still performs well (average mark is in progress) it's now clearly and statistically inferior to Nero Digital. The latter offers similar results to LAME at 160 kbps and is the only AAC implementation that appear to be more efficient than MP3. I must also recall that Nero Digital -q0,45 doesn't necessary offers ~130 kbps with other musical contents.


[!--sizeo:5--][span style=\"font-size:18pt;line-height:100%\"][!--/sizeo--]IIa. CLASSICAL: 30 orchestral & chamber samples on period instruments[/size]



Early instruments are usually harder to encode properly. It's confirmed here. LAME performances are significantly worse here. It's strange: 2 years ago LAME 3.98 alpha 2 was more constant and offered similar results between modern and period instruments recording. At least with -V5... With higher settings LAME also showed stronger regression in the past (-V2 with alpha 2 and now V4 with beta 5). LAME is here statistically tied with low anchor.
iTunes AAC is very constant. Nero Digital is now statistically tied with iTunes and with high anchor but ends this group with the best mark. It confirms on 30 additionnal samples how great it is.



[!--sizeo:6--][span style=\"font-size:24pt;line-height:100%\"][!--/sizeo--]VERY SMALL CONCLUSION[/size]


I haven't tested the whole “vocal music” group (30 samples) and most samples of the “solo instruments” (60). But intermediate results (20 samples on 60) of the later group are confirming previous ones:

LOW ANCHOR: 3.1
iTUNES AAC: 4.1
LAME MP3: 3.2
Nero AAC: 4.4
HIGH ANCHOR: 4.0

In other words, Nero Digital 1.0.7.0 -q0.45 looks like a jewel: better than iTunes 7.4.3.1 at the same bitrate, as good as MP3 at 160 kbps. Now it would be more than interesting to compare it to the newest QuickTime version (only available on Mac OS 10.5 ATM) and also expecting from Nero Team to fix current issues on 1.1.34.2.
… and hope from Apple they will correct all iPod classic (good name) issues with a new firmware.



___
appendix

bitrate table (150 full tracks, 16 hours of music from 150 classical music CD):
http://audiotests.free.fr/tests/2007.11/iPodTEST_bitrate.ods

2 gif files illustrating worrying ringing artefacts (+ agressive lowpass/ATH value) with Nero 1.1.34.2:
E22 sample (0-22000 Hz)
V19 sample (10000-19000 Hz zoom)

iPod compatible formats listening evaluation

Reply #1
A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy . I have found that the latest verison of Nero to be poor compared to the Feb 2007 version a few months ago with some AAD mixed samples such Iron Man by Black Sabbath which produces a annoying metallic noise in the background.
"I never thought I'd see this much candy in one mission!"

iPod compatible formats listening evaluation

Reply #2
A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy

Thanks.
I guess it would be better to wait for Sebastian's MP3 test (more listeners, different samples, but probably lower VBR setting) before stigmatising Helix.
iTunes also surprised me. It reacted differently from Fhg's encoder but finally ended with the same overall quality. It's worth noting that I used -m3 as VBR setting for Fraunhofer (which seems to be closer to 160 kbps than 130 kbps with most popular musical material - cf. AlexB bitrate table on the thread dedicated to the upcoming listening test). Would -m4 be as good as iTunes encoding? Maybe iTunes MP3 will appear to be the second best MP3 encoder in Sebastian's test.

iPod compatible formats listening evaluation

Reply #3

A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy

Thanks.
I guess it would be better to wait for Sebastian's MP3 test (more listeners, different samples, but probably lower VBR setting) before stigmatising Helix.
iTunes also surprised me. It reacted differently from Fhg's encoder but finally ended with the same overall quality. It's worth noting that I used -m3 as VBR setting for Fraunhofer (which seems to be closer to 160 kbps than 130 kbps with most popular musical material - cf. AlexB bitrate table on the thread dedicated to the upcoming listening test). Would -m4 be as good as iTunes encoding? Maybe iTunes MP3 will appear to be the second best MP3 encoder in Sebastian's test.

It sure is better to wait for Sebastian's since some encoders are likely to perform better at different genres and samples, but am suprised that iTunes mp3 performed well since Apple would spend more time devloping their AAC encoder.
"I never thought I'd see this much candy in one mission!"

iPod compatible formats listening evaluation

Reply #4
Made me re-think the use of Nero AAC 1.1+ instead of using 1.0.7.0.  Think I will go back to using the older Feb 2007 release over Aug 2007 release.  Good test by the way, well done.

EDIT: Its use for Classical Music that is.
Zune 80, Tak -p4 audio library, Lossless=Choice

iPod compatible formats listening evaluation

Reply #5
I've merged all 85 scores and analysed it with friedman:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Tukey HSD analysis

Number of listeners: 85
Critical significance:  0.05
Tukey's HSD:  0.226

Means:

Nero    160kbps  iTunes  LAME    100kbps 
  4.33    4.24    4.00    3.47    3.17 

-------------------------- Difference Matrix --------------------------

        160kbps  iTunes  LAME    100kbps 
Nero      0.089    0.331*  0.856*  1.156*
160kbps            0.241*  0.767*  1.067*
iTunes                      0.526*  0.826*
LAME                                  0.300*
-----------------------------------------------------------------------

Nero is better than iTunes, LAME, 100kbps
160kbps is better than iTunes, LAME, 100kbps
iTunes is better than LAME, 100kbps
LAME is better than 100kbps


Graphical results looks like this:




It is safe (for me) to say that:
• Nero Digital (Feb 2007) is better on a wide selection of classical music than iTunes AAC and LAME MP3
• AAC@130 kbps (Nero only, classical music) may offer similar (and even slightly better) quality than top-MP3@160 kbps
• iTunes AAC is better than LAME -V5
• AAC@100 kbps is clearly lower quality than top-MP3@130 kbps.
• Nero@130 kbps > Nero@100 kbps
• MP3@160 > MP3@130 (trivial conclusion but would it appear on a collective listening tests involving several “non-critical” listeners?)

=> Nero Digital AAC (Feb 2007) is what I'll use first, though it's not perfect at this quality setting (-q0,45).

iPod compatible formats listening evaluation

Reply #6
great test guru! wish i hadn't deleted my copy of Nero Digital v.1.0.7.0


iPod compatible formats listening evaluation

Reply #8
It's more fun to get the upcoming new Apple's vbr aac encoder in this test in the near future. With all fixes of the regressions found maybe the next version of Nero digital would get much better, which is also worth waiting.

iPod compatible formats listening evaluation

Reply #9
Using Nero Digital v.1.0.7.0 (February 2007): -q 0.45  I get bit rates around 155k, noticeably higher than 130k.  Am I getting unusual results?

iPod compatible formats listening evaluation

Reply #10
I don't think I have enough material to build a valid bitrate for non-classical music; from what I've tested (some old heavy metal stuff) it's indeed closer to 160 kbps than 130 kbps.
You finding are probably more a rule than an exception. I mentioned it in the test: the choosen VBR settings are based on average bitrate obtained with classical music, which is often (but not always) bitrate-friendly compared to modern or popular (i.e louder, less dynamical) music.

iPod compatible formats listening evaluation

Reply #11
A small sample of classical tracks was generating values around 145k, but more testing brought the average closer to your range.  Rock tends to be higher.

Great testing

iPod compatible formats listening evaluation

Reply #12
Its a shame that Nero Digital v.1.0.7.0 has a track length bug that effects playes such as Quicktime, GStreamer FAAD and XMMS FAAD plugins which causes the player to cut off at the last few secends. Which has been fixed on Aug 2007 and also listed on their release notes.
"I never thought I'd see this much candy in one mission!"

iPod compatible formats listening evaluation

Reply #13
I stay with lossless and wait for an update because in some very silent parts (also classical here) the encoder chooses very low bitrates even at q6, resulting in kind of digital hiss artifacts.

iPod compatible formats listening evaluation

Reply #14
I just noticed it, in real listening conditions and not during a listening test. It occurs with quiet piano recordings with -q 0,45. Bitrate is low (I expect this from a VBR encoder), between 80 and 100 kbps; but quality is sometimes low too (I don't expect this from VBR which is supposed to maintain a constant quality) 
I reached the best size/quality ratio with LAME -V5: ~100 kbps with this material and close to transparency files (but not always transparent). The most reliable VBR LC-AAC encoder in this situation seems to be iTunes which isn't (unfortunately) the most efficient one.
I'm now mixing the encoders for my growing M4A library.

iPod compatible formats listening evaluation

Reply #15
Guruboolez, have you tried the new AAC encoder in iTunes 7.5 / QuickTime 7.3 yet? I've encoded about 20 CDs from various genres to do comparisons... and I'm finding that at the 128 VBR setting, I'm getting lower bitrate and lower quality files than I did with the same settings in iTunes 7.4 / QuickTime 7.2. I'm not entirely sure what the changes are... but this new encoder seems to be a step backward... especially when you consider that upping the settings to 160 VBR creates a significantly higher bitrate and larger sized file.

iPod compatible formats listening evaluation

Reply #16
No, I haven't try yet and I don't think I can test it before christmas.
But as I plan to use iTunes soon for encoding several discs (piano music for which Nero Digital lowers the bitrate as well as the sound quality; harpsichord discs too for which iTunes@160 sounds better than Nero@~170) it would probably wise for me to check for possible issues with the newer encoder, especially after your interesting report.

iPod compatible formats listening evaluation

Reply #17
There's actually four encoding mode with the latest QuickTime aac codec: true vbr, cbr, abr, constrained vbr. I'm not sure which mode is the default vbr mode for iTunes.

guruboolez, I suggest you checking all 3 vbr modes when you do the test.

iPod compatible formats listening evaluation

Reply #18
I will only use and therefore test what I can use in batch encoding. An encoding mode available in QuickTime only is pointless for CD ripping / batch encoding purpose - unless someone will provide a tool alla iTunes_encode to exploit it in third-party apps.
I would be disappointed if iTunes didn't include the most interesting (true VBR) encoding mode available in QuickTime.

iPod compatible formats listening evaluation

Reply #19
There's actually four encoding mode with the latest QuickTime aac codec: true vbr, cbr, abr, constrained vbr. I'm not sure which mode is the default vbr mode for iTunes.

guruboolez, I suggest you checking all 3 vbr modes when you do the test.


I will only use and therefore test what I can use in batch encoding. An encoding mode available in QuickTime only is pointless for CD ripping / batch encoding purpose - unless someone will provide a tool alla iTunes_encode to exploit it in third-party apps.
I would be disappointed if iTunes didn't include the most interesting (true VBR) encoding mode available in QuickTime.


I'm fairly certain that iTunes is and always has used the constrained VBR setting.

iPod compatible formats listening evaluation

Reply #20
Quote
Nero Digital 1.0.7.0 -lc -q0,25

It's low anchor settings. Doesn't  default -q 0.34/0.35 produce  96-100 kbit/s LC?

iPod compatible formats listening evaluation

Reply #21
I'm fairly certain that iTunes is and always has used the constrained VBR setting.
I share this. The new iTunes uses a constrained VBR setting too; it's confirmed here.

It's low anchor settings. Doesn't default -q 0.34/0.35 produce 96-100 kbit/s LC?
I recall that I'm not using -q,025 but -q0.25 -lc. The -lc or -he settings forces the use of a different bitrate scale.
I tried to reach ~96kbps with the regular scale (it was indeed with something near to -q0.35) and the encoder forces the use of HE-AAC - which is pointless on an iPod compatible format setting. That's why I had to find the adequate -q level with the forced -lc mode.

 

iPod compatible formats listening evaluation

Reply #22
Christmas is coming up, and so is that iPod of yours ;-)
Any progress on your interesting effort, Guruboolez?

iPod compatible formats listening evaluation

Reply #23
is the test finished now ?

iPod compatible formats listening evaluation

Reply #24
Here ya go!
http://www.brothersoft.com/mp3_audio/rippe...udio_59677.html

Has only the newest version now, here's a working link for the older version:

http://download.chip.eu/de/Nero-AAC-Encode...7.0_155834.html

Thanks for your test, guru!
ZZee ya, Hans-Jürgen
BLUEZZ BASTARDZZ - "That lil' ol' ZZ Top cover band from Hamburg..."
INDIGO ROCKS - "Down home rockin' blues. Tasty as strudel."