Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat Listening Test @ 128 kbps - FINISHED (Read 165657 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #100
Quote
Quote
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
[a href="index.php?act=findpost&pid=357292"][{POST_SNAPBACK}][/a]

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
[a href="index.php?act=findpost&pid=357303"][{POST_SNAPBACK}][/a]

Well.. it's hard to say what effect the bug has exactly.
The fact is after about half of the track positions because the bitreservour isn't flexible enough, the quality may be worse than it could be.
People base rating on what they hear differs from original, so it can be just as well speculated that the overall score could be better in case of properly working reservour.

If it's speculated that people only listen the beginning, and the overcoding makes a huge quality improvement (which it shouldn't do, because we are near transparency level) then imo Nero score is overrated. But, the samples have the bug which causes both over- and undercoding (reservour induced), and about 50% of the track lengths are undercoded. And again, people rate based on the problems they hear.. Well, this is just my opinion, others may come to different conclusion.
Juha Laaksonheimo

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #101
Quote
Quote
That's why I wouldn't say that "AAC is the best tradeoff quality/compability". It can't be seriously defended by objective arguments.
[a href="index.php?act=findpost&pid=357364"][{POST_SNAPBACK}][/a]


Versus OGG Vorbis it definitely can.
[a href="index.php?act=findpost&pid=357365"][{POST_SNAPBACK}][/a]

Well, I give up


Multiformat Listening Test @ 128 kbps - FINISHED

Reply #103
One remark regarding the bit reservoir drain - IMO it could just make the quality worse, because bit reservoid is always drained, and not according to the psychoacoustic demands.

If you overcode something which is already "near transparent" - effects on this are very very subtle - if you undercode something which needs more bits - you end up with obvious artifact.  As it is well known from the psychoacoustic theory, average bit demand of the normal music signal is just about 128 kbps, maybe a bit more - so, even without big bit reservoir effects would be small - with overcoding, not too much is improved.

However, this causes frames that need more bits (200+ kbps) to actually receive much less bits than they should - therefore making encoding quite suboptimal.

But, I would rather skip discussion about this, we had a pretty long conversation (Sebastian, Guru, Juha, Roberto and myself) - and for sure I'd like to avoid repeating it as there are a lot of tasks I anyway need to do and this would take so much time.

What I think is way better - is actually testing the bugfixed mode - this encoder will appear soon (during next two weeks I hope) - as well as with bugfixed 128 kbps mode.  @Guru, I am sure you would be able to test it and give some remarks providing that you find some time.

Once again - I am very sorry (and I am personally dissapointed as well because the true quality of the new AAC encoder couldn't be tested properly) the bug was found too late - we will make sure and invest all our resources for such thing not happening again.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #104
Quote
I could send you all results if you really want to do it yourself.
[a href="index.php?act=findpost&pid=357306"][{POST_SNAPBACK}][/a]


Are they in an easy to process form? (Like an Excel sheet... or something similar)

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #105
Quote
Quote
I could send you all results if you really want to do it yourself.
[a href="index.php?act=findpost&pid=357306"][{POST_SNAPBACK}][/a]


Are they in an easy to process form? (Like an Excel sheet... or something similar)
[a href="index.php?act=findpost&pid=357373"][{POST_SNAPBACK}][/a]


No, as encrypted ABC/HR results.  That's why I said it would take too much time. I only have the valid results sorted and as TXT files.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #106
Interesting results. Somehow it proofs LAME got even better (there's really no need for version 3.90.3 anymore), the new vbr model is stable and for most people 128kkbps MP3 is transparent. The magical number until now was 192kbps. I think the truth is somewhere between them. I'll stick with LAME -V 3 --vbr-new. It's about ~150-160kbps.

Thanks to all people who participated in this listening test!

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #107
Bond: M4A is the same as Mp4.
I know Bond. But i want Mr.Q to see my encoded files as mp4 not m4a.

Off topic: Sebastian, when i saw your name, i tought that this look like a romanian name.
 
I'm from Romania too.
A good day.(night)

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #108
iam still not getting the ipod, mkay!?
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #109
I really like the idea of having the settings used included in the graph. As you said, people will post those graphs all over the place and therefore it is better to state what settings achieved those results. It helps educating people and will hopefully stop them from using those strange command lines (especially true for lame).

Many people will not follow the link (they have to type the address by hand). And even if they do, the settings are hard to find and there is a lot of reading. People that use those strange command lines will simply upgrade to the encoder tested and keep using their crappy command lines and wonder why it sounds like crap...many might not even know that there is more than one setting one can use.

Please consider it.
Quote

[a href="index.php?act=findpost&pid=357175"][{POST_SNAPBACK}][/a]
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #110
Sorry, no.
You can do it yourself and state that it's your plot - I have no problem with that. I am not responsible for the way people post that image and what they do and don't do. I doubt that image is going to appear on a forum out of the sudden and without any comments - people who post the plot can also create a link below or above. The encoder version is more than enough in the graphics.
Discussion about this is over.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #111
Quote
Sorry, no.
You can do it yourself and state that it's your plot - I have no problem with that. I am not responsible for the way people post that image and what they do and don't do. I doubt that image is going to appear on a forum out of the sudden and without any comments - people who post the plot can also create a link below or above. The encoder version is more than enough in the graphics.
Discussion about this is over.
[a href="index.php?act=findpost&pid=357432"][{POST_SNAPBACK}][/a]

ok then, can you post the xls/openoffice/whatever you used?

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #112
Very big congratulations to Sebastian for managing to produce interesting results after all the hardships he had to go through to discuss his test and later conduct it.

And here's to your upcoming tests! May the next test (64kbps?) be as successfull (but less stressful!) than this one. I know for sure I can hardly wait for it...

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #113
Quote
Quote
Quote
That's why I wouldn't say that "AAC is the best tradeoff quality/compability". It can't be seriously defended by objective arguments.
[{POST_SNAPBACK}][/a]


Versus OGG Vorbis it definitely can.
[a href="index.php?act=findpost&pid=357365"][{POST_SNAPBACK}][/a]

Well, I give up
[a href="index.php?act=findpost&pid=357368"][{POST_SNAPBACK}][/a]

I was about to say that on the Mac side of the fence, it is not so, and that there is bad support for it. However, I decided to check if things still are the way they used to be. They aren't.

There's a [a href="http://www.xiph.org/quicktime/]Xiph component[/url] available that allows you to play Ogg Vorbis files in QuickTime (and thus iTunes). Thanks to QuickTime 7, it even supports tagging, and in iTunes too. So Ogg Vorbis is a viable alternative, if you don't mind the lack of iPod-compatibility.

Ogg Vorbis reminds me of Apple in the mid-nineties: You really don't expect it to survive, but it does; quite well, even.

(By the way, this test is a very good argument against the people claiming that the 128kbps AAC used in the iTunes Music Store isn't good enough quality. It's very good quality, and probably more than good enough for the vast majority of listeners.)

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #114
How I can decrypt the results with the key.key file?

In hush I forgot to send my results of the sample 12.... I tested it and even found diferences between the samples.

Also, I forgot to add my name to most of my tests I think. I'm the anon26.

If you wanted something about what coment, you could have talked about shine's performace. It was not bad in the "song for guy" and few others.

I rated vorbis as the worst (4.0, but still) between the encoders that I could hear the diference in the Senor sample (excluding low-anchor). The only two other people that rated vorbis put vorbis in first or tied with the first ones. Not even guruboolez rated vorbis in this sample!?!? 
I think everyone hearing is diferent....

Maybe an equipament diference? I have an ECS's on-board sound card and some crap pc-speakers that come with the computer. I can't even plug an headphone here because the "eletric-zoom". I think it is the average home user equipament, but very far from what HA's people probably have...

I'm not an native english speaker nor I read any guide for artfacts in audio, so I don't know nothing about the nomeclature for audio artfacts, but I think it can be the pre-echo thing plus a bost in the source of the echo. This occur around 15.0s point of the sample. My coment at the time:
"3R Comment: make some drums more harsh/difuse"

This was and is clear for me. And I even thought about vorbis that time, because that harsh sounding. I think that I don't like the way that vorbis sond. Many times I prefer an 32kbps AACv2 than an 64kbps vorbis aoTuV 4 (I don't tested the 4.5b yet).

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #115
Quote
(By the way, this test is a very good argument against the people claiming that the 128kbps AAC used in the iTunes Music Store isn't good enough quality. It's very good quality, and probably more than good enough for the vast majority of listeners.)[a href="index.php?act=findpost&pid=357454"][{POST_SNAPBACK}][/a]

This test tested iTunes AAC in the new VBR mode. I think iTunes Music Store still offers standard AAC "CBR" files. So that argument is not valid. Or have they changed the format recently?

Actually, even the standard iTunes AAC is not strictly CBR. The bitrate varies, but not as much as in VBR mode and the average is always the 128 kbps. iTunes 128 kbps VBR allows more bitrate variation and higher average bitrates when needed. It does not allow a lower average than 128 kbps. So it is most likely usually better than the older standard mode or at least as good if the files are not difficult to encode, but that has not been directly tested.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #116
Thanks to everyone with ears who participated and Sebastian for organising the whole listening test.  The results are quite interesting and I'm always pleased to see Ogg Vorbis at the front.  It is particularly interesting to see the scores so consistently high, which makes me wonder whether the samples used were perhaps too easy (ie. there wasn't a codec-killer like 'Waiting')?

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #117
Vorbis did not finish 'at the front'.  The difference between AoTuV and iTunes AAC is within the margin of error and is not significant.  Moreover, the slightly higher score needs to be weighed against the fact the AoTuV used a slightly higher bitrate than iTunes.

The fact that all that all the codecs scored highly is unsurprising given the high bitrate.

There are three conclusions that can be drawn from this test:

1) AoTuV, iTunes AAC, and WMA Pro produce encoded files of roughly equivalent quality at bitrates near 140kbps.

2) Lame was very close to the others on most samples, but there were two samples where lame scored significantly lower.

3) With all of these encoders, some encoded files can be distinguished from the original.  The samples were not "too easy", as there were some samples which none of the codecs could encode transparently at the settings used.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #118
If my memory serves me, Ivan promised to publish results of PEAQ analyses of encoded sound samples from the listening test. May we hope to see them?
keeping audio clear together - soundexpert.org

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #119
The scores are interesting and it reflects an increase in quality over time. But is it so hard to find problems ?  I have many problem samples from my collection - at least for mp3 and its on 'normal' music.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #120
Quote
Vorbis did not finish 'at the front'.  The difference between AoTuV and iTunes AAC is within the margin of error and is not significant.  Moreover, the slightly higher score needs to be weighed against the fact the AoTuV used a slightly higher bitrate than iTunes.
[{POST_SNAPBACK}][/a]


From the pre-test discussions:
Quote
Quote
Quote
Updated bitrate table with Vorbis -q 4.25: [a href="http://maresweb.de/bitrates2.htm]http://maresweb.de/bitrates2.htm[/url]

I think that it would be more fair to use a lower quality setting for Vorbis.

Since the bit rate control of Vorbis is very flexible it seems wrong to select a setting that will give it the highest average bitrate of all encoders. People could challenge such a decision as favoritism towards Vorbis.

At -q 4 Vorbis has an average bitrate similar to that of iTunes and Nero and that would in my opinion be the right setting.

[/url]



According to this post:

Quote
I updated my bitrate table with Vorbis -q 4.20 and gathered all previous results in the same table:

bitrates_public2.xls


Vorbis -q 4 reached an average of exactly 128 kbps on Alex B's side. On the other hand, -q 4.2 and -q 4.25 didn't produce too high bitrates (around 134 kbps which is OK). So, what do the experts thing - should -q 4 be used or -q 4.2(5)?...
[a href="index.php?act=findpost&pid=347010"][{POST_SNAPBACK}][/a]

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #121
Quote
I really like AAC, as well as Vorbis, but I must say that I'm disappointed by the industry attitude and lack of interest for AAC.


Note that AAC has license fees:
http://www.vialicensing.com/products/mpeg4...ense.terms.html

Vorbis is license free and there are also open source MIT licensed floating point encoder end decoder, and a fixed point decoder. I also prefer Vorbis bacause is the only lossy format we linux users can use without infringing any patent and using only free software. We should all thank xiph.org and Aoyumi for their free work in creating a great patent-free codec.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #122
Quote
Thanks to everyone with ears who participated and Sebastian for organising the whole listening test.  The results are quite interesting and I'm always pleased to see Ogg Vorbis at the front.   It is particularly interesting to see the scores so consistently high, which makes me wonder whether the samples used were perhaps too easy (ie. there wasn't a codec-killer like 'Waiting')?
[a href="index.php?act=findpost&pid=357479"][{POST_SNAPBACK}][/a]

I think the tested samples are average or above average in degree of difficulty.

I have though about so called killer samples too. However, I think it would be very difficult to gather a good killer sample collection that would be fair for different codecs in a multi-format test. Usually these killers are especially bad for one particular codec.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #123
Quote
Vorbis did not finish 'at the front'.  The difference between AoTuV and iTunes AAC is within the margin of error and is not significant.  Moreover, the slightly higher score needs to be weighed against the fact the AoTuV used a slightly higher bitrate than iTunes.

The fact that all that all the codecs scored highly is unsurprising given the high bitrate.[a href="index.php?act=findpost&pid=357498"][{POST_SNAPBACK}][/a]

As sehested quoted all codecs produced similar bitrates in preliminary testing with a large amount of various files. Besides my tests guruboolez tested the bitrates with a big collection of classical music. Some others posted their bitrate findings too.

The used Nero encoder was changed after I added it to the bitrate table, and the tested version produced also about 134 kbps with my test files and in Sebastian's personal testing. (I'll update the table now even Nero was disqualified from the test.)

Each codec has different methods for keeping the quality level constant. Usage of high momentary bitrates is a completely acceptable method. I suppose it would have been good to include some samples that would have produced very small average bitrates, but because iTunes has a hard-coded 128 kbps low limit and Nero was used in ABR mode I think that would not been fair for the three other encoders. Also, that kind of samples tend to be even easier for the encoders, so possibly the quality differences would have been indistinguishable.

Multiformat Listening Test @ 128 kbps - FINISHED

Reply #124
Quote
I suppose it would have been good to include some samples that would have produced very small average bitrates, but because iTunes has a hard-coded 128 kbps low limit and Nero was used in ABR mode I think that would not been fair for the three other encoders.
[a href="index.php?act=findpost&pid=357525"][{POST_SNAPBACK}][/a]

I wouldn't say that's unfair. "Low bitrate" moments are a true part of real-world encoding, and it wouldn't be unfair to test them. A complete test should in reality include such samples. But with 18 samples only, it's impossible to represent each encoding situation.
iTunes (bitrate floor at 128 kbps) and Nero Digital (ABR) both have a limited efficiency. As example, if you encode monophonic albums or albums including mono tracks, or low volume musical compositions, both AAC encoders will systematically waste a big amount of bitrate. I've recently tested it by encoding some Jazz oldies: LAME's bitrate (-V5 --vbr...athaa...) was around 85 kbps whereas iTunes was 128 and Nero Digital 130. Same for a very recent complete set of Beethoven's sonatas, recorded in the last years in stereo: ~100 kbps for lame and ~130 for AAC. The limited efficiency is the reverse side of the developer's choice.
But on the other side, such limitation may have a positive effect on quality. Not for mono, but for low volume moments. LAME tend to produce ringing, which become easier to hear with a higher volume playback and sometimes really irritating after ReplayGain/MP3gain. That's worrying and I appreciate the limitation of both iTunes and Nero which are a warranty against psycho-acoustic failure or optimstic choice.

Now what matter is how would react these encoders with low bitrate (corresponding usually to low volume tracks/moments). I performed recently a listening test with 150 classical music tracks including several samples corresponding to this situation. My results are available on the forum. From my experience, Vorbis has no problem to handle this situation; the bitrate doesn't sink to a ultra-low value (rarely less than 100 kbps) and there's no compromise in high frequencies (no ringing). Unfortunately, I can't say the same for LAME. It has real problems here, and low bitrate may correspond to low quality. Hence the usage of --athaa-sensitivity to lower this annoyance (which highers the bitrate on such situation).

Quote
Also, that kind of samples tend to be even easier for the encoders, so possibly the quality differences would have been indistinguishable.

Not necessary true. The Debussy.wav sample revealed in 2004 that such samples could lead to obvious artefacts for some encoders (LAME, Musepack) but not others