New Public Multiformat Listening Test (Jan 2014)

Topic: New Public Multiformat Listening Test (Jan 2014) (Read 83214 times) previous topic - next topic

0 Members and 2 Guests are viewing this topic.

New Public Multiformat Listening Test (Jan 2014)

Reply #50 – 2013-12-09 19:07:15

Quote from: IgorC on 2013-12-09 18:35:04

Well, a lot of suff is going on.

Kamedo2,
Until now one of the conditions of HA tests is "no less than 10 results per sample".
Please have a look through these 10 "full" contributors. http://listening-tests.hydrogenaudio.org/i...-a/results.html zip file.

Sadly some of them have got tired let's say after 10 samples and after that they have just rated the low anchor.

I said that "more than 10 results per sample is a bit overkill", but if many are 5.0, around 10 res/sample might be about right. I'll later try to dotplot it, rather than just average and CI95% errorbar, so that I can have a better grasp of the score distribution.

New Public Multiformat Listening Test (Jan 2014)

Reply #51 – 2013-12-09 19:19:55

Quote from: Kamedo2 on 2013-12-09 19:07:15

I said that "more than 10 results per sample is a bit overkill", but if many are 5.0, around 10 res/sample might be about right...

That's why a good low anchor with acceptable quality has to be included. So people won't just rate a low anchor and submit other results as unrated 5.0. I admit, our low anchor ffmpeg at 96 kbps was very bad.

faac at 96 kbps (CBR) should do better job.

New Public Multiformat Listening Test (Jan 2014)

Reply #52 – 2013-12-09 19:22:01

I don't think the Apple AAC encoders shine at 80k. They shine at 96k. At 80k, I'm almost sure Opus would beat the AAC-LC encoders.
And 96k is likely to be the bitrate many people would use. For smartphones with speed limitation(128kbps is common), broadcasting at 96k seems natural.
http://www.hydrogenaudio.org/forums/index....howtopic=102876

New Public Multiformat Listening Test (Jan 2014)

Reply #53 – 2013-12-09 19:23:10

Kamedo2,

If here we're are at least 10 listeners then we can rise a number of samples or even work at what Jean-Marc has proposed.

New Public Multiformat Listening Test (Jan 2014)

Reply #54 – 2013-12-09 19:54:02

Quote from: C.R.Helmrich on 2013-12-09 10:20:47

I totally agree. Is there no interest in lower bit-rates? 48 kbps perhaps?

Everything is discussible. If people will prefer 48 kbps or any other rate, so be it.

New Public Multiformat Listening Test (Jan 2014)

Reply #55 – 2013-12-09 20:25:11

Quote from: Kamedo2 on 2013-12-09 19:22:01

I don't think the Apple AAC encoders shine at 80k. They shine at 96k. At 80k, I'm almost sure Opus would beat the AAC-LC encoders.
And 96k is likely to be the bitrate many people would use. For smartphones with speed limitation(128kbps is common), broadcasting at 96k seems natural.
http://www.hydrogenaudio.org/forums/index....howtopic=102876

Damn it poor IgorC, I am afraid Kamedo2 is right, AAC shines at 96, I was too excited to see it against Opus at 80 that I didn't even think about the older test already performed and AAC really didn't change much since.

Sorry again, revert back and last, definitive choice:

AAC-LC (Apple/qaac) VBR 96 kbps

AAC-LC (Fraunhofer/fhgaacenc) VBR 96 kbps

AAC-LC (Fraunhofer/fdkaac) VBR 96 kbps

Opus (1.1) VBR 96 kbps

New Public Multiformat Listening Test (Jan 2014)

Reply #56 – 2013-12-09 20:27:19

I'm interested in Opus and fdk-AAC as they are new players with high popularity potential. And how they compete with Apple-AAC @96 or @80. So:
1. Opus
2. fdk-AAC
3. Apple-AAC

New Public Multiformat Listening Test (Jan 2014)

Reply #57 – 2013-12-09 20:45:15

Quote from: IgorC on 2013-12-08 19:13:11

I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we probably should go to CVBR ~100 kbps. Anyway both Apple TVBR and CVBR are great.

We'll see about that (bitrate for TVBR). But anyway TVBR is recommended mode and it is mostly used. So results for another algorithms will not be so useful and informative. This should be a decisive argument.

New Public Multiformat Listening Test (Jan 2014)

Reply #58 – 2013-12-09 21:01:13

eahm, Serge Smirnoff,

Will update later.

Has FhG fdk VBR 80 or 96 kbps mode?

eahm,

Quote from: eahm on 2013-12-09 20:25:11

...
AAC-LC (Fraunhofer/fhgaacenc) VBR 96 kbps
...

Can I ask You what do You expect from testing FhG again? fhgaacenc uses the same FhG Winamp encoder that we have tested in the last public listening test. It came 2ºd, right after Apple AAC. The result will be same.
Anyway it's your choice.

Quote from: Steve Forte Rio on 2013-12-09 20:45:15

Quote from: IgorC on 2013-12-08 19:13:11
I think we should probably keep in mind both TVBR and CVBR. Because if TVBR will end up with ~94 kbps and other codecs at ~96-100 kbps then we probably should go to CVBR ~100 kbps. Anyway both Apple TVBR and CVBR are great.

We'll see about that (bitrate for TVBR). But anyway TVBR is recommended mode and it is mostly used. So results for another algorithms will not be so useful and informative. This should be a decisive argument.

Agree

New Public Multiformat Listening Test (Jan 2014)

Reply #59 – 2013-12-09 21:15:42

Quote from: IgorC on 2013-12-09 21:01:13

It came 2ºd, right after Apple AAC. The result will be same.

First, Apple did not win.

Second, what makes you so sure the results will be identical? I can provide random data for the contenders and not have my results tossed so long as I don't do anything stupid in ranking the anchors with respect to the contenders. These tests are subjective, after all. Also, even if people who participated in both tests gave the same results (don't hold your breath on that) what about people who participated in one test, but not the other?

Third, I see no reason to dismiss this test which don't give the exact* same result:
http://www.hydrogenaudio.org/forums/index....howtopic=100525

(*) Maybe it did if you actually pay respect to the error bars for both tests (just between Apple and FhG, they are actually statistically tied over-all in both tests!).

New Public Multiformat Listening Test (Jan 2014)

Reply #60 – 2013-12-09 21:20:30

Quote from: Garf on 2013-12-09 12:36:00

If the Winamp encoder in whatever the latest Winamp release was is current, that's great.

Are there relevant differences between the libfdk_aac that you sold to Google and this encoder?

Yes, Winamp 5.666 has the latest AAC encoder quality-wise, no new quality tunings which are ready for release.

I'll let you know when quality is improved. Or just ask

The Winamp/Sonnox/... encoder has a completely different code-base than fdkaac and is a bit better tuned, especially for VBR.

Chris

New Public Multiformat Listening Test (Jan 2014)

Reply #61 – 2013-12-09 21:42:22

greynol, what on earth are you getting at? Are you just trying to be a devil's advocate or troll the process?

Apple absolutely did win. p=.002.

IgorC didn't claim the individual test results will be identical, he claimed the overall ordering will be the same. Though exact scores would vary if you ran that test a thousand times, since Apple won the test by a statistically significant margin, it is to be expected that it would place first in a large majority of repeated tests.

There are tons of good reasons to value the last HA listening test higher than SoundExpert's unsound methodology. That has been discussed more than plenty already.

The last test had five codecs incl. anchor and several of those who did the test said it was too much. Now, at a higher bitrate, people are now trying to throw in every AAC encoder and mode under the sun. You won't get enough worthwhile participation to get any useful information out of the test if you do that.

New Public Multiformat Listening Test (Jan 2014)

Reply #62 – 2013-12-09 21:46:19

Quote from: jensend on 2013-12-09 21:42:22

Apple absolutely did win. p=.002.

I see it now, CVBR, not TVBR, though a retest could easily go the other way.

Quote from: jensend on 2013-12-09 21:42:22

There are tons of good reasons to value the last HA listening test higher than SoundExpert's unsound methodology. That has been discussed more than plenty already.

I've done my share of criticizing SE, TYVM. You apparently aren't familiar with the details of test I linked as none of the comments I've seen about SE's unsound methodology apply.

New Public Multiformat Listening Test (Jan 2014)

Reply #63 – 2013-12-09 21:53:04

I have to say that I'm most interested in Opus, Vorbis (aotuv), and mp3 (lame) around 80-96 kbps.

New Public Multiformat Listening Test (Jan 2014)

Reply #64 – 2013-12-09 22:06:37

Greynol,

Let me give You one example.
There were 3 public tests. Roberto's Sebastian's and mine. All of them have shown that HE-AAC is better than Vorbis.

The same way if there will be a new public test (well done, with enough people, with correct methodic). Apple AAC will be always be on top of other AAC encoders.
There can be variations but an average score of Apple will be always on top.

New Public Multiformat Listening Test (Jan 2014)

Reply #65 – 2013-12-09 22:26:26

Quote from: IgorC on 2013-12-09 21:01:13

Can I ask You what do You expect from testing FhG again? fhgaacenc uses the same FhG Winamp encoder that we have tested in the last public listening test. It came 2ºd, right after Apple AAC. The result will be same.
Anyway it's your choice.

I remember Chris said he/they did some tuning in the low bit rates, even on the very last release (.16). Wonder if anything changed up to ~96 kbps.

New Public Multiformat Listening Test (Jan 2014)

Reply #66 – 2013-12-09 22:26:56

Quote from: IgorC on 2013-12-09 21:01:13

Has FhG fdk VBR 80 or 96 kbps mode?

fdkaac.exe -m 1 (libfdk-aac 3.4.12) gave me 94.1 for the whole Shpongle album

New Public Multiformat Listening Test (Jan 2014)

Reply #67 – 2013-12-09 22:36:40

Here is a document I made while ago about kbps and settings: http://pastebin.com/4HiD8juZ

New Public Multiformat Listening Test (Jan 2014)

Reply #68 – 2013-12-09 23:11:39

I'm interested in:
Opus --bitrate 96
Apple -V 36 (target is ~96)
LAME -V 5 (target is ~128)

New Public Multiformat Listening Test (Jan 2014)

Reply #69 – 2013-12-10 01:13:53

Quote from: greynol on 2013-12-09 22:27:00

The whole point of this is that FhG could beat Apple in a re-match, especially when it tied Apple in a perfectly valid test, personal attacks against me aside.

Well, I see You're not familiar enough well with the results http://listening-tests.hydrogenaudio.org/i...ous/results.zip

And that's why You can't imagine why somebody (me in this case) can be sure that "FhG can't beat Apple in a re-match". (always speaking of 96 kbps, LC-AAC, stereo, 44.1 kHz)

If You analyse the results enough close, You will see that there are two group of people. First, the bigger one, who have prefer Apple encoder over FhG with statistically important differences. And the second (a smaller one) who have very slightly but still prefer FhG encoder over Apple's but there was no statistically valid difference. Someone could argue about sample selection. Well, it was a very representative set of 20 (!) samples which were automatically randomly picked from different groups of samples.

And now You get the SE test as argument.
Accroding to HA public listening test 2011 there was no a single person who has prefer Nero over any other AAC encoder (excluding of course the low anchor). Not a single person.
But according to SE test, the average score for Nero was higher than Apple's. Sorry, this can't be right. No.
And I can even explain to You what has happened. Nero uses a lot of long frames that makes it quite good for tonal samples, SE is generally consist of these kind of samples and doesn't have any kind of other problematic samples.
That's why Nero does that good. While HA's balanced set of samples consists of different kind of samples, including a transients where Nero performs very bad.

P.S. Anyway I don't mind to include FhG. In fact in my opinion FhG has the best in class quality among HE-AAC encoders at 64 kbps. And at some point it will shine in some future public test at those rate.

New Public Multiformat Listening Test (Jan 2014)

Reply #70 – 2013-12-10 01:26:54

SE's test results didn't agree with yours so they must be wrong, the graphical representation of the overall results of your test doesn't do justice to the test data, and you feel there is no point in looking into it any further; I get it.

The next time I see data from two different tests that aren't in agreement, I'll just ask you to point me in the right direction. The scientific method of repeating an experiment to confirm the results be damned.

New Public Multiformat Listening Test (Jan 2014)

Reply #71 – 2013-12-10 01:28:42

Quote from: IgorC on 2013-12-10 01:13:53

P.S. Anyway I don't mind to include FhG.

I think that would be great.

New Public Multiformat Listening Test (Jan 2014)

Reply #72 – 2013-12-10 01:39:05

Ah yes, editing your posts after people have replied in an effort to cover your tracks. Classy move, that. "'Hey, wasn't Trotsky in that photo last time I saw it?' 'No, Comrade. Nothing to see here.'"

Quote from: greynol on 2013-12-09 22:27:00

You're dismissing the SE result which doesn't exactly agree. I guess I'll have to take you at your word as to why that is.

Rather than taking my word for it, you could actually read the thread you linked to, along with any number of other threads on the subject.

AFAIK the only thing done differently for that test as compared to most SE tests was to quit with the artifact amplification. Though that was by far the most obvious problem with SE's methods, there have been many other concerns. A few of those include only having five samples (which have been considered unrepresentative), fine-tuning the rate settings to try to achieve a target on those five samples rather than allowing VBR encoders to make intelligent rate decisions given their target, not following normal ABC/HR or other established test protocol, and some statistical methods concerns.

This isn't about people "having an axe to grind." When you leap to uninformed conclusions and criticize people in an uncivil fashion it's pretty absurd for you to turn around and be hypersensitive - "Oh my, somebody called me out on my bad behavior! how rude of them!"

Without bothering to actually read the test results to find out whether there was a discrepancy, and without bothering to ask about in a civil fashion, you leapt to telling the person who organized and did so much of the work on that test that he was lying about its results. You disparaged scientific statistical methods in an incoherent fashion. Rather than considering what people have said about SE, including on the page you linked, you leapt to telling me I'm just ignorant. And you didn't expect to get any pushback on any of this unless someone "had an axe to grind"? Is that because you expect to usually be able to just bin anybody who dares to disagree with you?

New Public Multiformat Listening Test (Jan 2014)

Reply #73 – 2013-12-10 02:50:29

I merely questioned (and am still questioning) a blanket claim. The differences between encoders simply isn't as great as all this posturing implies. Meanwhile we are all to dismiss another set of test results as completely worthless. I for one find the tight grouping of samples encoded with FhG as being worthwhile to at least a few regular and outspoken members. Perhaps this might be taken into consideration for this current round of testing.

I will compromise my request so that we don't have to worry about the possibility of redundant results:
QAAC 80 kbits TVBR
Whichever FhG AAC encoder that will likely get used by the greatest number of people in a post-Winamp world
Opus
Any other non-AAC format that stands a chance at this bitrate.

New Public Multiformat Listening Test (Jan 2014)

Reply #74 – 2013-12-10 03:15:59

Quote from: greynol on 2013-12-10 02:50:29

QAAC 80 kbits TVBR

I think this would be qaac -V 27.

Quote

Whichever FhG AAC encoder that will likely get used by the greatest number of people in a post-Winamp world

fdkaac?

Quote

Any other non-AAC format that stands a chance at this bitrate.

I'm not sure what else there would be. I remember vorbis being lower than nero here.