Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New Public Multiformat Listening Test (Jan 2014) (Read 147190 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New Public Multiformat Listening Test (Jan 2014)

Reply #200 –
Quote from: 2012 on
Enable AAC_HE profile with VBR 2
Enable AAC_HE_V2 profile with VBR 1

With ffmpeg, you can do this with -profile:a aac_he{,v2}. Or you can use the -t parameter in aac-enc.


It looks like without explicitly setting the profile all vbr modes produce AAC-LC streams. I encoded only two albums (Shpongle and Prodigy) with "-m 1" and got 94.1kbit/s and 95.1kbit/s (AAC-LC) respectively with cut off at 13.1kHz.
keeping audio clear together - soundexpert.org

New Public Multiformat Listening Test (Jan 2014)

Reply #201 –
lvqcl, dakbyte, gainless,
Which FhG encoder(s) do You prefer to see in test?

New Public Multiformat Listening Test (Jan 2014)

Reply #202 –
Quote from: Kohlrabi on
So it's acceptable for AAC, but not for Musepack? I realize that interest in Muepack is low, and that it likely will not perform that well. But at least arguments should be consistent.

1) I didn't say that 14 kHz is good for AAC encoding  For me, it's also too low.
2) From FFmpeg and AAC Encoding Guide: "But beware, it defaults to a low-pass filter of around 14kHz. If you want to preserve higher frequencies, use -cutoff 18000. Adjust the number to the upper frequency limit you prefer."
So, this guide recommends to tune the encoder and not to use default lowpass value.


Quote from: IgorC on
Which FhG encoder(s) do You prefer to see in test?

FDKAAC has some drawbacks: experimental VBR mode, too low lowpass... I prefer FhG encoder as in Winamp.
 

New Public Multiformat Listening Test (Jan 2014)

Reply #203 –
Just make "wide open" test with fullband FAAC (120-125 kbps) and results will astonish you. That's is 10 year old anchor. I hesitate what from it someone away off will come off.

New Public Multiformat Listening Test (Jan 2014)

Reply #204 –
I'd rather like to hear comment on libFDK VBR mode from Chris.

New Public Multiformat Listening Test (Jan 2014)

Reply #205 –
Quote from: IgorC on
lvqcl, dakbyte, gainless,
Which FhG encoder(s) do You prefer to see in test?

For FhG I prefer the Winamp version, for the test the more popular and better rated Apple encoder seems more sensible though. As there shall be only 4 codecs included, my vote is now:

Opus (1.1)
Vorbis (AoTuV)
Apple AAC
FAAC at 96 kb/s as low anchor

96 kb/s as overall bitrate.

New Public Multiformat Listening Test (Jan 2014)

Reply #206 –
Gainless,
It's 4 codecs + low anchor (s).

New Public Multiformat Listening Test (Jan 2014)

Reply #207 –
Quote from: IgorC on
Gainless,
It's 4 codecs + low anchor (s).

Nevermind then 

Opus (1.1)
Vorbis (AoTuV)
Apple AAC (CVBR)
+ Helix Mp3, 128 kb/s
FAAC at 96 kb/s (low anchor)

New Public Multiformat Listening Test (Jan 2014)

Reply #208 –
Quote from: CoRoNe on
Although my music is predominantly lossless, I just listened 2x 2hours to 20 of my favorite songs, which I encoded with Opus 1.1 to 80 and 96kbps. Eventhough it was just casual listening, there actually wasn't a moment I could tell I was listening to lossy music! I was thus really surprised by the transparency of 80kbps. This leads my to believe that testing 96kbps is overkill and will be very hard!

My votes therefor go to:
Opus 1.1 - 80kbps (--bitrate 80)
Aac (Apple) - 80kbps*
Vorbis (aoTuVb6.03) - 80kbps (-q1)
Mp3 (Lame 3.99.5) - 80kbps (-b 80)

*I never encode to aac, so I know nothing about settings and which version is better, but as the title of this thread is "Multiformat Listening Test" it would only be logical to put forward 1 aac contestant.


I agree. Had a great deal of difficulty with the last 96kbps test using good equipment with old ears though.

New Public Multiformat Listening Test (Jan 2014)

Reply #209 –
Quote from: kennedyb4 on
For my vote I would like to finally, if possible, end the debate of CVBR vs TVBR in AAC, preferrably Apple's encoder. The last test showed only a tendency for CVBR to be rated higher with no clear winner.

Maybe the bitrate should be pushed lower to do this.

It will be hard to answer this question.  According to this  personal test CVBR is on par with TVBR 

New Public Multiformat Listening Test (Jan 2014)

Reply #210 –
Quote from: Gainless on
Quote from: IgorC on
lvqcl, dakbyte, gainless,
Which FhG encoder(s) do You prefer to see in test?



I prefer the FhG Winamp encoder. It's good to have a "higher quality than FAAC" open source encoder (FDK) but at 96kbps VBR it uses a too low lowpass cutoff which is very noticable for me. The Winamp encoder is obviously better with this.

I wonder if we could include Opus @80kbps and Opus @96kbps in the test aswell?  I think testing AAC-LC @80kbps doesn't make much sense, but it would be really interesting if Opus @80kbps could beat AAC-LC encoders @96kbps or not. Maybe Apple HE-AAC CVBR @80kbps or FhG VBR 2 for the same reason?

New Public Multiformat Listening Test (Jan 2014)

Reply #211 –
Opus' outcome will be very interesting of course, but I can't see why expectations for Opus are that high. When I did some listening @96kbps a few days ago Opus and AAC performed great, with regular music as well as problem samples, with one exception: harp40_1 was encoded pretty badly, but only by Opus.
lame3995o -Q1.7 --lowpass 17

New Public Multiformat Listening Test (Jan 2014)

Reply #212 –
I think a discussions of possible results generally leads to premature prejudice. That doesn't help.  Let's not discuss  quality of competitors. We will see later.

After the end of test  everybody will be welcome to give their opinions and observations.

New Public Multiformat Listening Test (Jan 2014)

Reply #213 –
Bitrate distribution of the major encoders:

New Public Multiformat Listening Test (Jan 2014)

Reply #214 –
Quote from: darkbyte on
I wonder if we could include Opus @80kbps and Opus @96kbps in the test aswell?
  I think testing AAC-LC @80kbps doesn't make much sense, but it would be really interesting if Opus @80kbps could beat AAC-LC encoders @96kbps or not. Maybe Apple HE-AAC CVBR @80kbps or FhG VBR 2 for the same reason?

It's still not clear whether Opus@96k is any good comparing to AAC@96.
And now it's evident there is a need in a new AAC test. But it should be organized separately from this one as it's too much work to do.



Until now the list of the most voted codecs:
1. Apple AAC  - 22 votes
2. Opus - 22
3.  Vorbis - 11
4. MP3@128k - 9

(?)+ low anchor - FAAC 96 kbps CBR or ABR

Also I think it will be useful to include a low-middle anchor in addition to a low anchor. Thread..
Shortly, a quality of recollected results should be higher because now a listeners should rank low-middle anchor higher than low anchor. If some particular listener rank the low anchor higher than  it's an indicator that something is wrong.

MP3@96 kbps has received a high number of votes , 8.   
Two birds in one shot! MP3@96kbps as low-middle anchor + we test it as one additional codec at 96 kbps.  Fortunately it's actually easy to test MP3@96 kbps. So it can be a good idea to add this codec despite we have already high enough number of codecs.

What do You think?

New Public Multiformat Listening Test (Jan 2014)

Reply #215 –
Quote from: Kamedo2 on
Bitrate distribution of the major encoders:


What about LAME 3.99.5 -V 4.99 (lvqcl's  post) or halb27's 3.100m?

It's rather non-MP3@96 (real bitrate ~100 kbps) vs MP3@128 (real bitrate ~135 kbps)

Edit: fixed link

New Public Multiformat Listening Test (Jan 2014)

Reply #216 –
So if mp3@128kbps is to participate I'd welcome 3.100alpha2 because of its improved behavior for tonal issue (listen to Angels_Fall_First for instance).
Sure because of improved short block behavior I prefer my lame3100m variant over the original.
lame3995o -Q1.7 --lowpass 17

New Public Multiformat Listening Test (Jan 2014)

Reply #217 –

New Public Multiformat Listening Test (Jan 2014)

Reply #218 –
Quote from: halb27 on
So if mp3@128kbps is to participate I'd welcome 3.100alpha2 because of its improved behavior for tonal issue (listen to Angels_Fall_First for instance).
Sure because of improved short block behavior I prefer my lame3100m variant over the original.

Yes, we know that. But what about an average people saying that it was alpha and not final?

New Public Multiformat Listening Test (Jan 2014)

Reply #219 –
I understand that.
We should just collect votes, and I gave mine. To be precise: I vote for lame3100m -V5.
lame3995o -Q1.7 --lowpass 17

New Public Multiformat Listening Test (Jan 2014)

Reply #220 –
mp3 encoders, including the lame 3.100.a2 64bit and the ultra-fast Helix.

New Public Multiformat Listening Test (Jan 2014)

Reply #221 –
Quote from: halb27 on
We should just collect votes, and I gave mine. To be precise: I vote for lame3100m -V5.

Sure.
Aslo it will be interesting to hear what Robert will tell.

New Public Multiformat Listening Test (Jan 2014)

Reply #222 –
Quote from: darkbyte on
I prefer the FhG Winamp encoder. It's good to have a "higher quality than FAAC" open source encoder (FDK) but at 96kbps VBR it uses a too low lowpass cutoff which is very noticable for me. The Winamp encoder is obviously better with this.

I wonder if we could include Opus @80kbps and Opus @96kbps in the test aswell?  I think testing AAC-LC @80kbps doesn't make much sense, but it would be really interesting if Opus @80kbps could beat AAC-LC encoders @96kbps or not. Maybe Apple HE-AAC CVBR @80kbps or FhG VBR 2 for the same reason?

Updated.


Guys, check if your codec choice was submitted appropriately.
There is still one day (tommorow) to submit changes.

New Public Multiformat Listening Test (Jan 2014)

Reply #223 –
Quote from: Garf on
Quote from: Kamedo2 on
I bootstrapped the last 2011 public listening test of AAC encoders @ 96kbps (280 donated results, 20 samples) to plan this upcoming test.
The past data may not be precisely applicable to an another future test, but you may get a 'sense' of 'How much effort do we need to bring the error margin down?' or 'Which plan is likely to make better use of the precious donated time?'. Enjoy!


If I interpret this correctly, instead of using 20 samples and a bunch (~14) of listeners, we could've used 65 samples with 2 listeners and have gotten an as accurate result (though way less useful for the developers) with less than half the effort? That's pretty mind-blowing.

Imagine there is 1 listener and infinite number of samples. Will it be as good as 10-15 listeners and 20 samples?

I think the graph implies an ideal correlation between the results of different listeners. Well, it's not totally uncorrelated but it's not 100%.
An ideal correlation would imply that all listeners have exactly the same hardware (headphones ...), exactly the same hearing, exactly the same age. etc.. etc.. etc...  That's why the results of one individual can't be enough representative even on infinite number of samples.

That's why I've mention about an inter-listener correlation a few times.

New Public Multiformat Listening Test (Jan 2014)

Reply #224 –
Quote from: IgorC on
Imagine there is 1 listener and infinite number of samples.

In case you have missed something. The listener is re-picked from the 25 listeners(2011 AAC@96) in each sample.
So if there is 1 listener and 1000 number of samples, The average workload for each listener is 40 samples.