Skip to main content
Topic: New Public Multiformat Listening Test (Jan 2014) (Read 84912 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New Public Multiformat Listening Test (Jan 2014)

Reply #225
MP3@96 kbps has received a high number of votes , 8.   
Two birds in one shot! MP3@96kbps as low-middle anchor + we test it as one additional codec at 96 kbps.  Fortunately it's actually easy to test MP3@96 kbps. So it can be a good idea to add this codec despite we have already high enough number of codecs.

What do You think?


I like this idea.  So I vote for Musepack @ 96kbps with its ~14kHz lowpass as low anchor and MP3@96kbps for the low-middle anchor.


New Public Multiformat Listening Test (Jan 2014)

Reply #226
I like this idea.  So I vote for Musepack @ 96kbps with its ~14kHz lowpass as low anchor and MP3@96kbps for the low-middle anchor.

Seems reasonable as  MPC isn't that good according to this personal test at  96 kbps

Imagine there is 1 listener and infinite number of samples.

In case you have missed something. The listener is re-picked from the 25 listeners(2011 AAC@96) in each sample.
So if there is 1 listener and 1000 number of samples, The average workload for each listener is 40 samples.

ah, ok. It's in context of  this particular whole set of data.

New Public Multiformat Listening Test (Jan 2014)

Reply #227
Most of people have prefered to test at 96 kbps.

The list  of codecs to test.
1.   MP3@128 kbps.
2.   Apple AAC
3.   Opus 1.1
4.   Vorbis  aoTuV
+middle-low anchor  FAAC 96 kbps *1
+low anchor (?) should be selected, discussable. A low anchor should have lower quality than FAAC 96 kbps.

and first approximation of the settings for ~96-100 kbps.
1. LAME 3.99.5 -V 5 or -V 4.99, halb27 LAME extension 3.100m, Helix?
2. QAAC, highest quality  (CVBR 96 or TVBR 45?)
3.  Opus  --bitrate 96
4.  Vorbis aoTuV -q 2 ... -q 2.5 (?)

The target bitrate is  ~96-100 kbps for Opus, Apple AAC, Vorbis. And MP3 ~130-135 kbps.


Agenda.
A choice of codecs.  December 8 – December 18. DONE.
Bitrate verification, a choice of settings – December 19 – December 23-25.
Sample selection  - December 25-26 – January 5.
Checking all conditions, preparations, dummy packages  - January  6 – January 10

*1  Probably MP3  at 96 k is too good to be a middle-low anchor.  MP3  96 k is actually close to Nero AAC  which still was pretty hard to spot for some listener in previous test.

It's time to verify bitrate, choose settings and an encoder for MP3.  Bitrate verification table.
Please submit the bitrates You get with the encoders. Example.

New Public Multiformat Listening Test (Jan 2014)

Reply #228
The target bitrate is  ~96-100 kbps for Opus, Apple AAC, Vorbis. And MP3 ~130-135 kbps.

I'd like to see the result of MP3 128k, not 135k. MP3 135k might be too hard for the beginner.

New Public Multiformat Listening Test (Jan 2014)

Reply #229
Quote
+low anchor (?) should be selected, discussable. A low anchor should have lower quality than FAAC 96 kbps.

Maybe VisualOn AAC @ 192Kbps?


New Public Multiformat Listening Test (Jan 2014)

Reply #231
Also I've noticed that not all codecs report bitrate equally. There can be differences ~2 kbps or so.

So if somebody will inform a bitrates it also will be useful to get a filesize or a real bitrate (total size/ total duration).

New Public Multiformat Listening Test (Jan 2014)

Reply #232
These are the rules from previous public test.
There're some suggestions to improve/change them.

Code: [Select]
Participants who don't want to worry too much about the grading rules
can simply ignore them. Listeners should do their best to rank the samples and
be careful to identify the hidden references. Listeners should ABX tests
they are at all unsure.

1) If the low anchor is not graded, or if any hidden reference is graded
below 4.5-5 (see App.) the result is INVALID.

2) For each sample with a ranked reference or an ungraded low anchor the
listener will have a single chance to submit a replacement test run for
that sample. The replacement test must cover all codecs, not just the
codecs with the ranked reference. (This also covers cases where the
reference is ranked but still at or above 4.5)

3) If a listener submits 2/10 (3 for 20 samples submitted) or more INVALID
results then only ABX results will be accepted, or the listener will be excluded
completely in cases of apparently abusive behavior.

App. These rules aren't extremely strict in order to allow for simple human
error while still excluding careless participants.

A stricter procedure to exclude all ranked references risks a systemic
bias against any codec which are very good on a few samples and thus
subject to more reference confusion by causing those samples to be excluded
and weighing the test towards other samples.


New Public Multiformat Listening Test (Jan 2014)

Reply #233
Average Bitrate of 122 songs, Speed in x realtime, command line.
Code: [Select]
91650 43.90 qaac --tvbr 45 -q 2 -o %o %i
99456 43.78 qaac --cvbr 96 -q 2 -o %o %i
95985 50.15 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 96k %o
97517 51.01 lame3.99.5 -V7 %i %o

New Public Multiformat Listening Test (Jan 2014)

Reply #234
Average Bitrate, Speed in x realtime, command line
AAC 96kbps:
91650 43.9 qaac --tvbr 45 -q 2 -o %o %i
99456 43.7 qaac --cvbr 96 -q 2 -o %o %i
95985 50.1 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 96k %o
92048 47.9 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 92k %o

MP3, Opus, Ogg Vorbis 96kbps:
97517 51.0 lame3.99.5 -V7 %i %o
95734 48.1 lame3.99.5 -V7.2 %i %o
92603 44.2 opus-1.1-rc-msvc2013\opusenc --bitrate 90 %i %o
98686 46.0 opus-1.1-rc-msvc2013\opusenc --bitrate 96 %i %o
86710 27.9 venc603(aoTuV) -q1.7 %i %o
95359 29.2 venc603(aoTuV) -q2 %i %o

MP3 128kbps:
124126 49.0 lame3.99.5 -V5 %i %o
130941 48.8 lame3.99.5 -V4.99 %i %o
128701 141.0 hmp3(Helix) %i %o -X2 -U2 -V60

New Public Multiformat Listening Test (Jan 2014)

Reply #235
Average Bitrate of 122 songs, Speed in x realtime (on i7 2.93GHz), command line
AAC 96kbps:
91650 43.9 qaac --tvbr 45 -q 2 -o %o %i
82546 42.9 qaac --cvbr 80 -q 2 -o %o %i
99456 43.7 qaac --cvbr 96 -q 2 -o %o %i
95985 50.1 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 96k %o
92048 47.9 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 92k %o

MP3, Opus, Ogg Vorbis 96kbps:
97517 51.0 lame3.99.5 -V7 %i %o
95734 48.1 lame3.99.5 -V7.2 %i %o
90787 49.4 lame3.99.5 -V7.5 %i %o
91595 44.7 opus-1.1-rc-msvc2013\opusenc --bitrate 89 %i %o
92603 44.2 opus-1.1-rc-msvc2013\opusenc --bitrate 90 %i %o
98686 46.0 opus-1.1-rc-msvc2013\opusenc --bitrate 96 %i %o
86710 27.9 venc603(aoTuV) -q1.7 %i %o
87603 27.3 venc603(aoTuV) -q1.8 %i %o
88547 27.1 venc603(aoTuV) -q1.9 %i %o
95359 29.2 venc603(aoTuV) -q2 %i %o

MP3 128kbps:
124126 49.0 lame3.99.5 -V5 %i %o
130941 48.8 lame3.99.5 -V4.99 %i %o
128701 141.0 hmp3(Helix) %i %o -X2 -U2 -V60

New Public Multiformat Listening Test (Jan 2014)

Reply #236
I think that tuning  q-parameters of encoders for a listening test using some arbitrary  set of tracks is not the best way of doing this. As I already  mentioned, using sets of tracks with different proportion of genres  will result in different target bitrates and corresponding codec  settings. It is just another source of variance for test results.  Different sets of tracks will favor different contenders (hopefully  not easy to understand whom exactly but who knows …).

My  suggestion is to tune q-parameters using test samples selected for  the test. This set of samples is the only unambiguous set and  it relates directly to the test. For example, Lame  V7 could be an anchor for this test. Using selected sound samples it  will produce some target bitrate (most likely not 96, doesn't  matter). Then all other codecs will be tuned to get close bitrates on  the same samples. This can be done either by averaging per-sample  bitrates or by concatenating all selected samples into one stream and  calculating target bitrate for it (much  easier). So all encoders will use equal number of bits for the stream but will  distribute them between samples differently, according to their psy.models. This  way the variance of codec settings due to arbitrary chosen audio  material will be completely eliminated. Bitrate verification of  resulting settings will still be necessary but just for reference purposes – to better understand what final bitrates could be  achieved with these settings with different genres. In previous HA  listening tests this was done vice versa – settings were tuned using arbitrary sound material and bitrates for selected test samples were given for reference.
keeping audio clear together - soundexpert.org

New Public Multiformat Listening Test (Jan 2014)

Reply #237
I prefer using a test set of hopefully representative music for deciding upon which settings to use.
And from the various results given here it looks like Apple --cvbr 96, Opus --bitrate 96, aoTuv -q2 give these contenders equal chances (maybe with a tiny bit stronger aoTuv setting). These are 'natural' choices moreover.
lame3995o -Q1.7
opus --bitrate 140

New Public Multiformat Listening Test (Jan 2014)

Reply #238
Average Bitrate of 122 songs, Speed in x realtime (on i7 2.93GHz), command line
AAC 96kbps:
91650 43.9 qaac --tvbr 45 -q 2 -o %o %i
106035 39.2 qaac --tvbr 54 -q 2 -o %o %i
82546 42.9 qaac --cvbr 80 -q 2 -o %o %i
99456 43.7 qaac --cvbr 96 -q 2 -o %o %i
92048 47.9 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 92k %o
95985 50.1 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 96k %o
97619 59.8 ffmpeg_r59211 -i %i -vn -c:a libvo_aacenc -b:a 96k %o

MP3, Opus, Ogg Vorbis 96kbps:
97517 51.0 lame3.99.5 -V7 %i %o
95734 48.1 lame3.99.5 -V7.2 %i %o
94860 45.9 lame3.99.5 -V7.3 %i %o
93983 45.9 lame3.99.5 -V7.4 %i %o
90787 49.4 lame3.99.5 -V7.5 %i %o
91595 44.7 opus-1.1-rc-msvc2013\opusenc --bitrate 89 %i %o
92603 44.2 opus-1.1-rc-msvc2013\opusenc --bitrate 90 %i %o
98686 46.0 opus-1.1-rc-msvc2013\opusenc --bitrate 96 %i %o
99691 41.3 opus-1.1-rc-msvc2013\opusenc --bitrate 97 %i %o
86710 27.9 venc603(aoTuV) -q1.7 %i %o
87603 27.3 venc603(aoTuV) -q1.8 %i %o
88547 27.1 venc603(aoTuV) -q1.9 %i %o
88957 28.3 venc603(aoTuV) -q1.95 %i %o
89326 26.3 venc603(aoTuV) -q1.99 %i %o
95359 29.2 venc603(aoTuV) -q2 %i %o
96169 26.1 venc603(aoTuV) -q2.1 %i %o
97673 25.9 venc603(aoTuV) -q2.2 %i %o

MP3 128kbps:
124126 49.0 lame3.99.5 -V5 %i %o
130941 48.8 lame3.99.5 -V4.99 %i %o
128701 141.0 hmp3(Helix) %i %o -X2 -U2 -V60

New Public Multiformat Listening Test (Jan 2014)

Reply #239
I prefer using a test set of hopefully representative music for deciding upon which settings to use.
And from the various results given here it looks like Apple --cvbr 96, Opus --bitrate 96, aoTuv -q2 give these contenders equal chances (maybe with a tiny bit stronger aoTuv setting). These are 'natural' choices moreover.
Agree. Using of "natural" (integer) values for q-settings is another simple approach which has its pros. It makes more difficult to compare actual efficiency of codecs but provides clear answer for end users what codec/settings are better as they rarely use fractional values for q.
keeping audio clear together - soundexpert.org

New Public Multiformat Listening Test (Jan 2014)

Reply #240
Well, yes. The main reason why bitrate verification is done by different members is to avoid later discussion how reasonable was a choice of bitrate settings.
Different codecs tend to inflate bitrate (somewhat) on different music genres or particulas cases. During a preperation of the last test we had quite different bitrate reports per member but when an average bitrate was calculated it was clear what settings to go with.


88547 27.1 venc603(aoTuV) -q1.9 %i %o
95359 29.2 venc603(aoTuV) -q2 %i %o

Then we should probably stay at -q2, maybe -q2.x.


We're open to debate which MP3 encoder to include.
Currently ~50% of people have preffered LAME (sheet particular encoders). If there won't be any further suggestions, the last stable LAME 3.99.5 will be used. If someone is interested to see a different MP3 encoder  then post your suggestion here.

It's good to have LAME as it's the most popular and well optimized encoder.

New Public Multiformat Listening Test (Jan 2014)

Reply #241
I tried things to be as fair as possible, but the qaac doesn't have a flexible option, and it's not possible for aoTuV to set it around 92kbps.
Here are my plans:
Plan A: 92kbps
91650 : qaac --tvbr 45 -q 2 -o %o %i
91595 : opus-1.1-rc-msvc2013\opusenc --bitrate 89 %i %o
89417 : venc603(aoTuV) -q1.9999 %i %o
92048 : ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 92k %o

Plan B: 99kbps
99456 : qaac --cvbr 96 -q 2 -o %o %i
99691 : opus-1.1-rc-msvc2013\opusenc --bitrate 97 %i %o
99763 : venc603(aoTuV) -q2.4 %i %o
99884 : ffmpeg_r59211 -i %i -vn -c:a libfaac -b:a 100k %o

I prefer the Plan B. For MP3 128kbps, lame 3.99.5 -V5 is a sweet choice, I believe.


Average Bitrate of 122 songs, Speed in x realtime (on i7 2.93GHz), command line
AAC 96kbps:
91650 43.9 qaac --tvbr 45 -q 2 -o %o %i
106035 39.2 qaac --tvbr 54 -q 2 -o %o %i
82546 42.9 qaac --cvbr 80 -q 2 -o %o %i
99456 43.7 qaac --cvbr 96 -q 2 -o %o %i
92048 47.9 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 92k %o
95985 50.1 ffmpeg_r57288 -i %i -vn -c:a libfaac -b:a 96k %o
98918 46.1 ffmpeg_r59211 -i %i -vn -c:a libfaac -b:a 99k %o
99884 46.7 ffmpeg_r59211 -i %i -vn -c:a libfaac -b:a 100k %o
91614 58.1 ffmpeg_r59211 -i %i -vn -c:a libvo_aacenc -b:a 90k %o
97619 59.8 ffmpeg_r59211 -i %i -vn -c:a libvo_aacenc -b:a 96k %o
100620 56.9 ffmpeg_r59211 -i %i -vn -c:a libvo_aacenc -b:a 99k %o

Opus, Ogg Vorbis 96kbps:
91595 44.7 opus-1.1-rc-msvc2013\opusenc --bitrate 89 %i %o
92603 44.2 opus-1.1-rc-msvc2013\opusenc --bitrate 90 %i %o
98686 46.0 opus-1.1-rc-msvc2013\opusenc --bitrate 96 %i %o
99691 41.3 opus-1.1-rc-msvc2013\opusenc --bitrate 97 %i %o
88547 27.1 venc603(aoTuV) -q1.9 %i %o
89326 26.3 venc603(aoTuV) -q1.99 %i %o
89417 27.2 venc603(aoTuV) -q1.9999 %i %o
95359 29.2 venc603(aoTuV) -q2 %i %o
96169 26.1 venc603(aoTuV) -q2.1 %i %o
97673 25.9 venc603(aoTuV) -q2.2 %i %o
99763 27.7 venc603(aoTuV) -q2.4 %i %o

New Public Multiformat Listening Test (Jan 2014)

Reply #242
As FAAC is a middle-low anchor there is no need to tune it to others IMHO. Let it be 96.
keeping audio clear together - soundexpert.org

New Public Multiformat Listening Test (Jan 2014)

Reply #243
Agree. It still sounds noticebly worse than tested codecs both at 92k or 100k.

 

New Public Multiformat Listening Test (Jan 2014)

Reply #244
Opus 1.1@96k has the same bitrate  as Apple CVBR 96. approx. 100 kbps.

In the other hand, Apple TVBR 45 is ~94 kbps. Vorbis has not a soft bitrate curve (q1.999->q2) as previos posts have shown.

I have run a few settings on the samples from previous test  http://listening-tests.hydrogenaudio.org/i...all_samples.zip


Real bitrate. Filesize*8bits/duration(s):
Vorbis aoTuv 6.03 -q2 - 97.4 kbps
Apple TVBR 45 - 96.2 kbps
Opus 1.1  92k - 97.0 kbps

I'm good with any bitrate for MP3@128k while bitrate is around ~ 128-135. It's a kind of high anchor here.

New Public Multiformat Listening Test (Jan 2014)

Reply #245
I'm good with any bitrate for MP3@128k while bitrate is around ~ 128-135. It's a kind of high anchor here.

I like the lame 3.99.5 -V5, even if the bitrate is slightly less than 128. It's the version used by many people. Or CBR 128kbps.

New Public Multiformat Listening Test (Jan 2014)

Reply #246
I like the lame 3.99.5 -V5, even if the bitrate is slightly less than 128. It's the version used by many people. Or CBR 128kbps.

Also vote for -V5, looks like higher bitrates will make the test too hard for listeners. And yes, -V5 is popular.

Vorbis aoTuv 6.03 -q2 - 97.4 kbps
Apple TVBR 45 - 96.2 kbps
Opus 1.1  92k - 97.0 kbps


May be we should settle on these settings now. After new samples will be selected I would like to ask Kamedo2 to plot his beautiful distribution of bitrates (like this one) for the selected samples and settings. Thus we'll confirm them finally.
keeping audio clear together - soundexpert.org

New Public Multiformat Listening Test (Jan 2014)

Reply #247
May be we should settle on these settings now. After new samples will be selected I would like to ask Kamedo2 to plot his beautiful distribution of bitrates (like this one) for the selected samples and settings. Thus we'll confirm them finally.
I don't think we should settle the settings at all until the final samples have been selected. For example, with the 20 samples from the last AAC listening test I get:
  • Vorbis aoTuv 6.03 -q2 - 95.3 kbps
  • Apple TVBR 45 - 93.3 kbps
  • Opus 1.1  92k - 97.0 kbps
The problem is that you cannot fine tune the Apple encoder as much as the Xiph encoders, so we should take whatever bitrate close to 96k Apple's encoder achieves, and take that as a target bitrate for the other encoders.
It's only audiophile if it's inconvenient.

New Public Multiformat Listening Test (Jan 2014)

Reply #248
Bitrate distribution of my Pops and Jazz albums library, 122 songs.

The Apple CVBR has a quite narrow bitrate distribution, nearly constant.

New Public Multiformat Listening Test (Jan 2014)

Reply #249
  • Vorbis aoTuv 6.03 -q2 - 95.3 kbps
  • Apple TVBR 45 - 93.3 kbps
  • Opus 1.1  92k - 97.0 kbps
The problem is that you cannot fine tune the Apple encoder as much as the Xiph encoders, so we should take whatever bitrate close to 96k Apple's encoder achieves, and take that as a target bitrate for the other encoders.

Let's settle on these settings for the time being and start to select samples. We'll return to the settings afterwards. Bitrate distribution of newly selected samples will help us to tune other encoders to Apple one.
keeping audio clear together - soundexpert.org

 
SimplePortal 1.0.0 RC1 © 2008-2020