Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 64 kbit/s Listening Test Plan (Read 11379 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

64 kbit/s Listening Test Plan

Here is my test plan for a group listening test I'd like to perform at 64 kbit/s.

http://ff123.net/64test/testplan.html

Please comment.

ff123

64 kbit/s Listening Test Plan

Reply #1 –
Sounds sensible, can't wait. Too bad ABC/HR limits people to windows, though.

64 kbit/s Listening Test Plan

Reply #2 –
Quote
Originally posted by ff123
Here is my test plan for a group listening test I'd like to perform at 64 kbit/s.

http://ff123.net/64test/testplan.html

Please comment.

Sounds good. One thing - speaking as someone with crap hearing, it would be useful if you included an 'obviously bad' sample, as well as an 'obviously good'. This would provide a bit more framing of the results, as well as giving the poor listeners like me some confidence that we *can* actually hear something.

Perhaps Blade@64?

64 kbit/s Listening Test Plan

Reply #3 –
Might be good to add QuickTime AAC @64 Kb/s (or some other AAC encoder), because MPEG-4 audio will become very interesting in months to come.

And, QuickTime is freely available, and fully legal (licensed) for end-users.

64 kbit/s Listening Test Plan

Reply #4 –
I'd like to keep the comparisons to a bare minimum, so although I'd like to compare an AAC, perhaps it's better to save this for a future test.  As for the "bad" reference file, I could add 64 kbit/s normal mp3, but I think the cost to the statistics (lose discriminating power with each additional sample) would not be worth the benefit.  As it is, it's almost not worth it to me to include the original as a sample.

ff123

64 kbit/s Listening Test Plan

Reply #5 –
Quote
As for the "bad" reference file, I could add 64 kbit/s normal mp3, but I think the cost to the statistics (lose discriminating power with each additional sample) would not be worth the benefit.  As it is, it's almost not worth it to me to include the original as a sample.

No problem. I can understand that including the 'bad' sample would give you very little information (already a high prior probability that just about everyone would rate it lowest in every test). Might a deliberately bad sample be good as a pre-screener, or have you decided that pre-screening is just too statistically complicated?

64 kbit/s Listening Test Plan

Reply #6 –
I dont understand the purpose of having the mp3 128 kbit sample in the test. I would be more interested in testing AAC+SBR 64 kbit instead of old boring mp3.

But that depends on the purpose of the test (which I btw can't find on your page). Is it to compare low bitrate codecs against eachother and rank them worst -> best, or to find out if new low bitrate codecs are on par with the old standard mp3 128 kbit yet or maybe both?

/Erik

64 kbit/s Listening Test Plan

Reply #7 –
Quote
Originally posted by ErikS
I dont understand the purpose of having the mp3 128 kbit sample in the test. I would be more interested in testing AAC+SBR 64 kbit instead of old boring mp3. 

But that depends on the purpose of the test (which I btw can't find on your page). Is it to compare low bitrate codecs against eachother and rank them worst -> best, or to find out if new low bitrate codecs are on par with the old standard mp3 128 kbit yet or maybe both?

/Erik


Purpose of the test is to compare low bitrate codecs against each other, but also to frame the relative quality against a "standard" compression scheme, i.e., mp3 at 128 kbit/s.

ff123

64 kbit/s Listening Test Plan

Reply #8 –
i agree that there should be an aac sample there as well.
Quote
Lame mp3 at 128 is used as a "standard" quality indicator
whats the meaning of that line? (i though 'standard' quality is lame aps?)
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

64 kbit/s Listening Test Plan

Reply #9 –
Quote
Originally posted by smok3
i agree that there should be an aac sample there as well.
whats the meaning of that line? (i though 'standard' quality is lame aps?)


It makes sense to calibrate a quality scale against mp3 128 kbit/s because a lot of people will understand what it means.

AAC:  well, I may consider this in place of the Original used as a sample, since I'm teetering on the fence on that one anyway.  Which is the preferred aac implementation at 64 kbit/s?

ff123

64 kbit/s Listening Test Plan

Reply #10 –
I suggest QuickTime @64  or PsyTEL @64 -  QuickTime is actually FhG's implementation of AAC, and it might be interesting to follow...

64 kbit/s Listening Test Plan

Reply #11 –
Looking forward to the test.

Being new to listening tests, I am especially happy to see you plan to include pre-test training

64 kbit/s Listening Test Plan

Reply #12 –
If the test is is VBR Vorbis vs CBR everything-else, the choice of test samples will decide who wins because you're comparing apples and oranges.  In such a test:

If the test sample is 'deerhunter', WMA will win.  (Vorbis will also encode it at only ~32kbps using -q <= 0)

If the test sample is 'Dire Straits _Money For Nothing_',  Vorbis will win and WMA will cause strokes requiring hospitalization.

The point here is that if people listen to both those tracks in Vorbis, they'll agree they sound of similar quality.  In WMA, deerhunter will sound very good (because it's relatively easy to encode) and Dire Straits will sound like a robot laser war.  When WMA falls apart, and it's easy to make it fall apart, it becomes unlistenable.  I hope to have designed Vorbis to be an order of magnitude more robust.  This was one of the design considerations in Vorbis low bitrate:  It Will Not Fall Apart Unless Dropped From A Great Height.  It is meant to be stable and reliable, not a quality rollercoaster.

Now, if you compare Vorbis using -b --managed, it's no longer comparing apples and oranges;  this is forcing Vorbis to average bitrate, much more similar to WMA/mp3-pro's restrictions.  It will cause the hard samples to sound like crap and the easy samples to sound really good.

(BTW, I no longer have the reservations about folks using Vorbis ABR/BBR like I did before.  The bitrate management engine in 1.0 is much nicer than the one in rc3)

64 kbit/s Listening Test Plan

Reply #13 –
Quote
Originally posted by xiphmont
If the test is is VBR Vorbis vs CBR everything-else, the choice of test samples will decide who wins because you're comparing apples and oranges.  In such a test:
Seems that ff123 suggested the use of the new mp3pro VBR. Though I thought it's only available with CoolEdit 2 pro, not with MMJB 7.2?
Juha Laaksonheimo

64 kbit/s Listening Test Plan

Reply #14 –
I would like to also request that AAC be included (both Quicktime and Psytel).

Since you are testing a relatively low bit rate (i.e. ideal for streaming/portable audio), I also think that it is important to separate the test into two categories: music and spoken word. Alternatively, you may wish to run a separate test to see what encoder/setting gets the lowest possible bitrate while still sounding indistinguishable to most people's ears with regard to spoken word.

64 kbit/s Listening Test Plan

Reply #15 –
64kbits is a bit overkill for spoken word..

If I'm not mistaken, aren't there some specialized spoken word codecs that far outperform mp3, wma already?

64 kbit/s Listening Test Plan

Reply #16 –
Quote
Originally posted by floyd
If I'm not mistaken, aren't there some specialized spoken word codecs that far outperform mp3, wma already?


Yes. Specifically, ACELP and VoxWare.

Soon, Speex probably.

64 kbit/s Listening Test Plan

Reply #17 –
Quote
I may consider this in place of the Original used as a sample, since I'm teetering on the fence on that one anyway.

yeah probably at 64kbps the goal to sound as close to original as possible isnt really a goal anymore, or am i wrong?
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

64 kbit/s Listening Test Plan

Reply #18 –
Quote
Originally posted by xiphmont
Now, if you compare Vorbis using -b --managed, it's no longer comparing apples and oranges;  this is forcing Vorbis to average bitrate, much more similar to WMA/mp3-pro's restrictions.  It will cause the hard samples to sound like crap and the easy samples to sound really good.


The suggestion is to use -b --managed for vorbis, and probably mp3pro cbr would be a fairer test as well (BTW, mp3pro VBR is included in MMJB 7.2).  You've convinced me.  I think I'll drop the original and include an AAC.  So the candidates are now:

1.  ogg vorbis 1.0 using -b 64 --managed
2.  mp3pro 64 from MMJB 7.2
3.  wma8 from WMP 7.1; decoded with WMP 7.1 and captured with Total Recorder
4.  an AAC (any preferences here?  what command line options?)
5.  lame 3.92 --alt-preset cbr 128 --scale 1

ff123

64 kbit/s Listening Test Plan

Reply #19 –
Quote
Originally posted by ff123
1.  ogg vorbis 1.0 using -b 64 --managed

I don't quite like it. Is it possible to use a binary -q search to get 64kbps?

64 kbit/s Listening Test Plan

Reply #20 –
Quote
Originally posted by tangent

I don't quite like it. Is it possible to use a binary -q search to get 64kbps?


Heh, I don't really like it either.  The point of VBR is to get more usable results.  It's mildly annoying to have to compete on WMA's brain-dead terms because WMA is incapable of sophisticated operation.

Perhaps the real lesson here is that forcing every question you want to know into a single test is the wrong solution.  Forcing everyone to use ABR/CBR will still provide useful results though.  I'm just bitching because WMA is tuned for specifically that, while 90% of Ogg tuning goes to VBR.

Perhaps what I'm really saying is that the test should have some tighter scope to begin with...
"Best compression for streaming at target rate foo" means ABR/CBR.
"Best compression for  using your music collection on a portable" clearly means 'use VBR if ya got it'

64 kbit/s Listening Test Plan

Reply #21 –
I'd have to agree with Monty.

I don't think that forcing Vorbis to a specific bitrate and using the managed mode is very representive of how most people will use the codec, so how useful are the results really going to be in that situation anyway?  Perhaps this is similar to --alt-preset 128 --scale 1.  Again, while it may be more technically correct to disable scaling, this isn't how most people are going to be using the preset.  The results aren't going to be very representative of a "real world" situation, but you can be certain that this is how the majority of readers will end up viewing them.

Anyway, I agree that perhaps the test focus should be split into two seperate categories like Monty proposed.  Smaller groups of tests should probably be done to test more specific aspects of the encoders and how they perform in different situations, and then some sort of summary or write up comparing their strengths and weaknesses could be presented.

You need to remember that there are going to be a lot of people looking and these results and using them as a "reference" of sorts for deciding upon which codec to use.  Because of this, every effort should be made to make the results both technically correct and relevant to real world usage, even if it means that it will require more of an initial effort and more seperate tests.

64 kbit/s Listening Test Plan

Reply #22 –
Quote
Originally posted by xiphmont
If the test sample is 'deerhunter', WMA will win.  (Vorbis will also encode it at only ~32kbps using -q <= 0)

Couldn't this problem be resolved by not fixing OGG's test samples at -q0? Instead, try a few different -q settings and find one closest to 64 kbps? IOW, assuming ff123 wants to test both 'deerhunter' and 'Dire Straits', he could encode the former with, say, -q 1, and the latter with -q 0,  (assuming both come out around 64 kbps), and compared them with 64 k wma (or mp3pro, whatever).

Or am I missing something?
tw101

64 kbit/s Listening Test Plan

Reply #23 –
Quote
Originally posted by ff123
Here is my test plan for a group listening test I'd like to perform at 64 kbit/s.

http://ff123.net/64test/testplan.html

Please comment.


what do you think about resampling ?
most AAC/mp3 encoder will resample the input into 32kHz or 22kHz when the low bitrate encoding.

oggenc does not do so, but I think "-q0 --resample 32000" will be better than "-q0", and using ssrc will be more better.
May the source be with you! // Takehiro TOMINAGA

64 kbit/s Listening Test Plan

Reply #24 –
Quote
oggenc does not do so, but I think "-q0 --resample 32000" will be better than "-q0", and using ssrc will be more better.

Actually, I don't think you'll need to resample at -q 0. It sounds incredible already. Adjusting the lowpass may be a better idea.

Dibrom's right that there are many tests you could do that would be ostensibly the same, but would have an inbuilt bias toward one or the other. The simplest is probably to test the codec 'as a stupid user would use them', so pure 64kbit for WMA, -q 0 for Vorbis, --alt-preset 64 for LAME. As xiphmont pointed out, the burden then falls on the test samples you are using.