Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 64 kbit/s Listening Test Plan (Read 11344 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

64 kbit/s Listening Test Plan

Reply #25 –
ff123, I suggest using QuickTime AAC for 64 kbps, because it is a FhG's implementation - and FhG is probably the best AAC encoder for very low bit rates, like 64 Kbps.

PsyTEL also might be worth trying (aacenc -br 64 -resample 32000 -c 12000)

If I remember correctly, FhG's and Dolby's professional AAC encoders are using 32000 Hz sampling rate with 12 kHz cut-off at 64 Kbps.  Some FhG's codecs, like one in LiquidAudio are using 10 kHz cut-off - but 12 kHz is much better (QT uses this)

64 kbit/s Listening Test Plan

Reply #26 –
Quote
Originally posted by Jon Ingram

Actually, I don't think you'll need to resample at -q 0. It sounds incredible already. Adjusting the lowpass may be a better idea. 

Dibrom's right that there are many tests you could do that would be ostensibly the same, but would have an inbuilt bias toward one or the other. The simplest is probably to test the codec 'as a stupid user would use them', so pure 64kbit for WMA, -q 0 for Vorbis, --alt-preset 64 for LAME. As xiphmont pointed out, the burden then falls on the test samples you are using.


Perhaps a solution could be to test both -q 0 and -b 64 --managed at the same time.  But now that's yet another sample.

ff123

Edit:  Ok, I dropped lame at 128 kbit/s, so it is now a pure 64 kbit/s test.  AAC entry will be Quicktime.

64 kbit/s Listening Test Plan

Reply #27 –
Quote
Originally posted by rjamorim


Yes. Specifically, ACELP and VoxWare.

Soon, Speex probably.


Not to beat a dead horse but... I and I am sure many others are looking for the best codecs and best codec settings that work with their hardware for a given type of content.  Otherwise, you are doing lossless codec comparisons for the sake of doing comparisons.

You are testing several codecs at a relatively low bit rate. Great! Low bit rate encoding matters for bandwidth and memory limited situations--webcasting and portable audio devices.  Even if webcasting were not choking in the US due to CARP, broadcasters are still going to use the most common codecs to ensure the easiest way to reach the largest number of listeners. Similarly, portable audio devices all support MP3, many support WMA (yuck), many will support AAC, some are talking about MP3pro and I am not aware of any large manufacturer supporting Ogg.

Continuing, it makes sense to look at what content is played back on these devices--and for many that is spoken word (books, language lessons, news, etc.) as much as it is music. While some care only about storage space at the expense of sound quality (say news in your native language) and for those situations Audible 4's 32Kb/s MP3 setting (the specifics of which I am uncertain) *works sort-of*, there are also situations where reproducing the full spectral range of voice is desireable (for example, learning a foreign language where the nuances may be missed in a muddy 32K MP3 recording).  For spoken word, I have found 64K AAC and MP3pro work well but LAME alt-preset 80 has significant artifacts on passages with echo. I suspect that I can do better than 64K within the confines of the onboard codec. Coincidentally, this also matters to webcast "talk radio" since they are not affected by CARP.

As background, I store everything lossless on my server and then downward encode for the portable device. For music, I reencode it in LAME alt-preset standard but am seriously eyeing AAC and wish the open-source community would rally behind that as it has with LAME. With 60% annualized growth in hard drive capacity, I think most users will take this approach in the future (lossless archive; low bit rate portable) and wish that testing would proceed accordingly.

64 kbit/s Listening Test Plan

Reply #28 –
Quote
Originally posted by costello
For music, I reencode it in LAME alt-preset standard but am seriously eyeing AAC and wish the open-source community would rally behind that as it has with LAME.

That's just not going to happen. Licensing, licensing, licensing.

64 kbit/s Listening Test Plan

Reply #29 –
Quote
Originally posted by Jon Ingram

That's just not going to happen. Licensing, licensing, licensing.


We are getting way off topic but....

Come again? How is LAME exempt from patent licensing fees for the MP3 format anymore than an open source implementation of AAC would be?

The only difference between the two as I see it is that the MP3 patents might be older and that much closer to expiration, that AAC's might cover a broader geographic region (though I haven't studied the issue), and that the two standards are administered by different licensing organizations.

The AAC licensing body should set up a shopping cart for open source software users to purchase the patent rights (under $2-3 per user) if they are located in a geographic region where the patents apply. I can only think that they haven't done this because they see the cost to them outweighing the benefit.

In any event, I would only be encoding into AAC in order to playback the file on my AAC enabled hardware device (the manufacturer of which presumably purchased the patent license, if not source code copyright license, outright). For everything else, I would just use the lossless original.

64 kbit/s Listening Test Plan

Reply #30 –
Sounds like a great test, ff123!

I would prefer only four samples: Ogg q=0 (VBR), MP3pro, WMA and AAC together with the original. (Ogg should average 64kbps so no problem with VBR for me).

Just one quick input: Since these samples will/should sound different from the orginal it may be bit of a problem to pick out the best codec from the test. We all have a different taste to what we accept as an okay artifact/subtle difference and what we consider too annoying.

Regards, stoff

64 kbit/s Listening Test Plan

Reply #31 –
Quote
Originally posted by stoff
Sounds like a great test, ff123!

I would prefer only four samples: Ogg q=0 (VBR), MP3pro, WMA and AAC together with the original. (Ogg should average 64kbps so no problem with VBR for me).

Just one quick input: Since these samples will/should sound different from the orginal it may be bit of a problem to pick out the best codec from the test. We all have a different taste to what we accept as an okay artifact/subtle difference and what we consider too annoying.

Regards, stoff


Different preferences is what will make this test very interesting, I think.

ff123

64 kbit/s Listening Test Plan

Reply #32 –
Quote
Originally posted by stoff
Sounds like a great test, ff123!

I would prefer only four samples: Ogg q=0 (VBR), MP3pro, WMA and AAC together with the original. (Ogg should average 64kbps so no problem with VBR for me).


Previous build of Ogg haven't averaged 64kbps qt -q0 (see audio.ciara.us/compare.htm for details).  I need to do some updating, but to date that hasn't been a strictly correct assertion (close, but not quite).

Cheers, Paul

64 kbit/s Listening Test Plan

Reply #33 –
Premature start for the 64kbps test.

Original sample: #41 by Dave Matthews band, 30 sec clip. Notice that this clip of music is quite hard to encode.
http://www.ff123.net/samples/41_30sec.flac

Vorbis 1.0 -b64 --managed (latest fixed compile actual bitrate 62.6kbps):
http://sivut.koti.soon.fi/julaak/41_30secOgg-b64M.ogg
Vorbis 1.0 -q0 (bitrate 70.7 kb/s):
http://sivut.koti.soon.fi/julaak/41_30secOgg-q0.ogg

Quicktime AAC-64 kbps (Fhg/Dolby, no intensity stereo selectable, 32khz sampling by default):
http://sivut.koti.soon.fi/julaak/41_quicktime_64.aac
Decoded Quicktime AAC-64 kbps:
http://sivut.koti.soon.fi/julaak/41_quicktime_64.flac

WMA8 64 kbps:
http://sivut.koti.soon.fi/julaak/41_30secWMA8-64.wma
Decoded WMA8 64 kbps:
http://sivut.koti.soon.fi/julaak/41_30secWMA8-64.flac

Mp3Pro-64 (Slowest high quality, allow mid/side, allow intensity stereo):
http://sivut.koti.soon.fi/julaak/41_30secMP3pro-64.3pm
Decoded mp3PRO 64kbps:
http://sivut.koti.soon.fi/julaak/41_30secMP3pro-64.flac

MP3pro WinAmp Plugin that recognaises .3pm files as MP3pro files, so you can still use Nullsoft/Mad mp3-decoder:
http://sivut.koti.soon.fi/julaak/mp3pro-v0.98hacked.exe

[*Additions*]
VQF-64 kbps (Yamaha 2.60b9 using high quality setting, 22khz sampling) decoded:
http://www.saunalahti.fi/~cse/41_30sec_vqf64.flac

FhG MP3Enc 3.1 64kbps (-qual 9, 22khz sampling by default)
http://sivut.koti.soon.fi/julaak/41_30secMP3Enc31.mp3
Juha Laaksonheimo

64 kbit/s Listening Test Plan

Reply #34 –
Quote
Originally posted by JohnV
Premature start for the 64kbps test.

Original sample: #41 by Dave Matthews band, 30 sec clip. Notice that this clip of music is quite hard to encode.


Indeed!

My very first (and very quick!) try with ff123's new fancy abchr tool:

------------------------------------------------------------------------------------------------------------
1L = C:Documents and SettingsStofferMy DocumentsMy MusicTest41_30secOgg-q0.wav
2L = C:Documents and SettingsStofferMy DocumentsMy MusicTest41_quicktime_64.wav
3R = C:Documents and SettingsStofferMy DocumentsMy MusicTest41_30secMP3pro-64.wav
4L = C:Documents and SettingsStofferMy DocumentsMy MusicTest41_30secWMA8-64.wav

---------------------------------------
1L File: C:Documents and SettingsStofferMy DocumentsMy MusicTest41_30secOgg-q0.wav
1L Rating: 3.5
1L Comment: Cymbal somewhat off, lacks detail, but not to annoying. Midtone a bit off (lacks power). Clearly the best!
---------------------------------------
2L File: C:Documents and SettingsStofferMy DocumentsMy MusicTest41_quicktime_64.wav
2L Rating: 1.0
2L Comment: Heavy distortions/artifacts. Forget about this one!
---------------------------------------
3R File: C:Documents and SettingsStofferMy DocumentsMy MusicTest41_30secMP3pro-64.wav
3R Rating: 2.1
3R Comment: Sqeshy-washy sound through. Better than 2.
---------------------------------------
4L File: C:Documents and SettingsStofferMy DocumentsMy MusicTest41_30secWMA8-64.wav
4L Rating: 1.0
4L Comment: Heavy distortions abound! Maybe as bad as 2. Rejected.
---------------------------------------

Don't put to much into my comments please! This was a quick test, but there is no doubt that Ogg at q=0 was superior to me! (I decoded the Ogg file with oggdropXPd)

I have never done testing at ~64 kbps before. It's quite fun actually!

Regards, stoff

64 kbit/s Listening Test Plan

Reply #35 –
I added the two additional encodes (VQF and MP3Enc) as well as a Psytel AAC encode.
Psytel's 64kbps setting (as proposed by Ivan) didn't sound as good as Quicktime's, but the -tape preset gave 74.7kbps and didn't sound too bad, so I put it in the mix too.

SUMMARY:
Ogg -q0: 3.6
Ogg -b64: 3.3
MP3Pro 64: 2.4
MP3Enc 64: 1.2
AAC Quicktime 64: 1.5
(AAC Psytel -tape: 2.2)
WMA8 64: 2.0
VQF 64: 2.3

Code: [Select]
1R = D:Music64kbps listening test41_30secWMA8-64.wav

2R = D:Music64kbps listening test41_30secVQF64.wav

3L = D:Music64kbps listening test41_30secMP3Enc31.wav

4L = D:Music64kbps listening test41_30secOgg-q0.wav

5L = D:Music64kbps listening test41_30secOgg-b64M.wav

6R = D:Music64kbps listening test41_30secMP3Pro-64.wav

7R = D:Music64kbps listening test41_30secPsyAAC-tape.wav

8R = D:Music64kbps listening test41_30secQTAAC_64.wav



---------------------------------------

1R File: D:Music64kbps listening test41_30secWMA8-64.wav

1R Rating: 2.0

1R Comment: Argh... pick a lowpass and stick to it!!!

---------------------------------------

2R File: D:Music64kbps listening test41_30secVQF64.wav

2R Rating: 2.3

2R Comment: Low bandwidth is annoying. The sound that is left, though, sounds fairly 

clean, with a tolerable amount of swirling. I can still place the sounds' 

locations in the stereo image. No high-freqs can be better than bad high-freqs!

---------------------------------------

3L File: D:Music64kbps listening test41_30secMP3Enc31.wav

3L Rating: 1.2

3L Comment: Very low bandwidth. Hi-hats flange and don't stay still in the stereo image.



Dropouts so bad, even I can hear them!

---------------------------------------

4L File: D:Music64kbps listening test41_30secOgg-q0.wav

4L Rating: 3.6

4L Comment: Hi-hats are a bit too sharp-sounding and flat (not enough spread to the right

channel). Its full-bandwidth encoding makes it sound pretty decent.



Guitar sounds nice!

---------------------------------------

5L File: D:Music64kbps listening test41_30secOgg-b64M.wav

5L Rating: 3.3

5L Comment: Sound is comparable to Sample 4. More flanging is apparent, and the

hi-hat attack isn't as sharp.



ABX vs. Sample 4: 10/12 (I prefer sample 4's better hi-hats)

---------------------------------------

6R File: D:Music64kbps listening test41_30secMP3Pro-64.wav

6R Rating: 2.4

6R Comment: Hi-hat artifacts are annoying, but the lowpass doesn't seriously detract

from the music.

---------------------------------------

7R File: D:Music64kbps listening test41_30secPsyAAC-tape.wav

7R Rating: 2.2

7R Comment: Hi-hats aren't placed well in the stereo image, and have a somewhat

annoying lowpass.

---------------------------------------

8R File: D:Music64kbps listening test41_30secQTAAC_64.wav

8R Rating: 1.5

8R Comment: So what if the lowpass isn't as bad as sample 2's... the hi-hats are all over

both channels!

---------------------------------------


Conclusion: Vorbis rules (at least on this sample, it does). VQF's low lowpass made it impossible not to distinguish, but I'd rather listen to it than to the samples that completely garbled the hi-hat.

64 kbit/s Listening Test Plan

Reply #36 –
It seems like the best way to set up this test with ogg is to vary the -q setting until an average bitrate of 64 kbits/sec is achieved. It may seem like a lot of hassle, but at least this way, in the end, you will be able to state that all formats were compared at the same average bitrate (or file size).... IMO this criterion is *the* criterion to use.

Perhaps you could use other methods to get ogg at 64 kbits/sec average, but there are two potential problems that I see: 1) they are less likely to be used in the real world, 2) they would generally not be as high quality (though I'm not sure about this).

Ogg's "-q" setting is a unique and, I think most would agree, an attractive feature of Vorbis. It should be part of the test for sure.

just my 2 cents.

64 kbit/s Listening Test Plan

Reply #37 –
Quote
Originally posted by shday

Ogg's "-q" setting is a unique and, I think most would agree, an attractive feature of Vorbis.


Well, it's not unique (MPC has --quality x.xx now), but at least it's attractive. 

Anyway, i just came across these graphs (done by xercist), bitrate vs. -q scale:
http://www.lammah.com/~xercist/vorbis/bitrate-graph/1.0/

64 kbit/s Listening Test Plan

Reply #38 –
Quote
Originally posted by shday
It seems like the best way to set up this test with ogg is to vary the -q setting until an average bitrate of 64 kbits/sec is achieved. It may seem like a lot of hassle, but at least this way, in the end, you will be able to state that all formats were compared at the same average bitrate (or file size).... IMO this criterion is *the* criterion to use. 


I think that most people will either use -q 0 or -b64 if they really can't deviate from a bitrate of 64 kbit/s.  So I think testing both settings is the fairest way.  I doubt people will encode particular tracks by trial and error by varying the -q x setting until they get a bitrate they are happy with.

ff123

64 kbit/s Listening Test Plan

Reply #39 –
Actually, I wasn't suggesting people would try to achieve an average bitrate of 64 kbits/sec by varying the -q setting. Most people would just trust -q 0 to do its thing.

I just thought you may want to use this method for the test only, for the reasons I stated.

Testing both -q 0 and -b64 sounds good also

64 kbit/s Listening Test Plan

Reply #40 –
Quote
Originally posted by ff123


I think that most people will either use -q 0 or -b64 if they really can't deviate from a bitrate of 64 kbit/s.  So I think testing both settings is the fairest way.  I doubt people will encode particular tracks by trial and error by varying the -q x setting until they get a bitrate they are happy with.

ff123


I agree with ff123.  If he's willing to do both -q0 and -b 64 --managed, that covers the Ogg bases.

Two points:

1) You need --managed with -b to get ABR now.

2) at < -q 0 (any negative -q) with 44.1kHz, Vorbis switches to a larger blocksize.  Thus -q -.001 and -q 0 could potentially be very different, depending on sample.  It's not a 'Don't do this!', it's 'be aware of it'.  -b 64 --managed is really the right way to do the ABR/CBR comparison as the bitrate management is much better than it was in rc3

64 kbit/s Listening Test Plan

Reply #41 –
Practice page added. The zipfile will be added when Ogg Vorbis 1.0 is officially released.  I'll also create links from the test planning page to initiate the practice test at that time.  Downloading the practice test will be "mandatory," in order to create the directories, get abchr.exe, flac.exe, oggenc.exe, and oggdec.exe.

http://ff123.net/64test/practice.html

ff123

64 kbit/s Listening Test Plan

Reply #42 –
Hello,

here are my results. Sorry for the comments, they aren't really saying much for two reasons:
- I don't have any experience in blind tests, this was my first one
- My English isn't that good when it comes to these special expressions 'how a sound sounds'

This ABX thing is quite cool... I'll test some friends' ears... *g*

1L = g:64k\\002vqf.wav = 41_30sec_vqf64.flac
2L = g:64k\\003mp3.wav = 41_30secMP3Enc31.mp3
3R = g:64k\\0043pm.wav = 41_30secMP3pro-64.flac
4R = g:64k\\008aac.wav = 41_quicktime_64.flac
5R = g:64k\\005ogg1.wav = 41_30secOgg-b64M.ogg
6L = g:64k\\007wma.wav = 41_30secWMA8-64.flac
7R = g:64k\\006ogg2.wav = 41_30secOgg-q0.ogg

---------------------------------------
1L File: g:64k\\002vqf.wav
1L Rating: 1.3
1L Comment: totally muffled, like everybody knows this from internet streams, i hate it
---------------------------------------
2L File: g:64k\\003mp3.wav
2L Rating: 1.0
2L Comment: even worse than sample 1
---------------------------------------
3R File: g:64k\\0043pm.wav
3R Rating: 2.9
3R Comment:
---------------------------------------
4R File: g:64k\\008aac.wav
4R Rating: 2.0
4R Comment:
---------------------------------------
5R File: g:64k\\005ogg1.wav
5R Rating: 2.2
5R Comment: the stereo is crappy... it seems to fade around from left to right and back with high speed ... the whole sound is no longer fluently, but stresses my ears... except this stereo thing, the sound-quality is very good
---------------------------------------
6L File: g:64k\\007wma.wav
6L Rating: 2.9
6L Comment: same level as sample 3 - I could recognize some weird artifacts at the beginning
---------------------------------------

>>> Huh? I thought it was the original - so I didn't make any changes... and the program didn't write anything down here... --> got a 5.0!

64 kbit/s Listening Test Plan

Reply #43 –
I've two another 2 cents to add...

How about encoding all the audio clips with ogg at -q 0 and then forcing all the other codecs to the same bitrates? This should be a fair comparison unless the other codecs have some special optimisation at 64 kbits/sec.

64 kbit/s Listening Test Plan

Reply #44 –
Quote
Originally posted by shday
I've two another 2 cents to add...

How about encoding all the audio clips with ogg at -q 0 and then forcing all the other codecs to the same bitrates? This should be a fair comparison unless the other codecs have some special optimisation at 64 kbits/sec.


Most of the other codecs can only encode at specified bitrates, for example 40, 64, 80, etc.

ff123

64 kbit/s Listening Test Plan

Reply #45 –
1L = f:abcMP3pro-64_flac.wav
2R = f:abcWMA8-64_flac.wav
3L = f:abcMP3Enc31.wav
4L = f:abcquicktime 64_flac.wav
5R = f:abcvqf64_flac.wav
6R = f:abcOgg-q0.wav
7L = f:abcOgg-b64M.wav

---------------------------------------
1L File: f:abcMP3pro-64_flac.wav
1L Rating: 3.2
1L Comment: Some smearing, but overall pretty impressive
---------------------------------------
2R File: f:abcWMA8-64_flac.wav
2R Rating: 1.3
2R Comment: Horrible arifacting, rining, flanging, etc...Totally unlistenable
---------------------------------------
3L File: f:abcMP3Enc31.wav
3L Rating: 2.0
3L Comment: Very muffled, but quite solid otherwhise
---------------------------------------
4L File: f:abcquicktime 64_flac.wav
4L Rating: 2.2
4L Comment: A lot of smearing and flanging, but bearable
---------------------------------------
5R File: f:abcvqf64_flac.wav
5R Rating: 2.8
5R Comment: Muffled, but very solid
---------------------------------------
6R File: f:abcOgg-q0.wav
6R Rating: 3.8
6R Comment: Little smearing, and little loss of stereo imaging, but quite good. I'm impressed
---------------------------------------
7L File: f:abcOgg-b64M.wav
7L Rating: 3.6
7L Comment: Loss of stereo imaging, a little more smearing than 6
---------------------------------------

64 kbit/s Listening Test Plan

Reply #46 –
I didn't notice LAME in there anywhere.

I've been using WMA at 96kbs for my portable, but was thinking of moving to --alt-preset 112. At these bitrates I can fit an entire CD on a 64MB CF card, or a double on a 128MB card.

Is LAME so bad at these bitrates that it's not worth testing?
flac > schiit modi > schiit magni > hd650

64 kbit/s Listening Test Plan

Reply #47 –
Hi,

I'm using LAME with 112 ABR for my portable... The sound isn't that bad, I think... I would prefer Vorbis, but my CPU is too weak...

Regards, fileman.
 

64 kbit/s Listening Test Plan

Reply #48 –
1L = 41_3041_quicktime_64.wav
2R = 41_3041_30secWMA8-64.wav
3R = 41_3041_30secMP3Enc31.wav
4L = 41_3041_30secOgg-q0.wav
5L = 41_3041_30secOgg-b64M.wav
6L = 41_3041_30secMP3pro-64.wav
7L = 41_3041_30sec_vqf64.wav

---------------------------------------
1L File: 41_3041_quicktime_64.wav
1L Rating: 2.0
1L Comment: very soft transients. just not enjoyable.
---------------------------------------
2R File: 41_3041_30secWMA8-64.wav
2R Rating: 1.5
2R Comment: weird artifacts in the cymbals, pitch seems to fluctuate
annoying like 6, but in a different way - sounds a bit like excessive noise reduction being performed.
---------------------------------------
3R File: 41_3041_30secMP3Enc31.wav
3R Rating: 1.3
3R Comment: quite nice mid-freq tonality, but the complete treble is f'd up beyond recognition. make it go away
---------------------------------------
4L File: 41_3041_30secOgg-q0.wav
4L Rating: 3.5
4L Comment: this codec owns this sample. bet it's ogg, because 5 is almost as good,
does sound quite similar and both show the same stereo collapsing (a bit less on 4).
amazing treble and transients compared to the rest!
---------------------------------------
5L File: 41_3041_30secOgg-b64M.wav
5L Rating: 3.0
5L Comment: some notes in the midrange sound "unstable". see 4 for other thoughts.
still a whole notch above the other competitors imo.
---------------------------------------
6L File: 41_3041_30secMP3pro-64.wav
6L Rating: 1.5
6L Comment: the stereo image sounds artificially wide
very weird artifacting at the beginning. very noisy, which is why i slightly prefer 2.
---------------------------------------
7L File: 41_3041_30sec_vqf64.wav
7L Rating: 1.0
7L Comment: :eek: *faints*
---------------------------------------


note: my first attempt to describe any kind of deficiencies in music reproduction, so bear with me

*puts Dave Matthews Band - Crash into cd player* aaah, I still prefer the real thing
A riddle is a short sword attached to the next 2000 years.