Skip to main content
Topic: Dial-up bitrate listening test (Read 27548 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Dial-up bitrate listening test

Reply #50
Hi guys !
About that idea with speach samples.
I understand, why ITU used them - they all are telephony people and speach is what they usually work on. So their choise is understandable.
This test is stereo audio test, right ?
But speach samples, except some radio plays are mono or pseudo stereo. I think (I may be wrong) that speach stereo samples are very artificial and not that common to specially test them a lot. 2 samples will be enought...
Just my 2 cents 
P.S. If we gonna to test mono low bitrate encodings then yes, speach is what we need...

Dial-up bitrate listening test

Reply #51
I'm in favor of letting the encoder decide about resampling.
This is perhaps something which would need a pre-test, the performance of SSRCed (22khz? 32khz?) souces vs. original bandwith sources.

As it was pointed out, none of the current Vorbis tunings focus on the managed low bitrate modes, so I'd definetly vote for using the current CVS, which also features the new managed bitrate code (libvorbis I 20031230 - more about it here).
oggenc --managed -b 30 should be a good cmd.line. I don't think imposing additional restrictions is neccessary when using Vorbis' managed bitrate mode.

dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

Dial-up bitrate listening test

Reply #52
Perhaps this should be two different tests?  Streaming speech (e.g. news conferences) and streaming music (internet radio) are quite different beasts.  Your choice of codec will very likely depend on the sort of content you're streaming.  Speex would not do that well in a music test but would likely win a speech test.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

Dial-up bitrate listening test

Reply #53
Quote
Hey, Ben !!!

"Note the RA8 stereo codec is way better than the RA mono codec, even with mono source."

?????

I'm from Missouri......

Quote
Hey, Ben !!!

"Note the RA8 stereo codec is way better than the RA mono codec, even with mono source."

?????

I'm from Missouri......


Just try it. It's the much better "cook" codec. Dramatic difference with most content.

Dial-up bitrate listening test

Reply #54
Quote
I'm in favor of letting the encoder decide about resampling.
This is perhaps something which would need a pre-test, the performance of SSRCed (22khz? 32khz?) souces vs. original bandwith sources.

As it was pointed out, none of the current Vorbis tunings focus on the managed low bitrate modes, so I'd definetly vote for using the current CVS, which also features the new managed bitrate code (libvorbis I 20031230 - more about it here).
oggenc --managed -b 30 should be a good cmd.line. I don't think imposing additional restrictions is neccessary when using Vorbis' managed bitrate mode.

dev0

Quote
I'm in favor of letting the encoder decide about resampling.
This is perhaps something which would need a pre-test, the performance of SSRCed (22khz? 32khz?) souces vs. original bandwith sources.


We'll definitely need to do some manual tweaking. Some codecs don't do resampling automatically at all, and others offer sub-optimal defaults. The right resampling rate likely will vary with different sources with some content (hence the "high response" RealAudio variants).

Quote
So, would CBR or hard-limited max bit rate be a better choice, considering this is a streaming test and that the results should reflect that (and not pseudo-idealized conditions)?


I agree. CBR is the appropriate mode for this test. Note that at least one codec (WMA9) supports a 2-pass mode, which I suggest should be used. That also begs the question of an appropriate buffer size for the CBR encoding - different codecs use different defaults. Some don't allow it to be set. Should we stick with the default (slightly biasing things towards codecs that by default sacrifice latency for compression efficiency), or pick a standard default for codecs that let the value be set? If so, something from 3-6 seconds is probably typical for internet streaming use.

Quote
Right. I don't think anyone on his right mind would encode audio to 32kbps at 44.100Hz. I am considering either resampling everything to 22050 (I guess that's what people encoding to this bitrate would do), or letting the encoders choose the appropriate sampling rate.


Well, this is a plausible sample rate for mono HE AAC @ 32 Kbps. I say go for the optimal setting the codec supports. In general, I'm cool with having this be a test of the maximum quality that can be produced within a given compatible bitstream, irrespective of how much encoder tweaking is required to get there.

There will be a slight bias towards codecs/architectures that use a better rate resampler if all our source files are 44.1, but it's not a bias I personally mind .

Quote
I have the demo of QDMC Pro 2, that lasts for 15 days. I'll use it for my test.


Just ask QDesign for a NFR review copy.  I'm sure they've given me a half-dozen over the years, on different platforms, just for the asking.

Dial-up bitrate listening test

Reply #55
Quote
Well, this is a plausible sample rate for mono HE AAC @ 32 Kbps.

Yup yup. Both Nero and CT use 44.1khz with 32kbps parametric stereo (mono+HE-AAC+stereo reconstruction information) by default. This also helps pre-echo.
Juha Laaksonheimo

Dial-up bitrate listening test

Reply #56
Heh, ok, I didn't want to post this, but I sent this as a PM to Roberto:

Quote
Yo dude!

I get 45333bps with my £5 modem, so there... but anyway, I didnt want to write this in the thread, BUT:

I dont know anyone who considers streaming music off the net on dial-up, so this test seems kind of moot if thats the reason for it. As ever, its interesting to see which codec performs best at these rates, but, I find it hard to believe that anyone on dialup would really care about the quality, they know its going to be junk, if they wanted quality, they certainly wouldn't bother.. or care.

I dunno, like I say, its good to see what is the best at 32kbps, but does anyone really care? 48kbps is a bit more interesting, and more of a worthwhile test, in my mind.

Anyway, taraaaaaaaaaaa

Lev-AGE


Afterwards, I thought I may have offended him, as he wanted to catch up with me online.  As it turned out, he loved it and begged me to post it up on the forum.... so -  here I am.

I'd like to just quickly re-iterate my thinkings in the PM (it wasn't phrased brilliantly, my powers of English are not up to that of an author): Any test of any bitrate is interesting, but nobody care's about streaming music on 56k.

Dial-up bitrate listening test

Reply #57
Quote
Heh, ok, I didn't want to post this, but I sent this as a PM to Roberto:

Quote
Yo dude!

I get 45333bps with my £5 modem, so there... but anyway, I didnt want to write this in the thread, BUT:

I dont know anyone who considers streaming music off the net on dial-up, so this test seems kind of moot if thats the reason for it. As ever, its interesting to see which codec performs best at these rates, but, I find it hard to believe that anyone on dialup would really care about the quality, they know its going to be junk, if they wanted quality, they certainly wouldn't bother.. or care.

I dunno, like I say, its good to see what is the best at 32kbps, but does anyone really care? 48kbps is a bit more interesting, and more of a worthwhile test, in my mind.

Anyway, taraaaaaaaaaaa

Lev-AGE


Afterwards, I thought I may have offended him, as he wanted to catch up with me online.  As it turned out, he loved it and begged me to post it up on the forum.... so -  here I am.

I'd like to just quickly re-iterate my thinkings in the PM (it wasn't phrased brilliantly, my powers of English are not up to that of an author): Any test of any bitrate is interesting, but nobody care's about streaming music on 56k.

Quote
I'd like to just quickly re-iterate my thinkings in the PM (it wasn't phrased brilliantly, my powers of English are not up to that of an author): Any test of any bitrate is interesting, but nobody care's about streaming music on 56k.


I think people would absolutely like to listen to music at 32 Kbps, it's just that the quality has historically been below entertainment quality. The point of this test is to see how much better things have gotten with modern codecs, and how they compare with past codecs.

Also, even though 32 Kbps audio as stand alone might not be popular today, 32 Kbps audio in a 200 Kbps video/audio stream for low-broadband users is extremely common.

Dial-up bitrate listening test

Reply #58
OK, so I'm really at a point where I don't know what to do.

Should I resample all files to a common sampling rate before hand, let codecs resample the stream themselves...

@dev0: For some codecs that would work but for, e.g, WMA, it allows you to choose between 44.1, 32 or 22.05k when encoding to WMA Std. at 32kbps. So, the choice in this case is really up to the person encoding.

Quote
Note that at least one codec (WMA9) supports a 2-pass mode, which I suggest should be used.


Unfortunately, it seems WMA 9 only works in two-pass mode down to 64kbps. At least, I couldn't get WMenc9 to do "Bitrate VBR" for bitrates lower than 64.

This is how codecs behave when fed with a 44.100Hz stream:

WMA: Lets you choose what sampling rate
QDesign: Keeps the sampling rate
Real: Keeps the sampling rate. Converts to mono?
Vorbis: Still trying to get it to output 32kbps $%#@!
MP3pro: Seems to only accept 32kbps if in mono  Doesn't seem to resample. If I resample beforehand to 32kHz, it accepts 32kbps.
HE-AAC+PS: According to Ivan, won't resample.

So, any idea?

Dial-up bitrate listening test

Reply #59
I think that if a codec resamples, you should keep its resampling choice.
For codecs that do not resample, perhaps we should do quick test to check if it would be better to resample (I guess in most cases it would be).


Btw, what will be your low anchor? Lowpassed versions or mp3?

Dial-up bitrate listening test

Reply #60
Quote
Perhaps this should be two different tests?  Streaming speech (e.g. news conferences) and streaming music (internet radio) are quite different beasts.  Your choice of codec will very likely depend on the sort of content you're streaming.  Speex would not do that well in a music test but would likely win a speech test.

I agree. But there are people using music codecs to stream voice content. I think there is some use to have a handful of speech samples in this test.

Quote
I think that if a codec resamples, you should keep its resampling choice.


Problem is, as you noticed, nobody resamples automatically. :/

Quote
(I guess in most cases it would be).


I agree.

Quote
Btw, what will be your low anchor? Lowpassed versions or mp3?


I'm still considering it. Considering so many people still stream with MP3 at very low bitrates, I guess it could be used as low anchor, and then use lowpass as high anchor.

Dial-up bitrate listening test

Reply #61
The sample rate and the bit rate should be the same for all codecs.  I realize that this may not show some codecs at their best, but it would keep the playing field level.  To be meaningfull, any test such as this needs well defined and strict 'test conditions.'  As always, contrary opinions are welcomed.  You'll not hurt my feelings.

Dial-up bitrate listening test

Reply #62
Quote
I'm still considering it. Considering so many people still stream with MP3 at very low bitrates, I guess it could be used as low anchor,

I think that it would be a good comparison point.

Dial-up bitrate listening test

Reply #63
For Vorbis use:
oggenc --resample 22050 --managed -b 32
It doesn't seem to accept higher samplerates when working at 32kbps.
Resampling beforehand using ssrc yields a siginificantly higher quality on all samples I tried so far though, so the usage of an external resampler for at least some codecs should be considered (which -again- raises a 'fairness' issue).
"To understand me, you'll have to swallow a world." Or maybe your words.

Dial-up bitrate listening test

Reply #64
Quote
The sample rate and the bit rate should be the same for all codecs.  I realize that this may not show some codecs at their best, but it would keep the playing field level.  To be meaningfull, any test such as this needs well defined and strict 'test conditions.'  As always, contrary opinions are welcomed.  You'll not hurt my feelings.

I disagree. I believe one should take a single codec at a time and ask: " If I had to use this codec with a 32 kbps upper (hard) limit, what would I do to make the most out of it?". That is what any sensible person should, and hopefully would, do in real life. Considering the broad use of CBR, one should probably not keep hopes too high though...

Maybe the result of this test can inspire some potential users to change to the winning codec/setting from a less optimal one. Hence this might heighten the average quality of streaming content. I think all these tests have an informing as well as educational influence. At least I hope they do.

IMHO a test like this is pretty much useless if it artificially tries to "make it fair". The only reason I see for not choosing the optimal encoding for each codec, would be to show the quality of a widely used (less optimal) setting, compared to what other codecs has to offer.


Apart from plain streaming audio, I imagine a MPEG-4 presentation of e.g. a band with video, audio, pictures and so on, might benefit from such low bitrates to keep the overall size of the mp4 file down. Maybe 32kbps is a little too low, but it would be interesting anyway.

All of this is very much IMHO of course. 

Dial-up bitrate listening test

Reply #65
Quote
For Vorbis use:
oggenc --resample 22050 --managed -b 32
It doesn't seem to accept higher samplerates when working at 32kbps.

oggenc --resample 24000 --managed -b 32

does work for me. That additional kHz frequency response may be beneficial.

Dial-up bitrate listening test

Reply #66
Quote
oggenc --resample 24000 --managed -b 32

does work for me. That additional kHz frequency response may be beneficial.

I don't think so. Logically, when taking 44.1 kHz material as a starting point, downsampling by a factor 1.8375 would give more (let's call it) rounding errors than by a plain factor 2.

Dial-up bitrate listening test

Reply #67
Quote
The sample rate and the bit rate should be the same for all codecs.  I realize that this may not show some codecs at their best, but it would keep the playing field level.  To be meaningfull, any test such as this needs well defined and strict 'test conditions.'  As always, contrary opinions are welcomed.  You'll not hurt my feelings.

I think you would cripple codecs completely and make test results completely false. 

The idea of the test is to show quality of commercially available solutions at their best - not to help "equalize" the results among codecs.

For instance, MP3 would definitely need 22.05 kHz for stereo sound at that bit rate - and some other solution naturally encodes at 44.1 kHz (mp3Pro, High Efficiency AAC) - mp3Pro and HE-AAC can't do 22.05 kHz (actually they could but not on that bit rate and it is useless anyway)  - so you're eliminating two codecs immediately.

Fair enough would be to use recommended default settings for each bit-rate, as this is the typical usage scenario - which reflects the biggest number of available streams/content available around.

Dial-up bitrate listening test

Reply #68
I agree with Ivan and upNorth. Resampling where no resampling is needed should definetly be avoided.
But some codecs (Vorbis, MP3, WMA etc.) do require resampling. But is it 'fair' to use a high quality SRC tool/implementation like SSRC? Or should each codec's frontend  handle the resampling?
In my (very brief) tests with Vorbis the encodes from SSRCed files (44.1->22.05) they sounded significantly (~1 on the ITU scale) better than the ones resampled by oggenc.
"To understand me, you'll have to swallow a world." Or maybe your words.

Dial-up bitrate listening test

Reply #69
Quote
Fair enough would be to use recommended default settings for each bit-rate, as this is the typical usage scenario - which reflects the biggest number of available streams/content available around.

I agree completely
Vital papers will demonstrate their vitality by spontaneously moving from where you left them to where you can't find them.

Dial-up bitrate listening test

Reply #70
Quote
I don't think so. Logically, when taking 44.1 kHz material as a starting point, downsampling by a factor 1.8375 would give more (let's call it) rounding errors than by a plain factor 2.

That´s true. However, I assume that really good resamplers don`t introduce severe artifacts even with "odd" resampling-factors (example: 44.1 <-> 48 kHz).

Dial-up bitrate listening test

Reply #71
Did a quick test using SSRC and Vorbis:

Code: [Select]
ABC/HR Version 1.0, 6 May 2004
Testname: Vorbis 32kbps resampling: giveuptheghost-sincealways

1R = dec\giveuptheghost-sincealways.sample18sec.ssrc22.wav
2L = dec\giveuptheghost-sincealways.sample18sec.oggenc24.wav
3R = dec\giveuptheghost-sincealways.sample18sec.ssrc24.wav
4L = dec\giveuptheghost-sincealways.sample18sec.oggenc22.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: dec\giveuptheghost-sincealways.sample18sec.ssrc22.wav
1R Rating: 2.5
1R Comment:
---------------------------------------
2L File: dec\giveuptheghost-sincealways.sample18sec.oggenc24.wav
2L Rating: 1.8
2L Comment:
---------------------------------------
3R File: dec\giveuptheghost-sincealways.sample18sec.ssrc24.wav
3R Rating: 2.3
3R Comment:
---------------------------------------
4L File: dec\giveuptheghost-sincealways.sample18sec.oggenc22.wav
4L Rating: 1.4
4L Comment:
---------------------------------------
ABX Results:


Seems like the samplerate used is a lot less important than the SRC tool used.
"To understand me, you'll have to swallow a world." Or maybe your words.

Dial-up bitrate listening test

Reply #72
And another quick one:
Code: [Select]
ABC/HR Version 1.0, 6 May 2004
Testname: Vorbis 32kbps resampling: eaves-teenagelifesentence

1R = dec\eaves-teenagelifesentence.sample17sec.ssrc22.wav
2R = dec\eaves-teenagelifesentence.sample17sec.oggenc22.wav
3L = dec\eaves-teenagelifesentence.sample17sec.ssrc24.wav
4R = dec\eaves-teenagelifesentence.sample17sec.oggenc24.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: dec\eaves-teenagelifesentence.sample17sec.ssrc22.wav
1R Rating: 2.1
1R Comment:
---------------------------------------
2R File: dec\eaves-teenagelifesentence.sample17sec.oggenc22.wav
2R Rating: 1.2
2R Comment:
---------------------------------------
3L File: dec\eaves-teenagelifesentence.sample17sec.ssrc24.wav
3L Rating: 1.9
3L Comment:
---------------------------------------
4R File: dec\eaves-teenagelifesentence.sample17sec.oggenc24.wav
4R Rating: 1.5
4R Comment:
---------------------------------------
ABX Results:


The differences between the ssrced 22050 and 24000 versions are sublte, but I perceive the 22.05khz one as less artifacted. Maybe somebody could verify that (I used fb2k's SSRC Resampler in 'Slow Mode' and foo_clienc).
cmd.line: oggenc --managed -b32 [--resample #]
Sample: eaves-teenagelifesentence.sample17sec.flac

I used the official 1.0.1 binary from Vorbis.com.
"To understand me, you'll have to swallow a world." Or maybe your words.

Dial-up bitrate listening test

Reply #73
Given that oggenc uses (last time I looked) bandlimited sinc interpolation, which is a fine algorithm, I think this is a bug or bad tuning of the parameters.

Dial-up bitrate listening test

Reply #74
Would lowpassing be considered fair?  It seems as though I can convince AoTuV down to 32kb and below using the lowpass settings in OggDropXPd.

I would agree with Lev that a 40 or 48kbs test would be more interesting.  I admit I had a top rate ISP, but I averaged just under the 50000bps mark most weekends back in the dark ages when I had a modem
< w o g o n e . c o m / l o l >

 
SimplePortal 1.0.0 RC1 © 2008-2019