HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Sebastian Mares on 2005-03-22 14:54:53

Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-22 14:54:53
Greetings!

I am planning to conduce a multi-format 64 kbps listening test under Roberto's supervision. The exact date for the beginning of the test is not set, since I am waiting for Apple to release their HE-AAC encoder sometime in April (according to the rumours).

So far, here is what I was thinking about. Feedback and suggestions are welcome.

Codecs:Nero and Apple HE-AAC will be included for sure.
What I don't really know is which Vorbis version to use - either AoTuV3 or 1.1. Same applies to WMA - either Standard due to compatibility or Professional for quality. Last but not least, should mp3PRO be tested again as reference (since there aren't any changes from version used in Roberto's last test)?

As you can see, there will be 6 or 7 (depending on mp3PRO) codecs.

Samples:I am planning to use 18 samples - 9 from Roberto's old listening test and 9 additional ones. What do you guys suggest?

The results will be calculated in the same way Roberto's results were calculated.

So, what are you thoughts? 

Regards,
Sebastian

Edit: Spelling and typo in the number of samples planned to use.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-22 14:58:24
As difficult sample, I would suggest Kraftwerk to replace Waiting (which was used in the last test and is not missing).
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-22 15:15:28
Good initiative. I will look forward to the test.

I suggest to leave mp3PRO out, since nobody use it.

As for WMA I would recommend using WMA std. It is more widespread than PRO and is what most people refer to as WMA. People already have a hard time distinguishing between std. and PRO so choosing PRO is likely to lead to misinterpretation of the test results. As an alternative one could consider including both WMA std and WMA PRO, anyway my vote on WMA std.

What about using LAME 3.97a8 as low anchor?
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-22 15:22:08
I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.
Title: 64 kbps listening test 2005
Post by: PaleGreen on 2005-03-22 15:24:13
I'd like to see both WMA Standard & Pro included.
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-22 15:26:11
Quote
I'd like to see both WMA Standard & Pro included.
[a href="index.php?act=findpost&pid=284530"][{POST_SNAPBACK}][/a]

I wouldn't include wma pro because no internet radio uses it and it would only confuse people.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 15:32:24
Quote
Quote
I'd like to see both WMA Standard & Pro included.
[a href="index.php?act=findpost&pid=284530"][{POST_SNAPBACK}][/a]

I wouldn't include wma pro because no internet radio uses it and it would only confuse people.
[a href="index.php?act=findpost&pid=284533"][{POST_SNAPBACK}][/a]


I wouldn't include WMA standard because it has already been tested and, as far as I know, it hasn't changed much, if any, since
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-22 15:34:16
Quote
I wouldn't include WMA standard because it has already been tested and, as far as I know, it hasn't changed much, if any, since
[a href="index.php?act=findpost&pid=284537"][{POST_SNAPBACK}][/a]


May I then suggest we use it as reference?
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 15:35:39
Quote
May I then suggest we use it as reference?
[a href="index.php?act=findpost&pid=284539"][{POST_SNAPBACK}][/a]


FhG 64kbps (low anchor) didn't change much either, AFAIK.
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-22 15:37:55
Quote
Quote
May I then suggest we use it as reference?
[a href="index.php?act=findpost&pid=284539"][{POST_SNAPBACK}][/a]


FhG 64kbps (low anchor) didn't change much either, AFAIK.
[a href="index.php?act=findpost&pid=284541"][{POST_SNAPBACK}][/a]

In that case we can leave out WMA completely
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-22 15:39:28
Perhaps Lame could be used as low anchor?
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-22 15:48:44
Quote
Quote
Quote
May I then suggest we use it as reference?
[a href="index.php?act=findpost&pid=284539"][{POST_SNAPBACK}][/a]


FhG 64kbps (low anchor) didn't change much either, AFAIK.
[a href="index.php?act=findpost&pid=284541"][{POST_SNAPBACK}][/a]

In that case we can leave out WMA completely
[a href="index.php?act=findpost&pid=284544"][{POST_SNAPBACK}][/a]

If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-22 15:49:23
Quote
I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=284529")

I second that suggestion. Atrac3plus wasn't tested yet (only atrac3@132 kbps). It could be very interesting to see how will perform this format, which correspond (according to optimistic estimations made by Sony) to atrac@132 and to atrac@292 kbps:

[a href="http://www.mdfr.com/faqs/atrac_roadmap_small.jpg]http://www.mdfr.com/faqs/atrac_roadmap_small.jpg[/url]

But it's necessary to upload them as lossless files (flac or something else).
Title: 64 kbps listening test 2005
Post by: Thoifl on 2005-03-22 15:53:54
Quote
What I don't really know is which Vorbis version to use - either AoTuV3

I would take AoTuV3, since it should be tweaked for low bitrates and is little testet.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 15:55:36
Quote
If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=284549")


Good point. But then again, most slashdotters can go .... themselves, for all that I really care...

Quote
Quote
I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.
[a href="index.php?act=findpost&pid=284529"][{POST_SNAPBACK}][/a]

I second that suggestion. Atrac3plus wasn't tested yet (only atrac3@132 kbps). It could be very interesting to see how will perform this format, which correspond (according to optimistic estimations made by Sony) to atrac@132 and to atrac@292 kbps:

[a href="http://www.mdfr.com/faqs/atrac_roadmap_small.jpg]http://www.mdfr.com/faqs/atrac_roadmap_small.jpg[/url]

But it's necessary to upload them as lossless files (flac or something else).
[a href="index.php?act=findpost&pid=284551"][{POST_SNAPBACK}][/a]


Well, same would have to happen with MP3pro, that is another alternative for 7th codec.

So, I guess the discussion is about of wether Atrac3+ or MP3pro should be tested.

I already feel sorry for Sebastian if people choose Atrac3+ 
Title: 64 kbps listening test 2005
Post by: DreamTactix291 on 2005-03-22 15:56:40
Quote
What I don't really know is which Vorbis version to use - either AoTuV3 or 1.1. Same applies to WMA - either Standard due to compatibility or Professional for quality.
Since aoTuV b2 (which 1.1 is essentially) has been fairly well tested it might be interesting to see how the relative newcomer aoTuV b3 fares IMO.  Even worse possible scenario of regression I can't see it being really that much worse than 1.1 (which I doubt it is worse but only testing can prove either way).

As far as WMA I'd go standard.  Sure it's not as high quality as pro but I've never even come across a WMA Pro file I didn't make myself in the wild.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 15:56:49
Quote
Perhaps Lame could be used as low anchor?
[a href="index.php?act=findpost&pid=284545"][{POST_SNAPBACK}][/a]


Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?
Title: 64 kbps listening test 2005
Post by: sTisTi on 2005-03-22 15:58:32
Quote
If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?
[a href="index.php?act=findpost&pid=284549"][{POST_SNAPBACK}][/a]

WMA std should definitely be included. I'm sure it will fail badly against Vorbis and HE AAC, so this test will be one more evidence against Microsoft's ridiculous claims and demonstrate the low quality of WMA compared to other modern codecs.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-22 15:59:28
atrac3plus could be tested over mp3pro, for some reasons:
- mp3pro was tested previously, and hasn't progress since.
- atrac3plus wasn't tested
- there are more device playing atrac3plus than mp3pro (all modern HiMD units, many CD players, Jukebox and Sony flash USB devices).
- atrac3plus have more supporters (I'd use another word to describe them...) than mp3pro, which has progressively lost importance with time and competition.
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-22 16:00:23
Quote
I already feel sorry for Sebastian if people choose Atrac3+ 
[a href="index.php?act=findpost&pid=284557"][{POST_SNAPBACK}][/a]

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-22 16:01:09
Quote
Quote
Perhaps Lame could be used as low anchor?
[a href="index.php?act=findpost&pid=284545"][{POST_SNAPBACK}][/a]


Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?
[a href="index.php?act=findpost&pid=284559"][{POST_SNAPBACK}][/a]

MP3 at 64 kbps must be a low anchor. It doesn't matter to use lame or fhg. All we need is a encoder which sound worse than all other competitors.
Testing lame at 128 and at 64 kbps is IMO a good idea.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-22 16:03:25
Quote
Quote
I already feel sorry for Sebastian if people choose Atrac3+ 
[a href="index.php?act=findpost&pid=284557"][{POST_SNAPBACK}][/a]

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.
[a href="index.php?act=findpost&pid=284564"][{POST_SNAPBACK}][/a]

I must warn you that basic MD lovers won't consider results as significant unless you've made encoding directly from device, and not from SonicStage. Hardware encoding is often considered as better than software encodings (placebo or not: I don't know — blind listening tests were never done to my knowledge).
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 16:03:27
Quote
If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.
[a href="index.php?act=findpost&pid=284564"][{POST_SNAPBACK}][/a]


Ah, that's good news, indeed. But he'll still have to install SonicStage. That flaming piece of poo ruined my win2000 installation.

Quote
I must warn you that basic MD lovers won't consider results as significant unless you've made encoding directly from device, and not from SonicStage. Hardware encoding is often considered as better than software encodings (placebo or not: I don't know — blind listening tests were never done to my knowledge).


That is certifiable bollocks. Hardware encoding has no reason to be better than software encoding, quite the opposite: hardware encoding faces limitations related to battery consumption, real-time processing on underpowered DSPs, integer-only processing, etc, etc.
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-22 16:06:49
Quote
Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?

I am not sure if the difference would be that big between FhG and 3.97b.

Anyway, whatever the choosen anchors, I'd like the graphs of results mentionning "low anchor" and "high anchor" instead of the codec name.
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-22 16:09:42
Quote
Quote
If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.
[a href="index.php?act=findpost&pid=284564"][{POST_SNAPBACK}][/a]


Ah, that's good news, indeed. But he'll still have to install SonicStage. That flaming piece of poo ruined my win2000 installation.

That is certifiable bollocks. Hardware encoding has no reason to be better than software encoding, quite the opposite: hardware encoding faces limitations related to battery consumption, real-time processing on underpowered DSPs, integer-only processing, etc, etc.
[a href="index.php?act=findpost&pid=284568"][{POST_SNAPBACK}][/a]


I have a working SS installation if that matters.

There are some hardware atrac3 vs SS atrac3 abx threads somewhere.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 16:11:31
Quote
I have a working SS installation if that matters.
[a href="index.php?act=findpost&pid=284572"][{POST_SNAPBACK}][/a]

Oh, yes, after I reinstalled my system from scratch it installed without problems. But it is (was?) a terrible buggy program, neverthless.

I think it's somewhat related to the security and encryption crap Sony installs everywhere.
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-22 16:14:02
Might be usefull to do the SonicStage installation under VMWare or something similar.
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-22 16:15:09
I can prepare the samples if required.
Title: 64 kbps listening test 2005
Post by: bond on 2005-03-22 16:23:24
what are the up- and downsides of the differen vorbis version, which could be used?

Quote
I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.
[a href="index.php?act=findpost&pid=284529"][{POST_SNAPBACK}][/a]

i would also love to see atrac3+ to be tested

and if there is space for another codec, apple lc-aac would be interesting too
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 16:33:03
Quote
and if there is space for another codec, apple lc-aac would be interesting too
[a href="index.php?act=findpost&pid=284579"][{POST_SNAPBACK}][/a]


I don't think so. Who uses LC AAC at such bitrates, if one can use HE AAC with much better quality?
Title: 64 kbps listening test 2005
Post by: bond on 2005-03-22 16:39:15
Quote
I don't think so. Who uses LC AAC at such bitrates, if one can use HE AAC with much better quality?
[a href="index.php?act=findpost&pid=284582"][{POST_SNAPBACK}][/a]

it would show the quality difference between he-aac and lc-aac at such a bitrate

well, just realised that we already tested that in your 64kbps test 
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 16:42:00
Quote
well, just realised that we already tested that in your 64kbps test  
[a href="index.php?act=findpost&pid=284584"][{POST_SNAPBACK}][/a]


Exactly

I don't think the gap got much different since.
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-22 16:42:04
Quote
Who uses LC AAC at such bitrates, if one can use HE AAC with much better quality?

Nokia users who did not replaced the tinny memory card they got with the phone?
Title: 64 kbps listening test 2005
Post by: Eli on 2005-03-22 16:51:49
I would be interested in seeing MPC in the test. I know its not tuned for these bit rates but it did suprisingly well at 128 and Im hoping that the new team does some tuning at low bit rates after work on 7.5 or 8 is well underway. While it currently has littly hardware support, its low CPU overhead for decoding could help extend battery life on flash players where decoding is the major drain on batteries.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 16:56:41
Quote
I would be interested in seeing MPC in the test. I know its not tuned for these bit rates but it did suprisingly well at 128 and Im hoping that the new team does some tuning at low bit rates after work on 7.5 or 8 is well underway. While it currently has littly hardware support, its low CPU overhead for decoding could help extend battery life on flash players where decoding is the major drain on batteries.
[a href="index.php?act=findpost&pid=284587"][{POST_SNAPBACK}][/a]


As you said yourself, it's badly tuned at these bitrates, so I don't see what is the point of such test currently.

Maybe if the dev team really tunes low bitrates after SV8 is released, then that might warrant a new test.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-22 17:10:02
OK... So far, MusePack and LC AAC will definitely not be included in the test. MusePack isn't the encoder to test at 64 kbps, the same applies to LC AAC.

MusePack's emphasis is on quality and transparency and was never designed for such low bitrates. People using MusePack will at least encode at 128 kbps. As Roberto already stated, if things change after SV8, MusePack should be included in a 64 kbps listening test.

As for LC AAC, I doubt that there are any major improvements since the last 64 kbps test (where Apple's implementation was used) and for such a low bitrate, it is logical to use HE-AAC. Also, I think it's a good idea to have a direct comparision between Nero HE-AAC and Apple HE-AAC and that makes AAC appear twice in the test.
Title: 64 kbps listening test 2005
Post by: mickywicky on 2005-03-22 17:29:37
Quote
Quote
Quote
If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.
[a href="index.php?act=findpost&pid=284564"][{POST_SNAPBACK}][/a]


Ah, that's good news, indeed. But he'll still have to install SonicStage. That flaming piece of poo ruined my win2000 installation.

That is certifiable bollocks. Hardware encoding has no reason to be better than software encoding, quite the opposite: hardware encoding faces limitations related to battery consumption, real-time processing on underpowered DSPs, integer-only processing, etc, etc.
[a href="index.php?act=findpost&pid=284568"][{POST_SNAPBACK}][/a]


I have a working SS installation if that matters.

There are some hardware atrac3 vs SS atrac3 abx threads somewhere.
[a href="index.php?act=findpost&pid=284572"][{POST_SNAPBACK}][/a]


Hmm yeah but either way, do those MD fanboys conclude all recording/copying/converting shoudl be done by the device rather than a PC??
Don't get me wrong, I have an MD player (old school model from 1996 or so) and I quite like it. It was very good back then too. And I did most of the copying using the device itself. Two words: *big hassle*.
I don't think most MD users would do that if they were given the alternative.
So, even if we asume that hardware encoding is better, that we really should use software encoding (there are hardware MP3 encoders too, after all) - at least until  ABX tests prove there is a noticeable difference.
Title: 64 kbps listening test 2005
Post by: SirGrey on 2005-03-22 18:40:01
Vote for atrac3+ and wma std.
WMA pro would be more interesting but less useful
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-22 18:53:23
Vote for Atrac3+

I would like to suggest the following sample for the listening test:

Supertramp - School (http://www.hydrogenaudio.org/forums/index.php?showtopic=32608&view=findpost&p=284616)

The 19s sample I have uploaded contains acoustic guitars, piano, cymbals and bass.

I have good experience with this sample in revealing subtle difference between encodings.

Let me hear what you think about the sample.

If desired I can upload a 30s version of the sample.
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-22 19:01:30
Sample proposition: the beginning of "Money" by Pink Floyd.

I do not have it available, but I am sure some Pink Floyd fan could upload it.
Basically it is background music with coins and cash machine sounds. I think that the coins coud be interesting.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-22 19:30:00
Quote
Sample proposition: the beginning of "Money" by Pink Floyd.

I do not have it available, but I am sure some Pink Floyd fan could upload it.
Basically it is background music with coins and cash machine sounds. I think that the coins coud be interesting.
[a href="index.php?act=findpost&pid=284619"][{POST_SNAPBACK}][/a]


Yeah, I can... The clipped remastered version or the "old" one?
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 22:04:26
Quote
Yeah, I can... The clipped remastered version or the "old" one?
[a href="index.php?act=findpost&pid=284631"][{POST_SNAPBACK}][/a]


SHINE ON 
Title: 64 kbps listening test 2005
Post by: Gecko on 2005-03-22 22:23:02
I think it would be interesting to use the SSE optimized aotuvb3/archer version of Ogg Vorbis.

A direct comparison to the non SSE version would be best, but since differences are expected to be small anyway, this is an opportunity to test it on multiple samples. If any problems come up, it could be tested against the regular version after the test.

Of course if you want to do the encoding at the user end, you'll have to distribute separate versions of the sample packages for people with non SSE capable CPUs.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-22 22:40:16
The problem in using Archer and aotuv3, IMO, is that the results won't be as valid as we would like them to be. If vorbis performs badly, we'll not know if it is becauseof a format weakness or just because the tweaking broke something. After all, these are beta versions - not for production environments.

Now, IF the Vorbis fellas around here conduce a little test before the real thing, like happened in my multiformat @ 128 test, then the choice of what version to use will be supported by more reasons than just "let's give it a try".
Title: 64 kbps listening test 2005
Post by: Eli on 2005-03-22 22:57:41
Quote
Quote
I would be interested in seeing MPC in the test. I know its not tuned for these bit rates but it did suprisingly well at 128 and Im hoping that the new team does some tuning at low bit rates after work on 7.5 or 8 is well underway. While it currently has littly hardware support, its low CPU overhead for decoding could help extend battery life on flash players where decoding is the major drain on batteries.
[a href="index.php?act=findpost&pid=284587"][{POST_SNAPBACK}][/a]


As you said yourself, it's badly tuned at these bitrates, so I don't see what is the point of such test currently.

Maybe if the dev team really tunes low bitrates after SV8 is released, then that might warrant a new test.
[a href="index.php?act=findpost&pid=284588"][{POST_SNAPBACK}][/a]


Well, it wasnt really tuned for 128 either was it and it is tied for first
Title: 64 kbps listening test 2005
Post by: S_O on 2005-03-22 23:01:08
Nero HE-AAC and Apple HE-AAC (has it already been released??) are tested, what about FhG / Coding Technologies HE-AAC? It is the only HE-AAC implementation available for free (in Helix Producer). I think it should be included.
Title: 64 kbps listening test 2005
Post by: Gecko on 2005-03-22 23:06:15
Quote
The problem in using Archer and aotuv3, IMO, is that the results won't be as valid as we would like them to be. If vorbis performs badly, we'll not know if it is becauseof a format weakness or just because the tweaking broke something. After all, these are beta versions - not for production environments.

Now, IF the Vorbis fellas around here conduce a little test before the real thing, like happened in my multiformat @ 128 test, then the choice of what version to use will be supported by more reasons than just "let's give it a try".
[a href="index.php?act=findpost&pid=284673"][{POST_SNAPBACK}][/a]


You're right. But I believe the differences regular/SSE would be minimal and this may be a chance to detect flaws (I don't expect any that result from the optimizations). This could be verified after the test by checking the problem cases against the regular version.

But if you need to run a separate test anyway then you might as well do it beforehand.  Maybe there are serious flaws which could ruin Vorbis' reputation!

The archer tunings are at RC3 right now and I think we will see a final soon. Ah, but of course the archer tunings will still remain beta, my bad.

edit: added quote
Title: 64 kbps listening test 2005
Post by: kurtnoise on 2005-03-22 23:19:29
Quote
Nero HE-AAC and Apple HE-AAC (has it already been released??) are tested, what about FhG / Coding Technologies HE-AAC? It is the only HE-AAC implementation available for free (in Helix Producer). I think it should be included.
[a href="index.php?act=findpost&pid=284681"][{POST_SNAPBACK}][/a]

The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests. There is only new thing : introduction of new target bitrates (32 & 48 kbps).
Title: 64 kbps listening test 2005
Post by: Gecko on 2005-03-22 23:36:39
Quote
Well, it wasnt really tuned for 128 either was it and it is tied for first
[a href="index.php?act=findpost&pid=284679"][{POST_SNAPBACK}][/a]

I just tried it on one sample at quality 1 and 2. MPC is not a contender at this bitrate. Sounds like you are underwater, bubbles everywhere - great effect. In comparison, Lame encoded at 64k is hifi. No ABX required.
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-22 23:50:39
I have two more candiates for listening test samples...

They are both exceptionally well mastered using the Blumlein technique where the entire sound picture is captured at a single point, carefully chosen to achieve the optimum balance between direct sound and reverberation.

This makes unnecessary all forms of electronic correction and manipulation and the music signal travels by the absolutely shortest possible path between the microphone and the final CD.

The first sample is recorded in an old stone church:

Vaquero (http://www.hydrogenaudio.org/forums/index.php?showtopic=32616&view=findpost&p=284684)

The second sample is recorded in a jazz club:

Four Brothers (http://www.hydrogenaudio.org/forums/index.php?showtopic=32618&view=findpost&p=284691)

What's unussual with these recordings is their depth of image, timbre and dynamics.

For more info on the recording:

Opus3 Recordings (http://www.opus3records.com/)
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-22 23:53:15
Yet another sample suggestion:

Dire Straits - Iron Hand (http://www.hydrogenaudio.org/forums/index.php?showtopic=32609&view=findpost&p=284624)
Title: 64 kbps listening test 2005
Post by: S_O on 2005-03-23 00:40:21
Quote
The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests. There is only new thing : introduction of new target bitrates (32 & 48 kbps).
Huh? I cannot remember a listening test with Helix HE-AAC being tested. There were a 64kbps test with RealAudio Cook once, but with Helix HE-AAC? I cannot remember.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-23 00:53:57
Quote
But if you need to run a separate test anyway then you might as well do it beforehand.  Maybe there are serious flaws which could ruin Vorbis' reputation![a href="index.php?act=findpost&pid=284685"][{POST_SNAPBACK}][/a]


Yes, I believe it would be very bad if Vorbis reputation got ruined thanks to a badly tuned encoder.

Also, this test is only supposed to start after Apple releases iTunes 5 / QuickTime 7. That should give people plenty of time to conduce paralel tests, if there is really any interest...

Quote
The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests.
[a href="index.php?act=findpost&pid=284689"][{POST_SNAPBACK}][/a]


It wasn't even tested back then :B
Title: 64 kbps listening test 2005
Post by: Acid Orange Juice on 2005-03-23 06:28:34
IMO this sample could be very useful for this test. In particular he cause a lot of phasing problems with lame --preset cbr 128 and Vorbis low bitrates.

Download HERE (http://www.hydrogenaudio.org/forums/index.php?showtopic=32625)
Title: 64 kbps listening test 2005
Post by: znode on 2005-03-23 06:39:07
Quote
But if you need to run a separate test anyway then you might as well do it beforehand.  Maybe there are serious flaws which could ruin Vorbis' reputation!
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=284685")


Yeah, misinterpretation of results is quite a problem. I've seen countless people claim vorbis to be always better than any other codec, in all cases, because of those [a href="http://www.rjamorim.com/test/multiformat128/results.html]http://www.rjamorim.com/test/multiformat128/results.html[/url] results at 128kbps.
Title: 64 kbps listening test 2005
Post by: HotshotGG on 2005-03-23 06:51:33
Quote
If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?


  I don't know what to make of that site so I don't ask. Informative yes. Always on par with issues at hand no. Rating system nah    For a site full of edcuated nerds they sure don't act like it sometimes  .

Quote
IMO this sample could be very useful for this test. In particular he cause a lot of phasing problems with lame --preset cbr 128 and Vorbis low bitrates.


channel coupling related maybe? hmm I will have to test that out myself after.

Quote
Yeah, misinterpretation of results is quite a problem. I've seen countless people claim vorbis to be always better than any other codec, in all cases, because of those tests


Well it would be great two see both Nero HE-AAC and Vorbis tied for first ;-D.  A streaming listening test was definitely going to be needed though eventually.
Title: 64 kbps listening test 2005
Post by: kurtnoise on 2005-03-23 08:39:01
Quote
Quote
The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests.
[a href="index.php?act=findpost&pid=284689"][{POST_SNAPBACK}][/a]


It wasn't even tested back then :B
[a href="index.php?act=findpost&pid=284708"][{POST_SNAPBACK}][/a]

ouups, sorry for the confusion...  I was tired last night.



So, He-AAC encoder from Producer could be interesting though...
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-23 09:37:51
Quote
Sample proposition: the beginning of "Money" by Pink Floyd.

I do not have it available, but I am sure some Pink Floyd fan could upload it.
Basically it is background music with coins and cash machine sounds. I think that the coins coud be interesting.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=284619")


[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=32628]http://www.hydrogenaudio.org/forums/index....showtopic=32628[/url]

Anyone interested in Time?
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-23 12:18:21
Why not taking also 3gpp's CT HE + PS AAC?
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-23 13:09:02
Quote
Why not taking also 3gpp's CT HE + PS AAC?
[a href="index.php?act=findpost&pid=284794"][{POST_SNAPBACK}][/a]


Wouldn't that be a HE-AAC 64 kbps listening test then?
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-23 13:11:17
All those propositions are transforming this into a 64kbps MPEG test...
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-23 13:33:09
Anyways...

Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.

As for the discussion about mp3PRO or ATRAC3+, I think that I will use ATRAC3+ since it is more wide-spread than mp3PRO and since mp3PRO didn't change since the last test.

Still not sure what to do with WMA - either Standard or Professional. I, personally, would choose Standard since it's the format you find in music stores and it's also what most people use so it's compatible with their players.
Title: 64 kbps listening test 2005
Post by: moi on 2005-03-23 13:58:14
I think you really need to include WMA Standard, as it is probably the most common format encoded at 64kbps, people will want to see how it compares with others in the test. Of course the newest, WMA 9.1, which is installed with WMP10.

You might also wish to include WMA Pro, to see how it compares at that bit rate to standard.

I really don't see why LAME at 128kbps should be  included in a 64kbps listening test, as it was the other time. Probably has something to do with the claim that WMA at 64kbps sounds "as good as" MP3 at 128kbps. I don't think many here believe that claim. In any case, IMO, a 64kbps listening test should only include music encoded at 64kbps. It is misleading to encode 128kbps in one format, and 64kbps in all the others. Everything in a 64kbps listening test should be encoded at 64kbps.

I think MP3Pro should be included, as it did very well on some 64kbps tests in the past. Not supported by many players, but by some, I think it should be included, whether or not it changed.
Title: 64 kbps listening test 2005
Post by: Redmond on 2005-03-23 14:14:55
Quote
You might also wish to include WMA Pro, to see how it compares at that bit rate to standard.
[a href="index.php?act=findpost&pid=284822"][{POST_SNAPBACK}][/a]


"Publicly available" WMA Pro encoders do not go down to 64Kb/s stereo.
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-23 14:22:21
Quote
I really don't see why LAME at 128kbps should be  included in a 64kbps listening test, as it was the other time. Probably has something to do with the claim that WMA at 64kbps sounds "as good as" MP3 at 128kbps. I don't think many here believe that claim. In any case, IMO, a 64kbps listening test should only include music encoded at 64kbps. It is misleading to encode 128kbps in one format, and 64kbps in all the others. Everything in a 64kbps listening test should be encoded at 64kbps.
[a href="index.php?act=findpost&pid=284822"][{POST_SNAPBACK}][/a]

A credible listening test should have a low and high anchor.
Title: 64 kbps listening test 2005
Post by: Aoyumi on 2005-03-23 14:32:19
Quote
Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.
[a href="index.php?act=findpost&pid=284811"][{POST_SNAPBACK}][/a]

When the test is performed, I need to submit the newest experiment version.
It is more clearly than aoTuV beta3 good with some samples (setting to the low bit rate).
Title: 64 kbps listening test 2005
Post by: PoisonDan on 2005-03-23 14:51:22
Quote
Quote
Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.
[a href="index.php?act=findpost&pid=284811"][{POST_SNAPBACK}][/a]

When the test is performed, I need to submit the newest experiment version.
It is more clearly than aoTuV beta3 good with some samples (setting to the low bit rate).
[a href="index.php?act=findpost&pid=284836"][{POST_SNAPBACK}][/a]

Were you planning on releasing a new version soon anyway? I wouldn't want you to feel rushed to get a version out the door just to be in time for this listening test...

At this moment, I'm extremely busy with real-life and work-related stuff, but next week I'll probably have some time to do a few Vorbis listening tests...
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-23 15:05:45
Well, take your time, since the test will start after Apple releases their HE-AAC encoder.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-23 15:33:38
Quote
"Publicly available" WMA Pro encoders do not go down to 64Kb/s stereo.
[a href="index.php?act=findpost&pid=284828"][{POST_SNAPBACK}][/a]


If I remember correctly, the publicly available encoder stays around 64kbps if you choose the lowest VBR setting (10).
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-23 22:15:03
Regarding the low anchor, do you think LAME or FhG should be used at 64 kbps?
Title: 64 kbps listening test 2005
Post by: sehested on 2005-03-23 22:57:05
Quote
Regarding the low anchor, do you think LAME or FhG should be used at 64 kbps?
[a href="index.php?act=findpost&pid=284976"][{POST_SNAPBACK}][/a]

I would like to see LAME as low anchor.

That would also demonstrate the improvements of the other codecs compared to the best MP3 encoder available.
Title: 64 kbps listening test 2005
Post by: ff123 on 2005-03-23 23:28:23
Quote
Quote
Regarding the low anchor, do you think LAME or FhG should be used at 64 kbps?
[a href="index.php?act=findpost&pid=284976"][{POST_SNAPBACK}][/a]

I would like to see LAME as low anchor.

That would also demonstrate the improvements of the other codecs compared to the best MP3 encoder available.
[a href="index.php?act=findpost&pid=284995"][{POST_SNAPBACK}][/a]


This presumes that lame is the best mp3 encoder at 64 kbps, which isn't a given.  The question of which mp3 encoder to use as a low anchor probably deserves a pre-test if people are interested in using the best-sounding one.

ff123
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-24 07:50:34
Two suggestions:

• I think that the current scale isn't really really suited to a 64 kbps and the expected distortions.
Artifacts “perceptible but not annoying” (4.0) are maybe not very common at this bitrate. And few encoders are able to reproduce (in my opinion) a sound with only "slightly annoying" (3.0) difference at 64 kbps. It's possible to change the corresponding scale with schnofler's abc/hr, and I wonder if it's not worth to think about it. If I remember correctly, the average notation I gave to most encoders during the 32 kbps was inferior to 1.5/5 


• A also suggest to reduce the length of all samples. I'm the first one to provide 30 seconds samples, but I perfectly know the drawbacks. Some people will rate one encoder on a short range located at the beginning, some other will evaluate another part (totally different from the first one), etc... Finally it's exactly if people have evaluate different samples. I suggest to limit the duration to 6 or 7 seconds.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-24 12:09:43
Quote
• A also suggest to reduce the length of all samples. I'm the first one to provide 30 seconds samples, but I perfectly know the drawbacks. Some people will rate one encoder on a short range located at the beginning, some other will evaluate another part (totally different from the first one), etc... Finally it's exactly if people have evaluate different samples. I suggest to limit the duration to 6 or 7 seconds.
[a href="index.php?act=findpost&pid=285081"][{POST_SNAPBACK}][/a]


I understand what you mean, but why not let testers decide which portion they want to ABX?
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-24 12:30:10
I also think that 30s might be too long.
Perhaps 6s is too short, but I think that 15s should be enough.

Letting testers deciding which portion to use is perhaps reducing "usefullness" of results. It is like they are testing different samples, but it makes correlation between results for the same sample harder.

If a sample has some quite different parts in a 30s set, then it could be intersting to split it into 2 samples, making interpretation of results easier.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-24 12:43:02
There's nothing wrong with that.
Now imagine a heterogeneous sample: beginning (first seconds) is quiet, whereas the following part is very different.
Suppose that most people will only rate the file on a small part (4-5 seconds). Suppose then that most people will favour the first thing they hear (beginning). Most, but not all... HE-AAC is very good on the beginning, but fail on the second part. Will the overall notation be representative? Isn't it better to provide to all people a short sample only?

If people will evaluate different part of one sample, it could be considered as evaluating two (or more) different samples (at least if a long sample propose some variety). But correct me if I'm wrong, the purpose of a collective test is to obtain results from different subjectivity evaluating the same thing (same sample, same listening material). We can't do that: people don't have the same hardware. But we can at least make one thing, and be sure that all people are listening to the same musical informations.


Is it clear?


EDIT: Gabriel was faster, and explained it better 
Title: 64 kbps listening test 2005
Post by: Aoyumi on 2005-03-24 12:52:36
Quote
Quote
Quote
Regarding Vorbis, I would love if some Vorbis users could start a small listening test and compare AoTuV3 and Xiph 1.1 so that the better version will be used in this test.
[a href="index.php?act=findpost&pid=284811"][{POST_SNAPBACK}][/a]

When the test is performed, I need to submit the newest experiment version.
It is more clearly than aoTuV beta3 good with some samples (setting to the low bit rate).
[a href="index.php?act=findpost&pid=284836"][{POST_SNAPBACK}][/a]

Were you planning on releasing a new version soon anyway? I wouldn't want you to feel rushed to get a version out the door just to be in time for this listening test...

At this moment, I'm extremely busy with real-life and work-related stuff, but next week I'll probably have some time to do a few Vorbis listening tests...
[a href="index.php?act=findpost&pid=284838"][{POST_SNAPBACK}][/a]

I am able to exhibit the version corresponding to the range of 64kbps at least.
I want it to be tested.
Title: 64 kbps listening test 2005
Post by: Music Mixer on 2005-03-24 12:56:48
One vote for atrac3+

I suggest to encode via SS3, because it seems to have improved.
I would upload some samples, but it is not possible, because i have only a 56 kbit connection.

(unfortunality)

P.S.: IMHO it sounds better than mp3 but worse than vorbis and he-aac at 64 kbit.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-24 13:02:45
Another suggestion (related to the sample): instead of focusing too much on musical genre (metal - jazz - classical ...), I think it would be better to choose sample for the kind of signal they represent: loud - quiet - noisy - tonal - attacks...

When I sent to Roberto the very quiet sample called Debussy.wav, which had apparently nothing hard to encode, most people were at the end surprised by the poor performance of the champion (musepack). This sample revealed severe issues with musepack (even wma & atrac3 were better) at moderate bitrate. I know that some lossy encoders have serious problems with very tonal music (-> ringing); some other suffers with low volume content. There's also pre-echo...


If you're interested, I could propose several samples.
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-24 13:12:10
One thing I'd like is to let encoders adapt to the content before the test position.
Most encoders have adaptative thresholds, and so need a few time to adapt at the beginning. It means that a specific piece would not be encoded the same if it is at the beginning of the track or in the middle.
I think that a 1s delay should be reasonable enough.

So would it be possible to:

*cut the first second of the decompressed sample?

or

*instruct the ABC/HR software to only allow testing past the first second?
Title: 64 kbps listening test 2005
Post by: ff123 on 2005-03-24 14:41:36
I'm not sure if abchr-java can force the following options, but I'm sure schnofler can modify his code:

1) the rating scale description should be changed to the "excellent" to "poor" labels; I already know this option exists, but it should be forced from the configuration file

2) the start time should be forced to X sec into the clip without allowing the listener to hear anything before that time, also specified from the configuration file.

ff123
Title: 64 kbps listening test 2005
Post by: moi on 2005-03-24 17:43:09
Quote
Quote
I really don't see why LAME at 128kbps should be  included in a 64kbps listening test, as it was the other time. Probably has something to do with the claim that WMA at 64kbps sounds "as good as" MP3 at 128kbps. I don't think many here believe that claim. In any case, IMO, a 64kbps listening test should only include music encoded at 64kbps. It is misleading to encode 128kbps in one format, and 64kbps in all the others. Everything in a 64kbps listening test should be encoded at 64kbps.
[a href="index.php?act=findpost&pid=284822"][{POST_SNAPBACK}][/a]

A credible listening test should have a low and high anchor.
[a href="index.php?act=findpost&pid=284834"][{POST_SNAPBACK}][/a]


What does that mean, a high and low anchor? I guess I really don't know what that means--it just seems strange, that for a 64kbps listening test, one of the formats would be tested at 128 kbps rather than at 64 kbps.

If it is to have a reference to compare to, then why not have one sample uncompressed, for listeners to compare the compressed versions with? (Perhaps that's already done. That makes sense, but I don't understand the " high and low anchor", I guess. Please explain.

Does "high anchor" always mean one format is tested at a higher bit rate than the others? For low anchor a lower bit rate? Will you test one format at 32kbps for the "low anchor"?

In the 128kbps listening test, was one of the formats tested at 192kbps for the "high anchor"?
Title: 64 kbps listening test 2005
Post by: beto on 2005-03-24 17:55:23
high anchor -> performs noticeably better than the codecs average being tested
low anchor -> performs noticeably worse than the codecs average being tested

afaik this is done to get meaningful statistic results. The high/low anchors are not part of the test itself in the sense that they are not evaluated. They are just a reference...

someone correct me if i am wrong.
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-03-24 17:57:18
The purpose of anchors is to bind the results to real world i.e. when you have an anchor your results won't anymore "float" in the air. When you have anchors, you can compare codecs which are featured in different listening test to each other to some extent.
Title: 64 kbps listening test 2005
Post by: schnofler on 2005-03-24 18:26:43
Quote
1) the rating scale description should be changed to the "excellent" to "poor" labels; I already know this option exists, but it should be forced from the configuration file
[a href="index.php?act=findpost&pid=285190"][{POST_SNAPBACK}][/a]

I'm not sure what exactly you mean by "forced from the configuration file". The custom rating labels can be specified in the test setup dialog and will be saved to the configuration file.

Quote
2) the start time should be forced to X sec into the clip without allowing the listener to hear anything before that time, also specified from the configuration file.
[a href="index.php?act=findpost&pid=285190"][{POST_SNAPBACK}][/a]

The offset setting could be used for this. Just adding 1000*X to each of the offsets will have exactly that effect.
Title: 64 kbps listening test 2005
Post by: ff123 on 2005-03-24 20:29:17
Quote
Quote
1) the rating scale description should be changed to the "excellent" to "poor" labels; I already know this option exists, but it should be forced from the configuration file
[a href="index.php?act=findpost&pid=285190"][{POST_SNAPBACK}][/a]

I'm not sure what exactly you mean by "forced from the configuration file". The custom rating labels can be specified in the test setup dialog and will be saved to the configuration file.

Quote
2) the start time should be forced to X sec into the clip without allowing the listener to hear anything before that time, also specified from the configuration file.
[a href="index.php?act=findpost&pid=285190"][{POST_SNAPBACK}][/a]

The offset setting could be used for this. Just adding 1000*X to each of the offsets will have exactly that effect.
[a href="index.php?act=findpost&pid=285254"][{POST_SNAPBACK}][/a]


What I meant is that Sebastian should be able to create a configuration file that everyone uses, and which will control the rating labels.  Doh, forgot about those offsets in the config file!  That's the easy solution, of course.

ff123
Title: 64 kbps listening test 2005
Post by: jaybeee on 2005-03-24 20:34:55
Quote
I also think that 30s might be too long.
Perhaps 6s is too short, but I think that 15s should be enough.

Letting testers deciding which portion to use is perhaps reducing "usefullness" of results. It is like they are testing different samples, but it makes correlation between results for the same sample harder.

If a sample has some quite different parts in a 30s set, then it could be intersting to split it into 2 samples, making interpretation of results easier.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=285160")


I've just uploaded an 18sec track [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=32691]here[/url] that I feel would be good for this test.  I deliberated over which section to use and also how long that section was to be - the song is 21min long and has a lot of demanding parts.  I think I chose the best part.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-24 21:22:41
Samples should be posted here, please: http://www.hydrogenaudio.org/forums/index....showtopic=32689 (http://www.hydrogenaudio.org/forums/index.php?showtopic=32689)
Title: 64 kbps listening test 2005
Post by: schnofler on 2005-03-24 23:13:21
Quote
What I meant is that Sebastian should be able to create a configuration file that everyone uses, and which will control the rating labels.
[a href="index.php?act=findpost&pid=285283"][{POST_SNAPBACK}][/a]

Yes, that is possible.

Quote
Doh, forgot about those offsets in the config file!  That's the easy solution, of course.
[a href="index.php?act=findpost&pid=285283"][{POST_SNAPBACK}][/a]

Heh. Yes, I was just about to get to work on the "new" feature myself, when I noticed it's not such a new feature, really. 
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-25 13:44:06
So far, the settings used will be:

Nero HE-AAC: VBR profile "Streaming :: Medium", High Quality
Vorbis: -q 0
WMA Standard: -a_codec WMA9STD -a_mode 3 -a_setting 64_44_2
LAME 3.96.1 (high anchor): -V5 --athaa-sensitivity 1

ATRAC3+ samples will be encoded using whatever settings produce 64kbps, same applies to Apple HE-AAC.

Regarding the low anchor, I would use Adobe Audition 1.5 and the FhG encoder at 64 kbps CBR, but others might want to use LAME.
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-25 15:28:26
For the high anchor, I would prefer Lame 3.97 (probably in abr setting) that will probably be at least in beta stage when the test should start.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-25 20:34:25
Quote
For the high anchor, I would prefer Lame 3.97 (probably in abr setting) that will probably be at least in beta stage when the test should start.
[a href="index.php?act=findpost&pid=285477"][{POST_SNAPBACK}][/a]


So, --preset 128 then?
Title: 64 kbps listening test 2005
Post by: Jojo on 2005-03-26 01:46:41
Quote
For the high anchor, I would prefer Lame 3.97 (probably in abr setting) that will probably be at least in beta stage when the test should start.
[a href="index.php?act=findpost&pid=285477"][{POST_SNAPBACK}][/a]

just out of curiosity are you saying that some ABR preset in the new LAME 3.97 built might be better than -V5 --athaa-sensitivity 1 ?
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-27 08:48:04
So, the list of codecs is now pretty much done:

Apple HE-AAC
Nero HE-AAC
WMA Standard
ATRAC3+
LAME 3.97 MP3 (high anchor)
Adobe Audition FhG MP3 (low anchor)
Ogg Vorbis (AoTuV3 or 1.1)

At this point, I would like to ask people again to test between the two Vorbis encoders. If you have time, you can also give Archer a try, but the test should focus on AoTuV3 and 1.1.
Title: 64 kbps listening test 2005
Post by: westgroveg on 2005-03-27 09:42:20
What about MP3+? would be interesting to see if the HE-AAC encoders can perform better MP3+ yet
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-27 10:00:24
Quote
What about MP3+? would be interesting to see if the HE-AAC encoders can perform better MP3+ yet
[a href="index.php?act=findpost&pid=285951"][{POST_SNAPBACK}][/a]


I suppose you mean mp3PRO... Well, it was tested last time and it performed quite well, but only came third after Nero HE-AAC and the high anchor LAME at 128 kbps.

(http://www.rjamorim.com/test/64test/plot12z.png)

I will not include it in this test because there are no improvements since the last test and also because it is a pretty rare format with little soft- and hardware support.
Title: 64 kbps listening test 2005
Post by: vinnie97 on 2005-03-27 21:13:59
There's a AoTuV-prebeta4 to check now which supposedly resolves some issues @ q0.  http://www.geocities.jp/aoyoume/aotuv/test.html (http://www.geocities.jp/aoyoume/aotuv/test.html)
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-30 19:27:11
I did a small listening test for WMA9 encoders. As samples, I've used all selected by Roberto for his last 128 kbps Multiformat Listening Test.


Two important things:

• I didn't browse HA since last thursday (If decisions were made in this topic since one week, I wasn't aware)
• this listening test was a very fast one. Too fast I would say. I didn't ABX anything; and I've probably miss some details.

(http://www.foobar2000.net/divers/tests/2005.03/wma/wma64results2.png)

• WMA9Pro is better, but bitrate doesn't tend to 64 kbps at -q10. WMA9Pro is nevertheless not that better.
• Statistically, CBR 2 pass and VBR 2 pass are tied, but CBR 64kbps 2-pass appeared to be a bit more constant in quality than VBR at low bitrate.


EDIT: blank log files (no comment, simple notation) are available here (http://www.foobar2000.net/divers/tests/2005.03/wma/WMA64_ABCHR_logs.7z).
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-30 19:40:18
Thanks for the test guruboolez! Weird that VBR is a bit worse than CBR - didn't expect that. I guess I will use CBR for WMA standard then.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-30 20:24:42
As I said, I was not fully satisfied by this test (too fast, too imprecise). If the collective test doesn't start in the next days, I think I could test CBR and VBR again, without WMApro this time, and with ABX phase in order to be sure that difference were audible.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-30 20:41:53
Quote
As I said, I was not fully satisfied by this test (too fast, too imprecise). If the collective test doesn't start in the next days, I think I could test CBR and VBR again, without WMApro this time, and with ABX phase in order to be sure that difference were audible.
[a href="index.php?act=findpost&pid=287014"][{POST_SNAPBACK}][/a]


Well, I guess it will take at least one more week for the test to start. Apple's HE-AAC encoder isn't even out yet.

Also, again, Vorbis tests are welcome to help deciding which encoder to use.
Title: 64 kbps listening test 2005
Post by: Busemann on 2005-03-30 21:05:47
Quote
Well, I guess it will take at least one more week for the test to start. Apple's HE-AAC encoder isn't even out yet.
[a href="index.php?act=findpost&pid=287022"][{POST_SNAPBACK}][/a]



I reckon HE-AAC is still a bit off, definitely more than 1 week.
Title: 64 kbps listening test 2005
Post by: slippyC on 2005-03-30 21:44:17
I know this is a little late in the thread and I may have missed it, but I think you ought to do both WMA and WMAPro.

Reason being is because WMA is widespread, but WMAPro is superior to Standard.  It would keep people from becoming confused and still giving the WMAPro codec a fair chance(to keep the WMA fanatics off your back).
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-03-30 22:09:58
Quote
I reckon HE-AAC is still a bit off, definitely more than 1 week.
[a href="index.php?act=findpost&pid=287032"][{POST_SNAPBACK}][/a]


Rumour has it that Tiger will be released mid-april. Of course, Apple rumours are NEVER trustworthy.

Quote
Reason being is because WMA is widespread, but WMAPro is superior to Standard.  It would keep people from becoming confused and still giving the WMAPro codec a fair chance(to keep the WMA fanatics off your back).
[a href="index.php?act=findpost&pid=287048"][{POST_SNAPBACK}][/a]


I think the level of usefulness of WMA Pro at 64kbps is too small. Microsoft clearly sends you the message that they only want you using it above 128kbps (in CBR mode at least). Forcing it to go lower with VBR results in too random bitrates that would make the test biased for one side or the other.

Also, it's worth remembering such low bitrates are useful for a) limited memory hardware or b) streaming. a) is pointless because no portable player supports WMA Pro. b) makes no sense because VBR doesn't mix well with streaming.
Title: 64 kbps listening test 2005
Post by: slippyC on 2005-03-31 04:50:46
Quote
I think the level of usefulness of WMA Pro at 64kbps is too small. Microsoft clearly sends you the message that they only want you using it above 128kbps (in CBR mode at least). Forcing it to go lower with VBR results in too random bitrates that would make the test biased for one side or the other.

Also, it's worth remembering such low bitrates are useful for a) limited memory hardware or b) streaming. a) is pointless because no portable player supports WMA Pro. b) makes no sense because VBR doesn't mix well with streaming.
[a href="index.php?act=findpost&pid=287061"][{POST_SNAPBACK}][/a]



Yea, I don't know of any portable support.  If I remember right someone on this board said that MS never intended it to be.

Anyway, was just a suggestion for some folks concerns.  I don't use WMAPro, have messed with it a little in the past, so didn't matter much to me.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-03-31 08:46:18
Suggestion: if you plan to publish a bitrate table, and in order to avoid (at least to limit) the usual discussion about 'fairness', 'apples & orange', etc... why not publishing a bitrate table of the full encoded song instead of the short samples (~20 sec), which are probably higher.

It's not really easy (you have to download full samples, or have to ask to people submitting samples to encode their files for you and then report the exact value), but it could be more representative of the bitrate of very short samples. On visual tests (video, cf. doom9.org), we have the bitrate for the full movie, and not for the selected frames. What do you think?
Title: 64 kbps listening test 2005
Post by: Gabriel on 2005-03-31 08:55:18
Regarding the low anchor, I would personnally like Lame @abr.
My own justification is:
*that would probably not change that much the notation of the low anchor, and anyway the purpose of the low anchor is to be low.
*that would allow me to have fine input regarding Lame's true performance @64k compared to modern codecs
Title: 64 kbps listening test 2005
Post by: AtaqueEG on 2005-03-31 08:59:41
Quote
Suggestion: if you plan to publish a bitrate table, and in order to avoid (at least to limit) the usual discussion about 'fairness', 'apples & orange', etc... why not publishing a bitrate table of the full encoded song instead of the short samples (~20 sec), which are probably higher.

It's not really easy (you have to download full samples, or have to ask to people submitting samples to encode their files for you and then report the exact value), but it could be more representative of the bitrate of very short samples. On visual tests (video, cf. doom9.org), we have the bitrate for the full movie, and not for the selected frames. What do you think?
[a href="index.php?act=findpost&pid=287162"][{POST_SNAPBACK}][/a]


This is a great idea.
And it would give you a "real life" scenario, showing the actual behavior of the codec with full songs, and how it would act on such tracks.
Title: 64 kbps listening test 2005
Post by: nyarlathotep on 2005-03-31 16:24:18
Quote
• I think that the current scale isn't really really suited to a 64 kbps and the expected distortions.
Artifacts “perceptible but not annoying” (4.0) are maybe not very common at this bitrate. And few encoders are able to reproduce (in my opinion) a sound with only "slightly annoying" (3.0) difference at 64 kbps. It's possible to change the corresponding scale with schnofler's abc/hr, and I wonder if it's not worth to think about it. If I remember correctly, the average notation I gave to most encoders during the 32 kbps was inferior to 1.5/5 

I second that suggestion. During the 32 kbps test, I gave most sample only between 1 and 2 because then I decided to follow the comments near the scale while (it seems that) a lot of people use the whole scale without taking into account the comments '“perceptible but not annoying” and so on...).

I mean that the situation should be clarified: should we use the whole scale or follow the comments?
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-03-31 16:54:47
Quote
I mean that the situation should be clarified: should we use the whole scale or follow the comments?
[a href="index.php?act=findpost&pid=287251"][{POST_SNAPBACK}][/a]


I will modify to the labels to excellent - poor.
Title: 64 kbps listening test 2005
Post by: Supernaut on 2005-03-31 19:30:30
Quote
I will modify to the labels to excellent - poor.

Perhaps you could post your choice of the labels so we can discuss and perhaps improve them, in order to make them more descriptive and as unambiguous as possible? I'm not sure "good" and "excellent" labels would help me make my results on scale with others' results...
Title: 64 kbps listening test 2005
Post by: Busemann on 2005-04-01 00:19:53
Quote
Rumour has it that Tiger will be released mid-april. Of course, Apple rumours are NEVER trustworthy.



Tiger has gone GM, the question is whether HE-AAC will be included in QT 7 after all..
Title: 64 kbps listening test 2005
Post by: kurtnoise on 2005-04-01 17:52:03
Sebastian...I don't know if you are already begun the preliminaries for these tests, but may I suggest to add a new candidate : Enhanced AAC+ v2 ??

Brandon Siegel has updated its CLI encoder for dbPowerAmp. It can now reach up to 64 kbps for stereo files. Check this thread (http://forum.dbpoweramp.com/showthread.php?t=7107) directly on dBAmp forum for more infos.


Another question : have you got enough samples or can we post new ones ?
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-04-01 17:59:32
HE-AAC v.2 = AAC + SBR (v.1) + Parametric Stereo (=> v.2). According to some tests requested by Ivan ~one year ago, Parametric Stereo have a negative impact at "high" bitrate (> 40 kbps IIRC). Of course, it was noticed on Nero AAC encoder, and it doesn't necessary mean than other encoders have similar reactions. Therfore, it could be worth to test.


To be entirely honest, I don't really like the idea of testing two HE-AAC encoders (Nero & Apple) in a same multiformat test. I don't like it, but I understand the reason I think. I would rather make preliminary tests, in order to choose the best challenger for each format (as we do for wma, vorbis...). There are many possible contenders:

- Nero AAC VBR & CBR (does VBR produce stable quality at low bitrate?)
- Apple VBR & CBR (same comment)
- Coding Technologie (in Real products) AAC
- the one pointed by kurtnoise.

But to make this possible, we need people to test them first.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-01 18:19:31
Quote
Sebastian...I don't know if you are already begun the preliminaries for these tests, but may I suggest to add a new candidate : Enhanced AAC+ v2 ??

Brandon Siegel has updated its CLI encoder for dbPowerAmp. It can now reach up to 64 kbps for stereo files. Check this thread (http://forum.dbpoweramp.com/showthread.php?t=7107) directly on dBAmp forum for more infos.


Another question : have you got enough samples or can we post new ones ?
[a href="index.php?act=findpost&pid=287538"][{POST_SNAPBACK}][/a]


Well, with Apple and Nero, the two most popular HE-AAC encoders, I have enough HE-AAC encoders covered.

Regarding samples, I have enough now, thank you. It's a bit hard to decide which ones to use, since almost all of them are more or less killer samples.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-01 18:22:51
Quote
To be entirely honest, I don't really like the idea of testing two HE-AAC encoders (Nero & Apple) in a same multiformat test. I don't like it, but I understand the reason I think. I would rather make preliminary tests, in order to choose the best challenger for each format (as we do for wma, vorbis...). There are many possible contenders:

- Nero AAC VBR & CBR (does VBR produce stable quality at low bitrate?)
- Apple VBR & CBR (same comment)
- Coding Technologie (in Real products) AAC
- the one pointed by kurtnoise.

But to make this possible, we need people to test them first.
[a href="index.php?act=findpost&pid=287542"][{POST_SNAPBACK}][/a]


I guess you are right, but who wants to conduce another pre-test? Look at how many people tested Vorbis.

Edit: By the way... Regarding WMA, I guess I will stick to Standard since Pro isn't really "tuned" for such bit-rates and as Roberto said, 64 kbps is used for either streaming or portable usage. None of those areas are covered well by Pro.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-04-01 18:28:16
Quote
I guess you are right, but who wants to conduce another pre-test? Look at how many people tested Vorbis.
[a href="index.php?act=findpost&pid=287548"][{POST_SNAPBACK}][/a]


I had this in mind when I said "I don't like it, but I understand the reason I think" 
Preliminary collective tests are possible, without someone to conduce it from A to Z. Remember last 128 kbps listening test: there were many choices for vorbis (CVS, aoTuV, Quantum Knot, Modest Tuning), and the final decision was based upon personal tests. We know the result, more than positive for Vorbis
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-01 18:36:47
Quote
Quote
I guess you are right, but who wants to conduce another pre-test? Look at how many people tested Vorbis.
[a href="index.php?act=findpost&pid=287548"][{POST_SNAPBACK}][/a]


I had this in mind when I said "I don't like it, but I understand the reason I think" 
Preliminary collective tests are possible, without someone to conduce it from A to Z. Remember last 128 kbps listening test: there were many choices for vorbis (CVS, aoTuV, Quantum Knot, Modest Tuning), and the final decision was based upon personal tests. We know the result, more than positive for Vorbis
[a href="index.php?act=findpost&pid=287549"][{POST_SNAPBACK}][/a]


Yes, but who has the time to spend on testing? I don't think I can justify a winner based on five or six test results.

Edit: BTW, everyone is free to conduce pre-tests.
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-04-01 18:59:02
Quote
I don't think I can justify a winner based on five or six test results.
[a href="index.php?act=findpost&pid=287550"][{POST_SNAPBACK}][/a]


I can't find the exact thread, but if I remember correctly, there were no more persons to test various fork of Vorbis in order to select the good one last spring.

Quote
Yes, but who has the time to spend on testing?

Time is not the problem. Motivation is probably a greater one. I don't think that many people will be interested to test HE-AAC implementation that are not used otherwise.

Nevertheless, removing one redundant challenger could make the test easier (each additionnal codec makes evaluation and the establishment of the hierarchy harder). There are currently 7 contenders (including anchor): that's very much, especially for untrained people.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-01 19:09:26
Quote
Nevertheless, removing one redundant challenger could make the test easier (each additionnal codec makes evaluation and the establishment of the hierarchy harder). There are currently 7 contenders (including anchor): that's very much, especially for untrained people.
[a href="index.php?act=findpost&pid=287559"][{POST_SNAPBACK}][/a]


Well, the most I can do is dump ATRAC3+ and if enough tests are available, one of the HE-AAC encoders.
The reason why I wanted both HE-AAC encoders to be featured is that a lot of work was done to improve the Nero encoder and it is also interesting to see how Apple performs against the big competitor (so there is a direct comparison).
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-01 19:12:53
Quote
I can't find the exact thread, but if I remember correctly, there were no more persons to test various fork of Vorbis in order to select the good one last spring.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=287559")


[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=20389]This?[/url]
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-04-01 19:23:56
Yes, this one. I've found the exact post (http://www.hydrogenaudio.org/forums/index.php?showtopic=20389&view=findpost&p=205464) I had in mind. Roberto had only 5 set of results (+ 2 partial results) to make a decision. Also note that aoTuV beta 1 was pre-tested, and that the non-tested aoTuV beta2 was included in the final collective test... and won (tied with mpc). Taking some risks is not necessary a bad thing
Title: 64 kbps listening test 2005
Post by: rudefyet on 2005-04-13 02:48:08
Quote
- Nero AAC VBR & CBR (does VBR produce stable quality at low bitrate?)


From personal tests the "fast" mode on Nero causes some serious problems/distortions around 96kbps or lower in LC-AAC and somewhere below 64kbps it causes trouble on HE-AAC (32kbps HE-AAC in fast mode sounded horrific! High mode sounded much better)

high mode with the "streaming" profile gives good quality 64kbps HE-AAC, fast mode sounds similar at 64kbps, but the bitrates only average around ~50-55kbps, i haven't done any ABxing to determine any quality differences though
Title: 64 kbps listening test 2005
Post by: negritot on 2005-04-13 06:42:37
So Quicktime 7 will be widely available in a little over two weeks. Is that enough time to narrow down the contenders? And can anyone verify that Quicktime 7 includes an HE-AAC encoder? AAC isn't even mentioned on the preview pages for Quicktime 7.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-13 07:32:28
Quote
So Quicktime 7 will be widely available in a little over two weeks. Is that enough time to narrow down the contenders? And can anyone verify that Quicktime 7 includes an HE-AAC encoder? AAC isn't even mentioned on the preview pages for Quicktime 7.
[a href="index.php?act=findpost&pid=290173"][{POST_SNAPBACK}][/a]


Well, I am still waiting for some Vorbis tests. Now that AoTuV PB 4 is out, somebody might want to test that against Xiph 1.1 and Archer.
Regarding HE-AAC tests, I doubt that anyone would like to test on their own. Also, I would rather get the test started pretty soon after Apple releases their HE-AAC encoder, so there won't be much room (time) for pre-tests. Therefore, I guess I will stick to my first decision featuring both codecs in the main test.
Title: 64 kbps listening test 2005
Post by: JohnV on 2005-04-20 22:29:33
Quote
So Quicktime 7 will be widely available in a little over two weeks. Is that enough time to narrow down the contenders? And can anyone verify that Quicktime 7 includes an HE-AAC encoder? AAC isn't even mentioned on the preview pages for Quicktime 7.
[a href="index.php?act=findpost&pid=290173"][{POST_SNAPBACK}][/a]

I was just visiting Apple booth at NAB Las Vegas and saw what they have and asked this. Answer is there's no HE-AAC in Quicktime 7, maybe in the future.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-04-20 23:35:01
Too bad. I suppose this test will have to wait then, as QT HE AAC would be the most interesting novelty.
Title: 64 kbps listening test 2005
Post by: aspifox on 2005-04-21 08:04:44
Quote
Too bad. I suppose this test will have to wait then, as QT HE AAC would be the most interesting novelty.
[a href="index.php?act=findpost&pid=292290"][{POST_SNAPBACK}][/a]

I'm still interested in the results without QT HE AAC.  It'd be nice to re-compare where the various codecs are after a year of development.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-04-21 14:30:14
I believe Sebastian will leave the decision to the forum members. He left me some offline messages on ICQ talking about not being online for a few days, but I couldn't understand why or when he will return.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-04-21 14:40:25
Quote
I believe Sebastian will leave the decision to the forum members. He left me some offline messages on ICQ talking about not being online for a few days, but I couldn't understand why or when he will return.
[a href="index.php?act=findpost&pid=292404"][{POST_SNAPBACK}][/a]


Well, IPS is updating the forum software to 2.1 Alpha and during this process, the blogs are offline, too. The problem is that I had a surgery near the Coccyx on Friday and am not allowed to sit or lay on my back for two weeks. I also have to go to the doctor and let him clean the open wound (which is a painful process, unfortunately) each morning.
Title: 64 kbps listening test 2005
Post by: Halcyon on 2005-05-22 14:59:03
ATRAC VS MP3PRO
=============
If it's not settled yet, I think the arguments for Atrac3+ and against Mp3Pro are pretty solid. I vote for Atrac3+ as well.

ATRAC encoder
------------------
However, I think somebody should really test whether the best (Sony) hardware Atrac encoders are _different_ from the latest SonicStage software release (assuming same version of Atrac now).

It's all very fine to assume there are no differences, but that's not a proof in scientific terms and it leaves way too much room for useless idle speculation, which is going to be rampant anyhow.

Maybe somebody with connections to the Atrac3 forum people could encourage a user there to encode a few tracks both with a hardware encoder and Sonicstage and see if they are bit-accurate?

Then again, even if we'd find out that hardware encoders are different in their output, it still doesn't solve the question of which encoder to use.

As such, for the sake of implementation easiness I recommend going with the encoder that the person doing the encoding is most comfortable using.

If somebody complains, we can ask him/her to conduct her own tests.

ANCHORS
=======
As for anchors, I think they should be of clearly higher/lower quality in many respects.

Remember, this is a subjective analysis and some people find some artifacts annoying, while other people are almost ignorant of them.

That is, making the anchors too difficult to spot will only muddy the results.

Bitrate bloat (esp. WMA Std)
=====================
I know this issue is not a favourite amongst many of us, but how will the bit-rate averaging issue be handled?

While a 5-25% difference in avg bitrate may not always be critical at 128-160kbps, it can have serious skewing at 64 kbps testing, no?

I for one have noticed in my own testing that getting WMA 9.1 Standard 2-pass VBR (ABR) to achieve anywhere near the advertised bitrate is really hard, considering there isn't much flexibility in choosing the target bitrates.

For example, I'm now encoding to 128 ABR for a test of mine and WMA9 constantly gives 140-160 ABR on most tracks, even though the target is set to 128 kbps (2-pass vbr, 9.1 WMA std, encoded from dbPowerAMP rel.11).

Can this issue be handled in any meaningful manner? Is it a problem with the chosen sample set?

BTW, the problem of average bitrate fluctuation with OGG (aotuv b3) and MP3 (lame 3.96.1)  is of much smaller magnitude - at least on my encodings.

SOFTWARE
=======

What software will be (can be used) to conduct the test when the testers download the sample pack?

I'm not very fond of ABCHR Java version myself.

Also, this is probably a FAQ, but I couldn't find an answer for this by searching, how will the samples be decoded into the final test form (in regards to clipping, gain/limiting, dither)?


CHECKING
=======
Would it be possible to check the test data submissions for accidental clicks?

For example, if for a certain sample set both samples are rated below 5.0, this is more than likely a mistake. It wouldn't be too difficult to check for this, if it's not already checked for.

I have had this happen to me on various occasions during the previous test (when clicking play/stop buttons and moving sliders): i'd rate a sample that I had ABXed properly at say 3.5. However, I had accidentally also moved the paired sample rating to 4.8 without noticing it.

Other than that, my hat goes to you on staring to pull this test together. It's not easy work, but somebody's gotta do it

regards,
Halcyon

PS I hope you recover soon from you operation. Take it easy though. Health is more important than testing
Title: 64 kbps listening test 2005
Post by: Latexxx on 2005-05-22 15:03:27
Hardware and software versions of atrac3plus won't be bit-identical because all hardware encoders (Hi-MD players) always resample the audio at 44,1 kHz even if the original is at that sample rate to compensate jitter.
Title: 64 kbps listening test 2005
Post by: Halcyon on 2005-05-22 15:58:22
Ah well, another good reason to forget about the comparison or use HW encoders, imho. BTW, do you have a source for this? Do all Sony Atrac HW gear use asynchronous sample rate conversion by default?
Title: 64 kbps listening test 2005
Post by: Defsac on 2005-05-25 10:44:52
Quote
I for one have noticed in my own testing that getting WMA 9.1 Standard 2-pass VBR (ABR) to achieve anywhere near the advertised bitrate is really hard, considering there isn't much flexibility in choosing the target bitrates.

For example, I'm now encoding to 128 ABR for a test of mine and WMA9 constantly gives 140-160 ABR on most tracks, even though the target is set to 128 kbps (2-pass vbr, 9.1 WMA std, encoded from dbPowerAMP rel.11).
[a href="index.php?act=findpost&pid=299470"][{POST_SNAPBACK}][/a]

Windows Media Encoder seems to handle bloat reasonably.
Bit rate (expected):   64.02 Kbps
Bit rate (average):   64.19 Kbps
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-05-25 11:35:34
Quote
Quote
I for one have noticed in my own testing that getting WMA 9.1 Standard 2-pass VBR (ABR) to achieve anywhere near the advertised bitrate is really hard, considering there isn't much flexibility in choosing the target bitrates.

For example, I'm now encoding to 128 ABR for a test of mine and WMA9 constantly gives 140-160 ABR on most tracks, even though the target is set to 128 kbps (2-pass vbr, 9.1 WMA std, encoded from dbPowerAMP rel.11).
[a href="index.php?act=findpost&pid=299470"][{POST_SNAPBACK}][/a]

Windows Media Encoder seems to handle bloat reasonably.
Bit rate (expected):   64.02 Kbps
Bit rate (average):   64.19 Kbps
[a href="index.php?act=findpost&pid=300407"][{POST_SNAPBACK}][/a]


According to my tests, I managed to get bitrates around 64 kbps for most of the test samples using WME, too.
Title: 64 kbps listening test 2005
Post by: Shade[ST] on 2005-05-26 23:17:05
Any news on what the final low anchor will be? Is it still to be Adobe Audition 1.5 FhG Encoder? and if so, in what mode? VBR (ABR)? CBR?
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-05-26 23:41:10
Most probably same as in Roberto's last 64kbps test: 64kbps CBR, allow M/S, no I/S, allow narrowing, no CRC.
Title: 64 kbps listening test 2005
Post by: ckjnigel on 2005-05-31 09:45:13
I'm way late looking in on this, but I sure would have liked to see the bsiegel AAC+ encoder for dbPoweramp tested.  The max rate supported is 64kbps.
I declare it the shiznit for Pocket PC users.
Title: 64 kbps listening test 2005
Post by: guest0101 on 2005-05-31 10:48:11
How about using the Coding Technolgies AACPlus encoder as released in the new Magix MP3 Maker 10 Deluxe package for the listening test? See thread about it at:
http://www.hydrogenaudio.org/forums/index....showtopic=34361 (http://www.hydrogenaudio.org/forums/index.php?showtopic=34361)

The CT encoder is sure to be included in many new products shortly. This is the first consumer implementation of a file based encoder using the CT AACPlus encoder I know of. It seems to sound pretty good to me compared to Nero and Quicktime, and it really should be put to the test I believe. Thanks!
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-05-31 11:51:07
As already mentioned, I don't mind people doing some private listening tests to decide which HE-AAC encoder to be used.
Title: 64 kbps listening test 2005
Post by: ckjnigel on 2005-06-02 23:13:31
I've already submitted my review of Magix MP3 Maker 19 Deluxe in the thread that guest101 initiated.  I really do not think this codec can be excluded from testing because XM Satellite is so much in the news and their claims for high fidelity so loudly shouted (see: http://tinyurl.com/bk4c9 (http://tinyurl.com/bk4c9) ).
Certainly, inclusion of this codec in the test will result in more widespread attention of the test results than would otherwise be the case.
Title: 64 kbps listening test 2005
Post by: Woodinville on 2005-06-06 18:53:00
Quote
As for WMA I would recommend using WMA std. It is more widespread than PRO and is what most people refer to as WMA



Why would you use an older version of WMA, when you are using the state of the art with an AAC codec? I think that isn't very equitable.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-06-06 19:15:17
Quote
Quote
As for WMA I would recommend using WMA std. It is more widespread than PRO and is what most people refer to as WMA



Why would you use an older version of WMA, when you are using the state of the art with an AAC codec? I think that isn't very equitable.
[a href="index.php?act=findpost&pid=304046"][{POST_SNAPBACK}][/a]


1. I don't know any hardware players supporting WMA Pro.
2. Most people rip to WMA Standard because that's the default setting in WMP AFAIK.
3. Music stores offer music in the WMA Standard format.

Title: 64 kbps listening test 2005
Post by: Woodinville on 2005-06-06 21:35:15
Quote
1. I don't know any hardware players supporting WMA Pro.
2. Most people rip to WMA Standard because that's the default setting in WMP AFAIK.
3. Music stores offer music in the WMA Standard format.



So it's just like AAC-HE, give or take a very little.  Don't you think it's a bit improper to be comparing the state of the art in one technology to 3 year old technology elsewhere?

I would suggest that you use at least WMA-Pro, in the interest of simple equity.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-06-06 21:53:06
Well, I am not sure, but did WMA Pro change since the last test?
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-06-07 02:59:21
Can WMA-Pro be even encoded properly at 64kbps?

AFAIK, even Microsoft themselves don't advertize WMA Pro for such bitrates. The CBR settings only go down to 128kbps, and forcing 64kbps with VBR 10 doesn't work very well.
Title: 64 kbps listening test 2005
Post by: Defsac on 2005-06-07 10:41:18
Quote
Can WMA-Pro be even encoded properly at 64kbps?

AFAIK, even Microsoft themselves don't advertize WMA Pro for such bitrates. The CBR settings only go down to 128kbps, and forcing 64kbps with VBR 10 doesn't work very well.
[a href="index.php?act=findpost&pid=304188"][{POST_SNAPBACK}][/a]

As far as I can tell even at VBR Q10 you have to have either 5.1ch 16 bit or 2ch 24 bit, you can't have 2ch 16 bit with WMA Pro.
Title: 64 kbps listening test 2005
Post by: moi on 2005-07-09 17:20:31
Quote
Quote
I'd like to see both WMA Standard & Pro included.
[a href="index.php?act=findpost&pid=284530"][{POST_SNAPBACK}][/a]

I wouldn't include wma pro because no internet radio uses it and it would only confuse people.
[a href="index.php?act=findpost&pid=284533"][{POST_SNAPBACK}][/a]


I really doubt the people here would be confused. Most people who read these forums know the difference between WMA Standard and Pro.

I really think both should be included. Standard, because that is the WMA that by far most people use, and most mp3 players don't support Pro.

Pro, because I (probably many others as well) would like to know how it compares with the others. It can be played back on a Pocket PC (or Windows Smartphone), as well as from a desktop PC, and hopefully there will be more mp3 player support for it in the future.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-07-09 17:48:46
For your information, the test will start once Apple's HE-AAC is available.
Title: 64 kbps listening test 2005
Post by: Busemann on 2005-07-09 18:07:54
Quote
For your information, the test will start once Apple's HE-AAC is available.
[a href="index.php?act=findpost&pid=312037"][{POST_SNAPBACK}][/a]


hmm.. For all we know it could be years away.
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-07-09 18:13:19
I think it'd be better to conduce an 128 kbps test once LAME 3.97 and Nero AAC are out.
Title: 64 kbps listening test 2005
Post by: Sagittaire on 2005-08-12 00:07:58
I vote for WMA9 Pro and Sdt because:
1) Not the same codec
2) wma9 pro was never tested to 64 Kbps
3) hardware compatibility is not a problem : HE-AAC or MP3Pro are not compatible with hardware
4) wma9 sdt vs wma9 pro fight is very interessing

WMA9 Standard setting
I think that for very bitrate the best setting could be that:
cscript.exe wmcmd.vbs -a_codec WMA9STD -a_mode 4 -a_setting 64_44_2 -a_peakbitrate 96000 -a_peakbuffer 5000 -input C:\Audio -output C:\Audio

-> Max bitrate is 96 Kbps : lower variability
-> Max buffer = 5 sec : lower variability
In fact with these setting wma9 is between real VBR and real CBR and IMO it's better for overall quality at very low bitrate

WMA9 Pro sample
In theory it's impossible to make WMA9Pro with exact bitrate at 64Kbps (no profil for 64 Kbps target bitrate) but if you want I can make these encoding with exactly 64 Kbps for each sample in two pass mode.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-08-12 01:16:24
Quote
2) wma9 pro was never tested to 64 Kbps


It is not even meant to be used at 64kbps.

Quote
3) hardware compatibility is not a problem : HE-AAC or MP3Pro are not compatible with hardware


HE AAC is actually playable on some cell phones. And MP3pro is playable on several Thomson/RCA devices, among other brands.

Quote
4) wma9 sdt vs wma9 pro fight is very interessing


It is not a fight. Not even Microsoft is interested in that fight! For them, WMA Std is the consumer codec, meant to be used for CDDA resolution at 64kbps. WMA Pro is the professional codec meant to be used at high resolutions, multichannel streams, at bitrates higher than 128kbps.

Quote
WMA9 Standard setting
I think that for very bitrate the best setting could be that:
cscript.exe wmcmd.vbs -a_codec WMA9STD -a_mode 4 -a_setting 64_44_2 -a_peakbitrate 96000 -a_peakbuffer 5000 -input C:\Audio -output C:\Audio

-> Max bitrate is 96 Kbps : lower variability
-> Max buffer = 5 sec : lower variability
In fact with these setting wma9 is between real VBR and real CBR and IMO it's better for overall quality at very low bitrate


Settings tweaking IST EVUL.

Quote
WMA9 Pro sample
In theory it's impossible to make WMA9Pro with exact bitrate at 64Kbps (no profil for 64 Kbps target bitrate)[a href="index.php?act=findpost&pid=319462"][{POST_SNAPBACK}][/a]


There you go. WMA Pro is not meant to be encoded at 64kbps.
Title: 64 kbps listening test 2005
Post by: pieroxy on 2005-08-12 09:03:42
Quote
Regarding samples, I have enough now, thank you. It's a bit hard to decide which ones to use, since almost all of them are more or less killer samples.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=287547")


Beware of the bias of testing only killer samples. Comparing codecs on killer samples will only give you a hint on how they perform... on killer samples. Not on regular music. See [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=36184]here[/url] for a thread on the subject.
Title: 64 kbps listening test 2005
Post by: level on 2005-08-12 18:20:58
Quote
Beware of the bias of testing only killer samples. Comparing codecs on killer samples will only give you a hint on how they perform... on killer samples. Not on regular music. See here (http://www.hydrogenaudio.org/forums/index.php?showtopic=36184) for a thread on the subject.
[a href="index.php?act=findpost&pid=319563"][{POST_SNAPBACK}][/a]

Not necessarily. You remember that many killer samples (in particular here in this thread) are only regular music. In other words, are situations of the real life. Tuning any encoder with "difficult parts of music" (killer samples) in order to do it more robust is the way as HA has managed to improve the performance of the encoders.
Title: 64 kbps listening test 2005
Post by: xmixahlx on 2005-08-12 19:21:49
Quote
Beware of the bias of testing only killer samples. Comparing codecs on killer samples will only give you a hint on how they perform... on killer samples. Not on regular music. See here (http://www.hydrogenaudio.org/forums/index.php?showtopic=36184) for a thread on the subject.
[a href="index.php?act=findpost&pid=319563"][{POST_SNAPBACK}][/a]

keep your ridiculous ideas on conducting listening tests to your own thread.


later
Title: 64 kbps listening test 2005
Post by: echo on 2005-08-12 20:29:11
Quote
Quote
Regarding samples, I have enough now, thank you. It's a bit hard to decide which ones to use, since almost all of them are more or less killer samples.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=287547")


Beware of the bias of testing only killer samples. Comparing codecs on killer samples will only give you a hint on how they perform... on killer samples. Not on regular music. See [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=36184]here[/url] for a thread on the subject.
[a href="index.php?act=findpost&pid=319563"][{POST_SNAPBACK}][/a]

Heh, now this has become a fact? 

Quote
I think it'd be better to conduce an 128 kbps test once LAME 3.97 and Nero AAC are out.

I think that would be more interesting for everyone...
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-08-12 20:55:39
Quote
I think that would be more interesting for everyone...
[a href="index.php?act=findpost&pid=319720"][{POST_SNAPBACK}][/a]


I bet the two encoders will be released right after my summer vaction is over.  Hope not.
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-08-12 21:39:05
Quote
I bet the two encoders will be released right after my summer vaction is over.  Hope not.[a href="index.php?act=findpost&pid=319726"][{POST_SNAPBACK}][/a]


You might have LAME 3.97 beta before your vacation is over, but you'll hardly have final!
Title: 64 kbps listening test 2005
Post by: Sebastian Mares on 2005-08-12 21:51:19
Would you test with a beta or a final?
Title: 64 kbps listening test 2005
Post by: rjamorim on 2005-08-12 22:11:26
Quote
Would you test with a beta or a final?[a href="index.php?act=findpost&pid=319738"][{POST_SNAPBACK}][/a]


Wise words from Roberto:

Ask the Developers opinion. That way, you save your face and don't get cornered in case everything blows up.
Title: 64 kbps listening test 2005
Post by: Sagittaire on 2005-08-13 00:52:28
Quote
Settings tweaking IST EVUL.


arghhhhhh .... 
then 2 pass streaming mode with default setting : it's an usual mode in WMEnc
cscript.exe wmcmd.vbs -a_codec WMA9STD -a_mode 4 -a_setting 64_44_2 -input C:\Audio -output C:\Audio

default are:
- 1.5*bitrate for maxbitrate
- 3 sec for maxbuffer

Quote
There you go. WMA Pro is not meant to be encoded at 64kbps.


In fact q10 quality is perhabs this mode ... but it's just impossible for you to obtain the good target bitrate for your specific samples tests ...
Title: 64 kbps listening test 2005
Post by: guruboolez on 2005-08-19 05:10:17
Quote
Quote
Beware of the bias of testing only killer samples. Comparing codecs on killer samples will only give you a hint on how they perform... on killer samples. Not on regular music. See here (http://www.hydrogenaudio.org/forums/index.php?showtopic=36184) for a thread on the subject.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=319563")

keep your ridiculous ideas on conducting listening tests to your own thread.
[a href="index.php?act=findpost&pid=319701"][{POST_SNAPBACK}][/a]

Pieroxy's ideas have nothing ridiculous. He was very clear: a listening test involving VBR and CBR encoders should include low-bitrate encodings, mid-bitrate encodings and high-bitrate encodings.
I'm [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=36184&view=findpost&p=319058]quoting[/url] Pieroxy:
Quote
So in an ideal test, you would have 160kbps, 130kbps and 90kbps samples. THAT would be representative, not merely 128kbps.


Now I'm quoting (http://www.hydrogenaudio.org/forums/index.php?showtopic=21904&view=findpost&p=213827) someone who's not used to post "ridiculous ideas" about listening tests, ff123:
Quote
Ideally, you'd like the bitrate distribution to look somewhat like a bell curve with its mean at 128 kbit/s.

=> Exactly the same "ridiculous" idea about the "ideal" test.

And ff123 followed:
Quote
For the 48 kbit/s test, if there are VBR codecs, I think we should strive to have about an equal number of bitrates above and below the average bitrate (which should work out to be 48 kbit/s on average across the sample set).

Would you also say that ff123 has ridiculous ideas about listening test organisation? I'd really like to see that...



I posted the bitrate distribution curve of VBR encoders involved in my latest 80 kbps listening test:
http://foobar2000.net/divers/tests/2005.07...UTION_WMA80.png (http://foobar2000.net/divers/tests/2005.07/80/80TEST_BITRATEDISTRIBUTION_WMA80.png)
http://foobar2000.net/divers/tests/2005.07...ION_aoTuV80.png (http://foobar2000.net/divers/tests/2005.07/80/80TEST_BITRATEDISTRIBUTION_aoTuV80.png)

For both encoders, ~50% of the total samples were encoded with a bitrate lower to the targeted bitrate, and the ~50 remaining percent were encoded with a bitrate superior to the targeted bitrate. The distribution is close to the perfect curve for WMA and classical music (150 samples), with both mean and median at exactly 80 kbps. The curve is perfectible for aoTuV (median is 75 kbps), but distribution is also near-symetrical from both side of the median value.
It was maybe luck - I don't know - or maybe a consequence of the huge number of samples involved in my test (150+35) which maybe help to get a good distribution.