Skip to main content

Topic: 48 kbps listening test, pretest discussion thread (Read 15551 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Shade[ST]
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Hello!

After assisting like many of you on the controversy of guruboolez' tests concerning the Nero AAC encoders and having seen many suggestions towards preparing a listening test around 48kbps - in particular with the different HE encoders, I decided to take matters into my own hands.
I currently have 8 samples testing for tonality, transients and stereo separation.  These are generally classical music, but also include electronic music, 80s pop, and  I might add a jazz recording for ambience reproduction.
As for encoders, I'm stumped.

Should I also include an old version of nero (old being relative) as to see the progress in a general evaluation ?  At this bitrate, many encoders is not really a problem, is it?  For now, I was thinking :

-Nero AAC Tape / PNS Enabled / HE / High (Should I enable hint track?)
-Ogg Vorbis, latest AoTuV, q -2 to -4 (whatever's approximately the same bitrates ~42-53)
-Lame ABR 48 (any tunings suggested?)
-mp3pro, probably, at any close bitrate I can achieve (adobe audition's encoder)
-Itunes AAC (HE if available)
-CT encoder (winamp 5.12) 48kbps, HE.

If I should use an older nero as well, where can I find it?

Also, could anyone help me prepare the ABC/HR logs? (eg, Sebastian Mares)
Thanks in advance,
Tristan.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
48 kbps listening test, pretest discussion thread
Reply #1
Is it a collective listening test or a personal exercise?

  • Shade[ST]
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #2
Quote
Is it a collective listening test or a personal exercise?
[a href="index.php?act=findpost&pid=349682"][{POST_SNAPBACK}][/a]

Collective

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
48 kbps listening test, pretest discussion thread
Reply #3
Isn't it a bit premature to start a new pretest discussion thread, two weeks before the end of the current 128 test?

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
48 kbps listening test, pretest discussion thread
Reply #4
If it is a collective test, may I ask for delaying it at least a week or so, because we could finalize new LC-AAC encoder being actually used with the SBR core, it would be also put in the next Nero Release along with the LC improvements currently being tested @128 kbps.

But in general I also think it is a bad idea to conduct few tests at the same time.
  • Last Edit: 12 December, 2005, 11:17:31 AM by Ivan Dimkovic

  • Shade[ST]
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #5
Quote
If it is a collective test, may I ask for delaying it at least a week or so, because we could finalize new LC-AAC encoder being actually used with the SBR core, it would be also put in the next Nero Release along with the LC improvements currently being tested @128 kbps.

But in general I also think it is a bad idea to conduct few tests at the same time.
[a href="index.php?act=findpost&pid=349685"][{POST_SNAPBACK}][/a]

I wasn't intending to start the test until the beginning of january, but, sure, I can wait.

I'll copy the contents of this thread into a notepad and set it aside.

As a digressional note, for those of you on linux.. emacs or vi?

  • naylor83
  • [*][*][*]
48 kbps listening test, pretest discussion thread
Reply #6
Quote
I wasn't intending to start the test until the beginning of january, but, sure, I can wait.

I'll copy the contents of this thread into a notepad and set it aside.

As a digressional note, for those of you on linux.. emacs or vi?
[a href="index.php?act=findpost&pid=349686"][{POST_SNAPBACK}][/a]


Well, I can't see the problem of starting the discussion now. It's not like the discussion has to be a huge amount of work for anyone, unless it gets out of hand like last time...
davidnaylor.org

48 kbps listening test, pretest discussion thread
Reply #7
I think the whole thing is too rushed. Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...
  • 128 kbps extension test featuring ATRAC3, MusePack and WMA Standard (later might be problematic because of the settings)
  • Low bitrate (32/48/64 kbps) multiformat test
  • Low bitrate (32/48/64 kbps) AAC test
I would also wait until the end of January and check if Apple has an HE-AACv2 encoder ready.
  • Last Edit: 12 December, 2005, 11:52:57 AM by Sebastian Mares

  • naylor83
  • [*][*][*]
48 kbps listening test, pretest discussion thread
Reply #8
Quote
Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...

[a href="index.php?act=findpost&pid=349698"][{POST_SNAPBACK}][/a]


We all know how that would end... Just be an enlightened dictator Shade, and use your wisdom.
davidnaylor.org

48 kbps listening test, pretest discussion thread
Reply #9
Quote
Quote
Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...

[a href="index.php?act=findpost&pid=349698"][{POST_SNAPBACK}][/a]


We all know how that would end... Just be an enlightened dictator Shade, and use your wisdom.
[a href="index.php?act=findpost&pid=349704"][{POST_SNAPBACK}][/a]


The result of the "Fifth competitor..." poll was pretty clear, so I see nothing wrong in running a poll again.

Edit 1: Also, so far, I see you have six encoders in mind. If you include an older version of Nero, that makes seven - without any anchor. That is too much, so I would sort out some of the encoders leaving only five and a high anchor - maybe the 128 kbps winner?

And sure, I can help you set up the ABC/HR configuration files. Do you have ICQ or another IM client, Shade[ST]?

Edit 2: Is there any major difference between the mp3PRO encoder included in Nero and the one from Audition?
  • Last Edit: 12 December, 2005, 12:30:12 PM by Sebastian Mares

48 kbps listening test, pretest discussion thread
Reply #10
Regarding the encoders again - personally, I would exclude LAME (not tuned for such bitrates - even developers recommend FhG for bitrates lower than 128 kbps because FhG also uses IS) and mp3PRO (no development since last test AFAIK and it is being replaced by HE-AAC anyways). On the other hand, that leaves you with only Ogg Vorbis as format other than AAC. Anyone knows how WMA Standard 9.1 is supposed to perform at this bitrate?
  • Last Edit: 12 December, 2005, 12:53:07 PM by Sebastian Mares

  • naylor83
  • [*][*][*]
48 kbps listening test, pretest discussion thread
Reply #11
Quote
The result of the "Fifth competitor..." poll was pretty clear, so I see nothing wrong in running a poll again.
[a href="index.php?act=findpost&pid=349705"][{POST_SNAPBACK}][/a]


Yes, the poll was great! I was more thinking about the discussion that went on for eternity. (Which I hardly need tell you is a heck of a long time, to quote Rowan Atkinson aka The Devil aka Toby.)
davidnaylor.org

  • Gambit
  • [*][*][*][*][*]
  • Developer
48 kbps listening test, pretest discussion thread
Reply #12
With all respect, Shade[ST], this is a really bad time to start this. The 128kbps test is running, and we should concentrate on that and not confuse people with another listening test talk. And while I understand your enthusiasm, I think, now that we have Sebastian running the test, you should show some respect towards him. I'm sure he wants to run consequent tests after this one, so it's a bit inept to "mess with his business" now.
  • Last Edit: 12 December, 2005, 03:18:20 PM by Gambit
Burrrn - http://www.burrrn.net/
MPEG Audio Collection - http://mac.sourceforge.net/

  • Shade[ST]
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #13
I was thinking maybe about making LAME the low anchor.. lame @ 32?  And maybe set ogg vorbis around a bitrate of 100, or something of the like, to make it perceptible to some, but not annoying -- Personally, I was quickly irritated in the 128 test by not being able to discern anything.

I don't mind adding WMA std to this test, either.  The samples are pretty much chosen, I think - I've spoken to guru about this, and I'll try to have sebastian mares tell me if my samples are ok or not.

I don't think many people use itunes to encode at such a low bitrate, but then again, who knows?  some cell phones come with itunes now.  I'm arguing with myself as to keeping it or not.

I sort of insist on this being a 48kbps test only : I don't like the number 32, it's ugly (dictator, says naylor83?), and IMO 64 kbps will still be too hard to make out.

This needs to be a quick-and-dirty test, making it easy to pick up a few samples in 15 minutes of spare time and send in results.  I'll also set up a gmail for recieving test results.  I could give out the password later to anyone wishing to organise another listening test, or to a HA admin or whatnot, if results are wished to be kept.

For now, that's all I can think of.  If anyone wants to hear / comment on samples I chose, you can PM me or leave a message on this thread (preference for PM), and contact me on MSN messenger (or, by email.)

Comments are welcome,
Peace,
Tristan.

  • Latexxx
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #14
I'd personally drop mp3pro because its marginal presence is being overshadowed by he aac.

FhG instead of lame would be a rational choice because most net radios use FhG AFAIK.

Some years ago Microsoft sais that wma std @ 48 is almost cd quality. What a pity if we can't even prove that there are better codecs at 48 kbps than wma thus wma should be definately included.

I can't really see any point in including Apple's aac unless they manage add support for sbr and ps before the test starts. So my list would basicly be the following.
  • mp3 (fraunhofer?)
  • ogg vorbis (aotuv?)
  • wma standard (2-pass cbr or whatever their superfancy preset in wmenc is)
  • Nero he aac v.2
  • CT aac+ v.2
  • high anchor (mp3 @ 128?)

No point in using a low anchor such as gsm efr because the artifacts will be clearly present and a high anchor will give enough context.

  • rjamorim
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #15
Quote
I wasn't intending to start the test until the beginning of january, but, sure, I can wait.[a href="index.php?act=findpost&pid=349686"][{POST_SNAPBACK}][/a]


I personally believe you should give listeners at least one month of rest until you start a new test.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • Shade[ST]
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #16
Quote
Quote
,Dec 12 2005, 02:20 PM]I wasn't intending to start the test until the beginning of january, but, sure, I can wait.[a href="index.php?act=findpost&pid=349686"][{POST_SNAPBACK}][/a]


I personally believe you should give listeners at least one month of rest until you start a new test.
[a href="index.php?act=findpost&pid=349759"][{POST_SNAPBACK}][/a]

Any reason in particular?  The same people don't have to be participating, as long as a reasonable amount of members do..

  • dimzon
  • [*][*][*]
  • Banned
48 kbps listening test, pretest discussion thread
Reply #17
Quote
  • mp3 (fraunhofer?)
  • ogg vorbis (aotuv?)
  • wma standard (2-pass cbr or whatever their superfancy preset in wmenc is)
  • Nero he aac v.2
  • CT aac+ v.2
  • high anchor (mp3 @ 128?)

No point in using a low anchor such as gsm efr because the artifacts will be clearly present and a high anchor will give enough context.
[{POST_SNAPBACK}][/a]

don't forget about 3GP reference encoder:
[a href="http://forum.dbpoweramp.com/showthread.php?t=7107]http://forum.dbpoweramp.com/showthread.php?t=7107[/url]

  • naylor83
  • [*][*][*]
48 kbps listening test, pretest discussion thread
Reply #18
Quote
Quote
Quote
,Dec 12 2005, 02:20 PM]I wasn't intending to start the test until the beginning of january, but, sure, I can wait.[a href="index.php?act=findpost&pid=349686"][{POST_SNAPBACK}][/a]


I personally believe you should give listeners at least one month of rest until you start a new test.
[a href="index.php?act=findpost&pid=349759"][{POST_SNAPBACK}][/a]

Any reason in particular?  The same people don't have to be participating, as long as a reasonable amount of members do..
[a href="index.php?act=findpost&pid=349790"][{POST_SNAPBACK}][/a]


Yeah. And a 48 kbps test would be much easier, if you ask me...
davidnaylor.org

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
48 kbps listening test, pretest discussion thread
Reply #19
Quote
don't forget about 3GP reference encoder:
http://forum.dbpoweramp.com/showthread.php?t=7107


There are two things to take care when using it:

- It works only with 48 kHz input
- At 48 kbps, it will produce HE-AAC v1 Streams

Speaking of which, I will organize a small pre-test soon with HE-AAC vs. HE-AAC v2 at 48 kbps,  with two HE-AAC and two HE-AAC v2 implementations - so we can see which mode is better for each of the implementators (also would be good for reference for others encoding their material at 48 kbps).

Or, maybe we should include both HE-AAC and HE-AAC v2 in the 48 kbps test, but it would already be too much of the competitors?
  • Last Edit: 13 December, 2005, 04:43:32 AM by Ivan Dimkovic

  • Gabriel
  • [*][*][*][*][*]
  • Developer
48 kbps listening test, pretest discussion thread
Reply #20
My opinions is that it could be a good idea to first do an AAC low bitrate test before a low bitrate multiformat test.

This way, we would have less competitors per test, and perhaps more results (in the meaning of more information).

To me competitors would be (at least):
*Apple
*Nero
*CT commercial
*3gpp reference code (ie CT non commercial)

Is any product using some FhG encoder?

I think that bitrate of such test should be 32kbps, as this would allow to use HE-AAC v2 with the 3gpp code.

This way we would know which AAC encoder is better at low bitrates, but also how does the 3gpp reference code fares against commercial encoders.
This would also allow to reduce the number of contenders in a 48kbps multi-format test, as we would already have some info regarding AAC.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
48 kbps listening test, pretest discussion thread
Reply #21
But @Gabriel, do you think that the 32kbps resutls would extrapolate well to 48 kbps to be useful for the selection of the encoder?  I have a fear that some codecs >might< show completely different behavior in terms of ranking - not sure about that, but just maybe.

Also, there is another problem @48 kbps - while @32 kbps it is pretty clear that HE-AAC v2 will outperform HE-AAC v1 in all implementations, at 48 kbps this is starting to be a bit questionable as it has been shown on few internal tests of HE-AAC v1 vs HE-AAC v2 at 48 kbits/s.

I will definitely organise a separate small listening test of HE-AAC v1 and v2 codecs @48 kbps (and Ogg maybe) - by using of the 10-or-so average worst samples (combined average SDG) from the previous 128 and 64 kbps tests.

My intial plan was to use:

* 5 samples from the previous multiformat 64 kbps test, with worst average SDG
* 5 samples from the previous 128 kbps test, with worst average SDG

This will help chose the proper AAC mode at 48 kbps for the future 48 kbps multiformat listening, because there is an open issue whether Parametric Stereo (PS) or HE-AAC v2 is improving quality enough or the stereo-field regressions are too bad to justify the quality increase on some clips.  Testing these two modes side-by-side directly would help us make the choice for the future.

Also I think this will have very big value for the people considering HE-AAC/HE-AAC v2 for their 48 kbps encodings/streams - as it would show whether Parametric Stereo is good for that bitrate or not.  Currently there is not yet enough test data to make advisory - this test would definitely help

I planned to do this test soon (in 10 or so days maybe)- I will post it on HA, too - as well as methodology and the selection criteria for the samples which will be completely transparent and unbiased toward any codec/solution.

For sure there will be:

- CT HE-AAC v1 (aacPlus)
- CT HE-AAC v2 (aacPlus v2)
- Nero Digital - HE-AAC v1  [new codec]
- Nero Digital - HE-AAC v2  [new codec]

- And probably LAME @128 kbps as high-anchor
- LAME @48 kbps (edit: or, even better, AAC @48 kbps), as low anchor


But I am not sure whether to reveal the codec names in the test resutls (I would anyway do it for the orgnizers of the future tests), as I am sure that there will be doubts about the credibility here (as always  - and by not promoting any codec  I am trying to stay as neutral as possible - this test is about the v1 and v2 HE-AAC technology, not about the particular implementations.
  • Last Edit: 13 December, 2005, 07:29:27 AM by Ivan Dimkovic

  • stephanV
  • [*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #22
Quote
But I am not sure whether to reveal the codec names in the test resutls (I would anyway do it for the orgnizers of the future tests), as I am sure that there will be doubts about the credibility here (as always  - and by not promoting any codec  I am trying to stay as neutral as possible - this test is about the v1 and v2 HE-AAC technology, not about the particular implementations.
[a href="index.php?act=findpost&pid=349871"][{POST_SNAPBACK}][/a]

You are kidding right?

As long as results are reproducible (if we now, or in the soon future, can use an encoder to create the files that were tested)and the selection of samples is transparent and open for discussion, I can't see any reason why it would hurt the credibility of the test if the names where revealed.

The reason I'm bringing this up is two-fold:

1. The results will be less interesting and people might feel less inclined to participate. (assuming your test is open for us)

2. Technology and implementation are not always very separable. For example, you can't conclude that MP3 is a better technology than Vorbis or AAC by comparing LAME with a very bad AAC or Vorbis encoder. I don't know if this comparison holds also when it comes to SBR and PS, but it would be something to consider I think.
"We cannot win against obsession. They care, we don't. They win."

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
48 kbps listening test, pretest discussion thread
Reply #23
Quote
You are kidding right?


I wish - but reading the usual lisetning test rants I got that feeling

Anyway - I am discussing this matter with another well known name in audio compression,  as he initally proposed idea of teaming up for such test - if he agrees, we might do this together and make the test even better.

Quote
As long as results are reproducible (if we now, or in the soon future, can use an encoder to create the files that were tested)and the selection of samples is transparent and open for discussion, I can't see any reason why it would hurt the credibility of the test if the names where revealed.


Well...  I agree on all that, but nevertheless we might have some wise guys claiming that someone here "biased" the test towards something - which I clearly woudln't ever do.

Anyway  I also think a very high level of transparencu could be achieved if:

a) Encoders are available to everyone
b) Encoded bitstream verification is done by few people

c) Optionally, that the selection of the samples comes after the encoders were released (to avoid doubts that encoders were "produced" for the sample sent), or to provide clear and transparent way of selecting samples.

Quote
The reason I'm bringing this up is two-fold:

1. The results will be less interesting and people might feel less inclined to participate. (assuming your test is open for us)


Good point.  Taken

Quote
2. Technology and implementation are not always very separable. For example, you can't conclude that MP3 is a better technology than Vorbis or AAC by comparing LAME with a very bad AAC or Vorbis encoder. I don't know if this comparison holds also when it comes to SBR and PS, but it would be something to consider I think.


Actually we would compare two state-of-the-art HE-AAC v1 and v2 solutions  - so the test cannot be wrong with that regard.
  • Last Edit: 13 December, 2005, 08:18:37 AM by Ivan Dimkovic

  • Shade[ST]
  • [*][*][*][*][*]
48 kbps listening test, pretest discussion thread
Reply #24
Any word of yet as to how any of these tests are going to be set up?

I think Gabriel has a good idea at comparing various AAC contenders before a multiformat test;  Also, organizing many tests might make people feel more enthusiastic about the community, and thus conduce a retour en force of hydrogenaudio.

à la grâce de Dieu,
Tristan.