Skip to main content

Topic: Public Listening Test [2010] (Read 121274 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Larson
  • [*][*][*]
Public Listening Test [2010]
Reply #50
i've tested your script nao and thank you so much i have been looking for a solution for windows for such a long time. Conversion time was normal,fast and it did its job at Q127. Having also a macbook and using XLD i've compared a few files and they are identical. I confirm the "bug" that these AACs result in 0 kbps of bitrate in dbpoweramp tab,instead in Windows 7 explorer it's fine.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #51
Quote
Lame.exe and Nero.exe consume about 5% or less

Maybe your HDD is too slow or fragmented?

Added: try to set "Tools > Converter > Thread count" to 1.
  • Last Edit: 05 January, 2010, 12:51:45 PM by lvqcl

Public Listening Test [2010]
Reply #52
I don't see a thread count option under the Converter area of the fb2k preferences.  My hard drive isn't too slow or fragmented either.  In fact, it is an SSD drive so fragmentation has little to no affect and the speeds are above that of a 7200 RPM drive.  I do appreciate the work of nao, I don't know if that was clear or not in my first post regarding their tool.  In fact, I would use it as a viable encoding option if things weren't so slow.

Edit: my source files are ALAC.  I thought that might be an issue so I converted the 5 minute file to FLAC (q 5) and tried converting with foobar2000 again.  Still the same results, nearly 4 minutes to convert that 5 minute file.  I don't have access to my XP machine so I won't be able to test until tonight.  As previously said, Windows 7 explorer is fine displaying the correct bitrate on my end (just not dBpowerAMP).

Edit 2: I went ahead and tried dBpowerAMP's CLI encoder (version R13.2) under Windows 7.  Same exact results whether I encode from an ALAC, FLAC, or PCM WAV file.  However, dBpowerAMP will display the encoding speed (unlike foobar2000) which hovers at around 1.3x.
  • Last Edit: 05 January, 2010, 01:59:47 PM by kornchild2002

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #53
For Windows users, I made a tiny tool to access the QuickTime AAC encoder from the command-line.
qtaacenc

Can you specify also samplerate because all encoded files are 32 khz?

Great tool.

  • jido
  • [*][*][*]
Public Listening Test [2010]
Reply #54
Quote from: jido link=msg=0 date=
This is Hydrogenaudio, why not using LossyWAV for the high anchor? Let's get the word out!

Since people have been almost flooding this thread with proposals for encoders to be tested in a single test, it's time for me to give my 2 cents.

The question is what we want. Of course you could put all codecs of interest (LossyWav, iTunes CVBR and TVBR, nero 1.3.3 and 1.5.3, CT/Winamp, LAME, WMA, etc.) into one test. But trust me, if you do that, you will get mostly inconclusive results, i.e. waste a lot of listening effort, due to listener overload, as I already explained.

LossyWav's objective is to be transparent. AAC at 96 kbps usually is not transparent, so its objective is to be near-transparent, i.e. "as good as possible". If you want to check whether LossyWav is transparent, do a separate ABX test against the unprocessed original (or maybe an ABX-HR test including AAC at 256 kbps or so, if you want.) Then people can focus on whether the codecs under test really are transparent, without being distracted by at the same time having to evaluate the quality of lower-bit-rate codecs.

The idea was not to test LossyWAV transparency, but to download smaller files. BTW that could also apply to the proposed 7 kHz-filtered low anchor.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #55
A 7-kHz anchor (and 48-kbps LC) should be about mid-way between "very bad quality" and the quality of the 96-kbps LC encoders.

Are you sure 48 kbps LC is not too exaggerated? Do you expect any contender to be worse or as bad as 64 kbps?
LC-AAC 56 kbps or 8-khz anchor could be a good compromise. Both looks good for me.


As poll indicates
there are two widely used AAC encoder on HA.
Coding Technologies to be dropped as it's only CBR and not so popular here.

It's clear now at least for me speaking about propsal list:
1. Nero
2. Apple
3. FH
4. Divx

The idea was not to test LossyWAV transparency, but to download smaller files. BTW that could also apply to the proposed 7 kHz-filtered low anchor.

I think it's not good idea from point of view of credibility of final test. There will be rumors like "not 100% lossless references"
I'm completely against of lossyWAV here.
  • Last Edit: 05 January, 2010, 06:52:40 PM by IgorC

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #56
For Windows users, I made a tiny tool to access the QuickTime AAC encoder from the command-line.
qtaacenc

Many thanks.
But, it adds samples at the beginning and decoded .m4a file has more samples than original file. Is it possible to make gapless files?

  • Enig123
  • [*][*][*]
Public Listening Test [2010]
Reply #57
I might have missed something, but where to get the Fraunhofer encoder?
  • Last Edit: 05 January, 2010, 08:19:30 PM by Enig123

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #58
...
Agreed. But for completeness sake: Fraunhofer is currently finalizing quality tunings on their encoder which have been going on for about two years. Release is scheduled for end of January. If there is any interest, I can ask if it's possible to provide an evaluation encoder for this test.

Chris

Chris
If I don't reply to your reply, it means I agree with you.

  • nao
  • [*][*]
Public Listening Test [2010]
Reply #59
Updated qtaacenc.

dBpowerAMP reports the bitrate as being 0kbps.

Fixed.

Can you specify also samplerate because all encoded files are 32 khz?

Now possible with the --samplerate option. "--samplerate keep" or "--samplerate 44100" meets your requirement. With the default setting, the optimum samplerate is automatically chosen according to the bitrate and quality.

  • tedgo
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #60
@nao
Thanks for your qtaacenc .
I love it!
Maybe you better should've offered it in an own thread instead of hiding it in this "public listening test" thread.

Unfortunately the encoded files aren't gapless...
So i've to ask the same as Ivqcl: Is it possible to make gapless files?

  • Sylph
  • [*][*][*][*]
Public Listening Test [2010]
Reply #61
I might have missed something, but where to get the Fraunhofer encoder?


Nowhere.

Only in MAGIX MP3 Maker and I'd have to check whether it has a VBR mode...

Public Listening Test [2010]
Reply #62
Maybe you better should've offered it in an own thread instead of hiding it in this "public listening test" thread.


Maybe a mod would be nice enough to take the posts dealing with nao's command line tool out of the listening test thread and put them in a new one.  That way we can continue to help test qtaacenc for nao without filling up this thread with posts that aren't really about the listening test.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #63
I don't think that a <128kbit/s listening test makes any sense nowadays. Nobody is using those bitrates and results cannot be extrapolated into regions of general transparency.

One encoder might be a worse low bitrate performer, but still have much better resilience against killer samples than another, once the bitrate passes a certain barrier. And since we have reached a point were most popular encoders are totally transparent for most music at 128-192kbit/s, we should now focus onto what encoder knows less killer samples. A listening test in its traditional form is not suited to evaluate that. The limited, initial choice of test samples will determine the outcome.

A [a href='index.php?act=findpost&pid=678439']lengthier[/a] version of this...

  • halb27
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #64
... we should now focus onto what encoder knows less killer samples. A listening test in its traditional form is not suited to evaluate that. The limited, initial choice of test samples will determine the outcome. ...

Yes, the value of the outcome of such listening tests is often rated too high IMO. Especially at such a pretty low bitrate. Especially as many people just take the average score of each encoder to get a quality order for the encoders.
But: Despite this listening tests do have a value IMO, though a restricted one.

As for your approach: this would give valuable information, though restricted information as well because the universe of killer samples is not known.
lame3995o -Q1

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #65
... we should now focus onto what encoder knows less killer samples. A listening test in its traditional form is not suited to evaluate that. The limited, initial choice of test samples will determine the outcome. ...

Yes, the value of the outcome of such listening tests is often rated too high IMO. Especially at such a pretty low bitrate. Especially as many people just take the average score of each encoder to get a quality order for the encoders.
But: Despite this listening tests do have a value IMO, though a restricted one.

As for your approach: this would give valuable information, though restricted information as well because the universe of killer samples is not known.

Well, an educated audio codec developer knows a lot about the universe of killer samples for his/her codec. I work as an AAC developer, and I've seen about two dozen high-bitrate killer samples until now. They all fall into very few specific categories of sounds, hence you could look for similar sounds (with high chances that they'll be killer samples as well), or even design your own. Today, one decade after version 1.0 of the encoder I work on, many initial killer samples are not killer samples any more. Of course, some still are, simply because AAC is not perfect, or because appropriate input analysis would make the encoder unacceptably slow. For example, you will never find an AAC encoder which at 128-160 kbps stereo will give you a transparent encoding of the emese sample.

So, rpp3po, I don't understand what you mean by "A listening test in its traditional form is not suited to evaluate that". I think it is. You just need to know the abovementioned categories.

Chris
  • Last Edit: 08 January, 2010, 03:21:55 PM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #66
So, rpp3po, I don't understand what you mean by "A listening test in its traditional form is not suited to evaluate that". I think it is. You just need to know the abovementioned categories.


With "listening test in its traditional form" in this context I mean public listening tests as the one in discussion. A public listening test like this has usually a limited test of samples. And the more 'emesesk' samples the set includes the worse AAC will look in comparison to MP3 and vice versa. So the outcome is severely influenced by the choice of samples and thus of limited use as a tool to evaluate which is best overall. The encoder with the least problematic samples in the set wins. You can intentionally exclude problematic samples from the test, but what are you going to get then? Good results for all encoders for most music. But we already know that. My point is, isn't it time to move on with testing to compare which encoders have the lowest probability of failing over broad collections of music, even if that would be harder to accomplish than with traditional 8-16 sample shootouts?

For the individual developer, who knows its encoder inside out, a "traditional" listening test with select samples can be of great value, no question.
  • Last Edit: 08 January, 2010, 04:48:20 PM by rpp3po

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #67
And the more 'emesesk' samples the set includes the worse AAC will look in comparison to MP3 and vice versa.

Are you sure? Just because a certain encoder has problems with "emesesk" samples doesn't mean that every AAC encoder has problems with them. Btw, to me, MP3 (LAME) doesn't do too well with such samples, either.
Quote
You can intentionally exclude problematic samples from the test, but what are you going to get then? Good results for all encoders for most music. But we already know that. My point is, isn't it time to move on with testing to compare which encoders have the lowest probability of failing over broad collections of music, even if that would be harder to accomplish than with traditional 8-16 sample shootouts?

Being a developer, all I care about is problematic samples  Sure, I agree, "lowest probability of failing over broad collections of music" should be an encoder's ultimate goal. But how else would you test for that than via 8-16 sample shootouts? How about asking our expert listeners (guruboolez, /mnt, IgorC, sauvage78, etc.) for the most critical items they could find in their high-bitrate tests, and put all those items in one public test (bitrate to be defined)? Wouldn't that be a good starting point?

Chris
If I don't reply to your reply, it means I agree with you.

  • MichaelW
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #68
A question from the peanut gallery.

Do different codecs have specifiably different types of killer samples?

My thought is that we know that a number of good codecs give excellent results for most music, but it might be possible, perhaps, to give specific recommendations matching codec to genre (codec A is good for harpsichord, for example, but falls over on distorted guitars, on which codec B does better).

IF this were possible, it might be more practically useful for listeners than another very tight general comparison of good codecs over "average" music.

Edit: punctuation
  • Last Edit: 08 January, 2010, 05:44:25 PM by MichaelW

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #69
A question from the peanut gallery.

Do different codecs have specifiably different types of killer samples?

Yes, there are many of this kind of sampls.

My thought is that we know that a number of good codecs give excellent results for most music, but it might be possible, perhaps, to give specific recommendations matching codec to genre (codec A is good for harpsichord, for example, but falls over on distorted guitars, on which codec B does better).

IF this were possible, it might be more practically useful for listeners than another very tight general comparison of good codecs over "average" music.

Edit: punctuation

The problem of light or killer samples can be eliminated if randomizer will be used.
http://www.hydrogenaudio.org/forums/index....st&p=678087


I'm fine with nao's Apple TVBR encoder.

Availability of encoders
I think any AAC encoder can be tested while it's available at least in one commercial product.
No problem for Apple,Nero and CT(Winamp).
Divx encoder is still beta but every person can download it after registration on Divx home page.

Chris, what about FH encoder?
At least demo version with limited time will be fine. This way we can check that bitrate and quality settings are the same for all samples.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #70
The encoder with the least problematic samples in the set wins. You can intentionally exclude problematic samples from the test, but what are you going to get then? Good results for all encoders for most music. But we already know that. My point is, isn't it time to move on with testing to compare which encoders have the lowest probability of failing over broad collections of music, even if that would be harder to accomplish than with traditional 8-16 sample shootouts?

For the individual developer, who knows its encoder inside out, a "traditional" listening test with select samples can be of great value, no question.

It's just different approach. Good one actually.

Somebody would say that testing killer samples wouldn't be representative but the same way light samples aren't representative neither.

I'm in favor to rise bitrate to 128 kbps and include more difficult samples randomly (and don't include samples which are only difficult for one specific competitor). Emese-like samples are horrible at this bitrate.
  • Last Edit: 11 January, 2010, 09:31:32 PM by IgorC

  • halb27
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #71
[...I'm in favor to rise bitrate to 128 kbps and include more difficult samples randomly (and don't include samples which are only difficult for one specific competitor). Emese-like samples are horrible at this bitrate.

I second that. Will get a result which is more relevant for practical purposes. And in case it comes out that there are encoders (may be all of them) which are perfect or near-perfect except for say artificial music this  would be a fine result.
As for samples which are difficult for a specific encoders however it would be helpful to have a list of known such samples.
  • Last Edit: 12 January, 2010, 02:45:38 AM by halb27
lame3995o -Q1

  • /mnt
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #72
The encoder with the least problematic samples in the set wins. You can intentionally exclude problematic samples from the test, but what are you going to get then? Good results for all encoders for most music. But we already know that. My point is, isn't it time to move on with testing to compare which encoders have the lowest probability of failing over broad collections of music, even if that would be harder to accomplish than with traditional 8-16 sample shootouts?

For the individual developer, who knows its encoder inside out, a "traditional" listening test with select samples can be of great value, no question.

It's just different approach. Good one actually.

Somebody would say that testing killer samples wouldn't be representative but the same way light samples aren't representative neither.

I'm in favor to rise bitrate to 128 kbps and include more difficult samples randomly (and don't include samples which are only difficult for one specific competitor). Emese-like samples are horrible at this bitrate.


That was the problem with the recent Mp3 test. Since some of the killer samples on test, such as the Final Fantasy sample, seemed to be targeted against LAME 3.97.

I would recommend using a couple of Kraftwerk tracks such as The Robots (0:20 - 0:50 or 4:10 - 4:40) and Musique Non Stop (0:10 - 0:40). Since those tracks produce very obvious artifacts at 128 with every AAC encoder.
  • Last Edit: 12 January, 2010, 03:58:25 PM by /mnt
"I never thought I'd see this much candy in one mission!"

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #73
/mnt
Thank you for data.

As poll indicates there is more interest in 128 kbps test.

Bad news most probably FH codec won't be available for public and because of this it won't get into the test.
As we moved to 128 kbps the question of high anchor is opened again.
Also CT encoder can be included into the test despite of its CBR but it's very popular (Winamp,etc.)

Proposal list  (3 competitors + 1 high anchor +1 low anchor) :
1. Apple TVBR
2. Nero
3. Winner of pre-test (CT vs Divx)

High anchor (?): LAME 3.98.2 -V2
Low anchor(?): LC-AAC 64 kbps.
  • Last Edit: 14 January, 2010, 12:37:31 PM by IgorC

  • halb27
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #74
I'd welcome to have winamp's CT AAC encoder included.
As a high anchor I guess Lame -V2 can be worse for some problem samples than a good AAC encoder @128 kbps. From previous discussion I thought lossless was to be the high anchor.
As a low anchor LC-AAC @64 kbps is fine IMO.
  • Last Edit: 14 January, 2010, 03:11:01 PM by halb27
lame3995o -Q1