Heterogeneous listening test. Pretest thread.

Topic: Heterogeneous listening test. Pretest thread. (Read 20383 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Heterogeneous listening test. Pretest thread.

2006-02-02 23:37:30

While access to SoundExpert raw statistics is not ready yet, I plan to add SBC (Bluetooth, 375 kbit/s, joint stereo) codec to the system. Now I’m looking for interesting codec contenders for this test. My proposals are:

1. Lame 320 CBR
2. AAC 320 CBR or VBR

Actually SE testing methodology allows comparing codecs of different bitrates according to single impairment scale, so it is possible to include, for example, AAC 64, HE-AAC+PS 24 or what’s ever interesting to this test. The total number of contenders supposed to be 3-5. During the listening test (2 weeks) SE will give out test files of the above contenders only in order to get results more quickly.

The only obligatory participant is SBC. All others are up to you. Thank you.

EDIT: link added

EDIT 02.26.2006:

320+ Kbit/s listening test is opened now. List of contenders:

- aac CBR@327.2 (Winamp 5.2)
- aac+ CBR@327.0 (Winamp 5.2)
- mp3 CBR@320.2 (Lame 3.90.3)
- mp3 CBR@320.2 (Lame 3.97b2)
- SBC CBR@372.1 (Bluetooth)
- wma 9.1 CBR@322.8

This test will last for about two weeks during which test files of the above contenders only will be available for downloading. Volunteers are invited.

Heterogeneous listening test. Pretest thread.

Reply #1 – 2006-02-03 00:58:37

who do you expect to hear the difference at 320??

Heterogeneous listening test. Pretest thread.

Reply #2 – 2006-02-03 01:16:28

Quote

who do you expect to hear the difference at 320??
[{POST_SNAPBACK}][/a]

Using standard testing methodology - nobody. Using SoundExpert one - everybody. Detailes are [a href="http://www.soundexpert.info/prozone.htm]here[/url].

Heterogeneous listening test. Pretest thread.

Reply #3 – 2006-02-03 09:21:04

Quote

While access to SoundExpert raw statistics is not ready yet, I plan to add SBC (Bluetooth, 375 kbit/s, joint stereo) codec to the system. Now I’m looking for interesting codec contenders for this test. My proposals are:

1. Lame 320 CBR
2. AAC 320 CBR or VBR

Actually SE testing methodology allows comparing codecs of different bitrates according to single impairment scale, so it is possible to include, for example, AAC 64, HE-AAC+PS 24 or what’s ever interesting to this test. The total number of contenders supposed to be 3-5. During the listening test (2 weeks) SE will give out test files of the above contenders only in order to get results more quickly.

The only obligatory participant is SBC. All others are up to you. Thank you.

EDIT: link added
[{POST_SNAPBACK}][/a]

Wonderful,
As for Lame, I'm begging you to use several versions which behave totally different:
- current Lame version of course
- Lame 3.90.3 alt-preset insane as for the famous alt-preset setting
- Lame 3.90.3 -b320 -h.

3.90.3 -b320 -h uses gpsycho psymodel which should come out very favorably with respect to pre-echo and transient behavior.
3.90.3 api means nspsytune but with a transient behavior presumed better than with current lame.
Current lame should be in the test of course as well.

For the background concerning the transient behavior see Gabriel's problem sample thread. Or do a quick abxing with my contribution there: [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=39334]trumpet sample (trumpet.flac)[/url].

Heterogeneous listening test. Pretest thread.

Reply #4 – 2006-02-03 10:23:13

Quote

As for Lame, I'm begging you to use several versions which behave totally different:
- current Lame version of course
- Lame 3.90.3 alt-preset insane as for the famous alt-preset setting
- Lame 3.90.3 -b320 -h.[a href="index.php?act=findpost&pid=361629"][{POST_SNAPBACK}][/a]

I’m not sure it would be interesting to compare lames’ versions only. Maybe two lames (say, lame 3.97b2 --preset insane and lame 3.90.3 -alt-preset insane) are enough and the fourth competitor to be AAC 320 CBR?

Heterogeneous listening test. Pretest thread.

Reply #5 – 2006-02-03 11:36:43

Quote

Quote
As for Lame, I'm begging you to use several versions which behave totally different:
- current Lame version of course
- Lame 3.90.3 alt-preset insane as for the famous alt-preset setting
- Lame 3.90.3 -b320 -h.[a href="index.php?act=findpost&pid=361629"][{POST_SNAPBACK}][/a]

I’m not sure it would be interesting to compare lames’ versions only. Maybe two lames (say, lame 3.97b2 --preset insane and lame 3.90.3 -alt-preset insane) are enough and the fourth competitor to be AAC 320 CBR?
[a href="index.php?act=findpost&pid=361638"][{POST_SNAPBACK}][/a]

Ooooh, sorry, my proposal was towards Lame participation only.
Sure other competitors are highly welcome.

I'd be very pleased if you could arrange to have all of the 3 Lame candidates in the test because of the special interest in mp3 which still exists, because of Lame as the most interesting mp3 encoder, and because of the expected different behavior of the three candidates.

I know three times Lame doesn't look good optically speaking. So if this is really too much can you please use 3.90.3 -b320 -h as this IMO is the most interesting candidate in comparison to current Lame because of gpsycho (very good at pre-echo and transient).

Heterogeneous listening test. Pretest thread.

Reply #6 – 2006-02-03 16:03:53

Quote

So if this is really too much can you please use 3.90.3 -b320 -h as this IMO is the most interesting candidate in comparison to current Lame because of gpsycho (very good at pre-echo and transient).
[a href="index.php?act=findpost&pid=361647"][{POST_SNAPBACK}][/a]

BTW, are there many people still use “- preset insane” in practice or it’s just an academic interest?

Heterogeneous listening test. Pretest thread.

Reply #7 – 2006-02-03 18:23:00

Quote

BTW, are there many people still use “- preset insane” in practice or it’s just an academic interest?

I don't think a lot of people are using 320 kbps or similar bitrate at all because most of the time the bitrate range around say 150 kbps yields practically perfect results with current lame. Nonetheless from time to time you find posts here from people who want to use mp3 at highest quality possible, for instance on their home stereo equipment.
File size ususally doesn't matter to these people, it's mp3 that matters because of usability on their equipment.
Despite Lame's success at moderate bitrate there is room for improvement using high bitrate. Current Lame has a weakness with certain kinds of trumpet or saxophone-like sounds even at high bitrate (see the trumpet sample I gave), and I found with these samples 3.90.3 (or 3.91) high bitrate --abr x or -b x is better than the corresponding 3.90.3 alt-presets (I admit at 320 kbps I cannot hear a difference). While difference between 3.90.3 abr x and --alt-preset x is audible to me at around 224 kbps but not at bitrates beyond 256 kbps I can clearly distinguish current Lame 320 kbps which is not transparent with the trumpet sample.

Of course this is not the entire story cause it applies only to these trumpet-like sounds. It might happen that current Lame is to be preferred so much on other kind of music that these things don't count very much. I do not expect it but who knows.
That's why it is of vital interest for those who want to use high bitrate for the sake of utmost quality to learn about real performance, and your kind of tests are great at finding out differences that are unheard otherwise. Apart from showing up real problematic behavior it gives a feeling of a safety margin.

Heterogeneous listening test. Pretest thread.

Reply #8 – 2006-02-05 23:57:27

Quote

Nonetheless from time to time you find posts here from people who want to use mp3 at highest quality possible, for instance on their home stereo equipment. File size ususally doesn't matter to these people, it's mp3 that matters because of usability on their equipment...

...That's why it is of vital interest for those who want to use high bitrate for the sake of utmost quality to learn about real performance, and your kind of tests are great at finding out differences that are unheard otherwise. Apart from showing up real problematic behavior it gives a feeling of a safety margin.
[a href="index.php?act=findpost&pid=361706"][{POST_SNAPBACK}][/a]

Ok, I see. So for now candidates list looks like this:

1. SBC (Bluetooth, 375 kbit/s, joint stereo)
2. lame (3.97b2) --preset insane
3. lame (3.90.3) -b320 –h
4. LC AAC 320 (QT)
5. CT aacPlus High Bitrate Encoder v1.2 320 (Winamp)

Heterogeneous listening test. Pretest thread.

Reply #9 – 2006-02-21 22:45:48

At last, six contenders are ready for testing:

SBC CBR@372.1 (Bluetooth) - Subband codec for A2DP profile, CBR, 372.1 kbit/s FBR
CODER: SBC Encoder LIB Version 1.5 (Philips)
- usage: sbc_encoder.exe -v -j ref.wav
- 44100 Hz Joint Stereo
DECODER: SBC Decoder LIB Version 1.5 (Philips)

mp3 CBR@320.2 (Lame 3.97b2) - MPEG-1 Layer 3 CBR, 320.2 kbit/s FBR
CODER: Lame 3.97b2
- usage: lame --preset insane ref.wav out.mp3
- 44100 Hz Joint Stereo
DECODER: MAD 0.15.2b

mp3 CBR@320.2 (Lame 3.90.3) - MPEG-1 Layer 3 CBR, 320.2 kbit/s FBR
CODER: Lame 3.90.3
- usage: lame -b320 -h ref.wav out.mp3
- 44100 Hz Stereo
DECODER: MAD 0.15.2b

wma 9.1 CBR@322.8 - Windows Media Audio 9.1 (Standard) CBR, 322.8 kbit/s FBR
ENCODER: MS Windows Media Encoder 9.00.00.2980
- Mode: CBR
- Audio format: 320kbps, 44 kHz, stereo CBR
DECODER: MS Windows Media Encoder 9.00.00.2980

aac CBR@327.2 (Winamp 5.2) - MPEG-2 AAC Low Complexity CBR, 327.2 kbit/s FBR
CODER: LC-AAC Encoder v1.2 from Winamp 5.2 build 449 Beta
- CD Ripping
- Bitrate: 320kbps
- Channel Mode: Independent Stereo
- Bitstream Options: MPEG-4
- 44100 Hz
DECODER: Winamp 5.2 build 449 Beta
- Disk Writer plug-in v2.1

aac+ CBR@327.0 (Winamp 5.2) - MPEG-2 AAC High Efficiency CBR , 327.0 kbit/s FBR
CODER: aacPlus (HE-AAC) High Bitrate Encoder v1.2 from Winamp 5.2 build 449 Beta
- CD Ripping
- Bitrate: 320kbps
- Channel Mode: Independent Stereo
- Bitstream Options: MPEG-4
- 88200 Hz
DECODER: Winamp 5.2 build 449 Beta
- Disk Writer plug-in v2.1
- Convert to format: PCM 44100Hz; 16bit; Stereo

iTunes/QT aac encoder showed a serious flaw with glockenspiel sample even at 320 kbit/s, so I decided not to include the encoder to this test (till the next version at least).

I’m still not sure about Nero LC-AAC 320 CBR. I’ve heard about new version coming soon. In such a case I would prefer to add the new encoder later as well.

Heterogeneous listening test. Pretest thread.

Reply #10 – 2006-02-21 23:14:11

Quote

While access to SoundExpert raw statistics is not ready yet, I plan to add SBC (Bluetooth, 375 kbit/s, joint stereo) codec to the system. Now I’m looking for interesting codec contenders for this test. My proposals are:

1. Lame 320 CBR
2. AAC 320 CBR or VBR

Actually SE testing methodology allows comparing codecs of different bitrates according to single impairment scale, so it is possible to include, for example, AAC 64, HE-AAC+PS 24 or what’s ever interesting to this test. The total number of contenders supposed to be 3-5. During the listening test (2 weeks) SE will give out test files of the above contenders only in order to get results more quickly.

The only obligatory participant is SBC. All others are up to you. Thank you.

EDIT: link added
[a href="index.php?act=findpost&pid=361554"][{POST_SNAPBACK}][/a]

You might consider WMA Pro at the same rate as the other signals, as well.

Heterogeneous listening test. Pretest thread.

Reply #11 – 2006-02-21 23:33:41

Quote

You might consider WMA Pro at the same rate as the other signals, as well.[a href="index.php?act=findpost&pid=366121"][{POST_SNAPBACK}][/a]

I decided to add wma std first because of broader support in hardware players and also 320 is the highest possible bitrate for all contenders (except for SBC). So maybe later.

Heterogeneous listening test. Pretest thread.

Reply #12 – 2006-02-21 23:53:53

would it be possible to explain simply, exactly how the SE methodology enables every person to detect a difference? I don't understand the whitepaper.

Heterogeneous listening test. Pretest thread.

Reply #13 – 2006-02-22 00:26:32

Quote

would it be possible to explain simply, exactly how the SE methodology enables every person to detect a difference? I don't understand the whitepaper.
[a href="index.php?act=findpost&pid=366127"][{POST_SNAPBACK}][/a]

Simple answer: the difference is amplified.

Not so simple answer: for every (natural) sample tested a set of (at least) 3 samples created with artifacts amplified to different extent. The samples of this set are tested as usual in SE listening tests. Final grade of the natural sample is calculated analytically on basis of those listening tests.

Hope this was helpful.

Notice