Skip to main content

Topic: Public Listening Test [2010] (Read 121095 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #250
Before I purchased Audition, I used GoldWave to cut and edit samples: www.goldwave.com/

I think now that the looping issue seems to have disappeared, we should revert to the intermediate decision to run CVBR at 124 kb/s or so. Otherwise, we'd have to raise the bit rate for TVBR as well, right? IIRC, halb27 or rpp3po offered to encode using iTunes on a Mac?

Edit: Found the original discussion here. Since I can't find anything previous of that sort: rpp3po, do you volunteer to CVBR-encode the concatenated test sample when it becomes available?

Chris
  • Last Edit: 27 March, 2010, 03:42:09 PM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #251
Yes, CVBR 124 kbps is better to be compared to --TVBR 60  and Nero q041.
Who is MAC user?

rpp3po was going to encode to CVBR 124 but he isn't with us anymore. http://www.hydrogenaudio.org/forums/index....showtopic=79075
He was great guy here.
  • Last Edit: 27 March, 2010, 03:44:24 PM by IgorC

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #252
Oh my, I didn't see that thread! That's so sad  Thanks for pointing me to it!

Bit off-topic, but I still have some messages from him in my inbox. It was about this delay removal tool Synchrotron. Maybe we should give that tool a try as a tribute to/in memory of him (or however you say that in English...)?

Chris
If I don't reply to your reply, it means I agree with you.

  • .alexander.
  • [*][*]
Public Listening Test [2010]
Reply #253
I think now that the looping issue seems to have disappeared.


Well then, it's a good time to make bets about (bitrate overhead) vs (quality scores) correlation coefficient.

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #254
Fyi #1, Apple seems to have updated its encoder. See nao's version number report. Let's test this new version even if it turns out to deliver bit-identical encodings for our target bit rate.

Fyi#2, we are still open for suggestions on which item to use as test sample #16. If you know a sample which is a piece-of-cake ABX at 128 kb and which has not been mentioned in that thread, please don't hesitate to let us know about it. The deadline is today... well, maybe the end of the week

Thanks,

Chris
If I don't reply to your reply, it means I agree with you.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #255
The choice of samples is completed now.

The problem is bitrate.

1. Is anybody MAC user here? We need shifted bitrate for CVBR --124 and it's possible only under MAC.

2. As already all people know we choose CBR (there is no VBR that would feet into average 130 kbps) for Divx but it doesn't produce 128 kbps neither. The real bitrate is 125-125.5 kbps.  It's at least 2.7% less compared to other encoders. Is it ok for you or should we do not include Divx encoder into test?

Suggestions are welcomed.
  • Last Edit: 13 April, 2010, 11:04:52 PM by IgorC

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #256
At this moment we have encoded samples with:
1. QuickTime 7.6.6 . TVBR 60 Highest. Here it produces the same output as older 7.6.5
2. Nero 1.5.4.0 -q 0.41 (VBR)
3. CT encoder 8.2.0 (Winamp 5.5.572 (January 13, 2010)  dll & MediaCoder GUI) 130 CBR
  • Last Edit: 13 April, 2010, 11:39:40 PM by IgorC

  • halb27
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #257
... Divx but it doesn't produce 128 kbps neither. The real bitrate is 125-125.5 kbps.  It's at least 2.7% less compared to other encoders. Is it ok for you or should we do not include Divx encoder into test?

Suggestions are welcomed.

I wouldn't care about a 2.7% difference.
lame3995o -Q1

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #258
Me neither. But for fairness sake I will evaluate the DivX encoder against a 128-kb bitstream of the CT 8.2 encoder (true bitrate about 126 kb).

Igor, please prepare/send the DivX and CT 128-kb bitstreams so I can do the pre-test. Thanks!

Edit: Found someone at the office with a full version of Winamp, and we noticed that it's possible to create both MP4- and ADTS-formatted LC bitstreams (format conversion in the playlist library). See this tutorial. So actually, it won't be necessary to make a 130-kb encoding to compensate for ADTS overhead.

Chris
  • Last Edit: 14 April, 2010, 09:11:00 AM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #259
DivX -v 5 = 136.3 kbps
iTunes 128 vbr = 133.6 kbps
QT tvbr 65 highest = 130.2 kbps
Nero -q 0.42 = 132.9 kbps
CT (Winamp 5.572) = 128.0 kbps
   
average   = 132.2 kbps  (=> test "target" = 132 kbps)

The bitrates are approximately within +- 3%.

My reasoning - the bitrate tests in Excel format (I tested my usual 25+25 complete tracks):
http://www.hydrogenaudio.org/forums/index....st&p=700607


Some notes:

-- Nero -q 0.42 would be closer to the average than -q 0.41 with my test set. (I tested 0.41, 0.42 and 0.43)

-- For QT CVBR, please don't consider using anything else than a setting that can be found in iTunes.

-- Winamp 5.572 "MP4 LC-AAC 128 kbps" produces 128 kbps files according to foobar. (not 130 kbps)

-- Use the real Winamp program for encoding "CT/Winamp MP4 LC-AAC", not MediaCoder.

-- DivX -b 128 produced 124.8 kbps with my test set - propably caused by this:
Quote
Known issues for DivX Plus HD AAC Encoder Beta 1:

The CBR mode does not pad frames if the input audio complexity does not require using all of the available bitrate.  ...
(a quote from: http://labs.divx.com/node/11682)


Sorry, I don't have time to write more now. Testing and creating the Excel sheet took too much time already and I am late from something else...


EDIT

A quick edit though,

-- It would be interesting to see how DivX VBR works even though its bitrates are on the "high side" with some complex tracks. IMO, its behavior is more like true VBR than Nero's or QT's "true" VBR behavior. It is more in the line with LAME, Vorbis or Musepack VBR behavior. For comparison I added LAME -V5 to my Excel sheet.

-- I used the latest versions of all SW (checked the versions today).

-- Fixed the "quote" link.

-- I have posted some useful instructions for the DivX encoder here: http://www.hydrogenaudio.org/forums/index....st&p=686087. Additional info can be found from the above "quote" link.
  • Last Edit: 14 April, 2010, 10:49:21 AM by Alex B

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #260
Alex B
A big thank you.  I'm totally agree with all your statements and your setting will be used if everybody agree.
Only one thing. I've sent a PM to Benski (Winamp's developer) asking  if bitrate shifting  132 for CT (MediaCoder) is ok. While real bitrate for 132 is actually 133. 133 is average for other 4 encoders.

Chris, I will send you all samples as soon as we reach agreement about CT encoder.
Sorry.
  • Last Edit: 14 April, 2010, 12:51:04 PM by IgorC

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #261
Thanks, Alex!

Igor, I already got the following bitstreams:

  • Apple QT 7.6.6 TVBR. Got it from you.
  • Apple iTunes CVBR 128kb. I installed iTunes on Win32 on Monday, encoded it by hand from the GUI.
  • nero 1.5.4.0 -q 0.42, I also checked 0.41, but 0.42 is closer to the Apple encoders on our test set.
  • Dolby/CT 8.2.0 128kb MP4. Encoded by hand via Winamp 5.572. See my previous post. I think we should use that one.

So the only thing missing is the DivX VBR bitstream (I agree with Alex, VBR is more interesting in our case).

If you want I can forward you my bitstreams for cross-checking tomorrow.

Chris
If I don't reply to your reply, it means I agree with you.

  • benski
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #262
Alex B
A big thank you.  I'm totally agree with all your statements and your setting will be used if everybody agree.
Only one thing. I've sent a PM to Benski (Winamp's developer) asking  if bitrate shifting  132 for CT (MediaCoder) is ok. While real bitrate for 132 is actually 133. 133 is average for other 4 encoders.

Chris, I will send you all samples as soon as we reach agreement about CT encoder.
Sorry.


Sure, setting the bitrate manually shouldn't result in any problems.  132kbps won't be as "well tuned" as 128kbps but the quality should always be nominally better than 128kbps

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #263
Good, then we take Winamp 132kb MP4 and DivX -v5. Once I receive the bitstreams from Igor, we will do the pre-test. If anyone else is interested in joining, please send a personal message to me or Igor. A few extra listeners won't hurt.

Chris
If I don't reply to your reply, it means I agree with you.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #264
ok, I will send you the files in a few minutes. CT 132 and Divx -v5
  • Last Edit: 14 April, 2010, 03:18:18 PM by IgorC

  • Sylph
  • [*][*][*][*]
Public Listening Test [2010]
Reply #265
Question: is anyone interested in adding the — if I have the right info — Liquid Audio AAC now in AVS Audio Converter 6 to this listening test?

And Fraunhofer's from MAGIX is also out of the picture?

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #266
Impossible.
Too late. The sample are already defined and no any other encoder will be accepted.

Before I will start to argue I would like to ask you. Where were you during 3 months of discussions? You knew perfectly about it.

But even if it wasn't late those encoders still had very tiny possibility.
Reasons:
1. Did you ABX Liquid Audio AAC encoder yourself? It's awful as there is no any simple lowpass. No talk about something more advanced tools. 
2. Sizes of packages of MAGIX. 100-200 MB just for AAC encoder?  Apple is exclusion as it's well know here and personally I think it's reasonable price for high quality encoder.
3. Those combos encoders OGG-AAC-MP3-AVR-BLA encoders.... etc etc.. have no serious encoders.
4. AAC from RealPlayer was excluded as it has adware (plus not so good quality CBR encoder) and today it's shame for company who put adware to their products. Maybe MAGIX has no direct adware but it's somehow get some extra control in OS which isn't funny.
  • Last Edit: 15 April, 2010, 01:09:45 PM by IgorC

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #267
Relax, Igor  My 2 cents:

1. We had defined a limit on the number of encoders in the test. This limit is reached. So we like to test only the best encoders available. I trust Igor when he says that the Liquid Audio encoder is not one of the best.

2. Since the encoder version used in the Magix software, the Fraunhofer encoder has undergone some considerable quality tuning. We decided not to include the latest Fraunhofer encoder because IIRC it's currently not availble in a cheap software or as a demo version "for everyone" (and because I'm helping to coordinate the test).

If you like to know how the current Fraunhofer encoder sounds, I'm sure there will come a time when you can listen to it through some software.

Chris
If I don't reply to your reply, it means I agree with you.

  • Sylph
  • [*][*][*][*]
Public Listening Test [2010]
Reply #268
Wow. One wants to help and gets attacked.  Wow. I have MAGIX so it wouldn't cause me any additional effort to encode.

And I actually didn't think you had any knowledge about AAC and any such stuff when this thing started so I thought it was going to flop. But I see C. R. Helmrich joined with many others and it's going great!

  • Sylph
  • [*][*][*][*]
Public Listening Test [2010]
Reply #269
If you like to know how the current Fraunhofer encoder sounds, I'm sure there will come a time when you can listen to it through some software.

Chris


I think that time will not come, sadly. With every day that passes, it seems less likely a Fraunhofer AAC will be implemented anywhere. Especially with QT & Nero AAC popularity.

But it's strange MAGIX hasn't implemented the latest one in MP3 Maker v16. 

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #270
Here's a line graph of the bitrates in my test. It shows interesting variation from a track to another. Naturally, it doesn't tell how the bitrate varies inside the files in individual frames, but seems like the encoders don't always agree with each other when they estimate the needed bitrate. For comparison, I included LAME -V5.



The tested tracks are:
Code: [Select]
Various

1 AC/DC - Highway to Hell
2 Adiemus - Boogie Woogie Llanoogie
3 Barry White - Sho' You Right
4 Björk - Possibly Maybe (Lucy Mix)
5 Davis Bowie - Starman
6 Dido - Life For Rent
7 Duran Duran - Astronaut
8 ELO - Livin' Thing
9 Erich Kunzel & The Cincinnati Pops - Theme from The Pink Panther
10 Evanescence - Going Under
11 Faithless - Mass Destruction
12 Garbage - Bleed Like Me
13 Jamiroquai - World That He Wants
14 Kraftwerk - Tour de France Etape 1
15 Morrissey - Irish Blood, English Heart
16 Paco De Lucia - Rumba Improvisada
17 Santana - Oye Como Va
18 Simply Red - You Make Me Believe
19 Sting - Desert Rose
20 The Beatles - Let It Be
21 Tina Arena - Symphony of Life
22 U2 - Vertigo
23 Whitney Houston - Queen Of The Night
24 Yello - Planet Dada
25 Yo-Yo Ma - Libertango

Classical

26 Aldo Ciccolini - Satie, Sports Et Divertissements, Le Flirt
27 Alfred Brendel - Beethoven, Piano Sonata No 15, Op 28 Scherzo
28 Baroque Festival Orchestra - Vivaldi, Sinfonia in C major, 3. Presto
29 Berlin Philharmonic Orchestra - Mozart, Requem in D moll K 626, Sanctus
30 Berlin Philharmonic Orchestra - Strauss, Also Sprach Zarathustra (Thus Spoke Zarathustra)
31 Berlin Radio Symphony Orchestra - Mahler, Symphony No  8, 2 Chailly, Ewiger Wonnebrand
32 Christophe Rousset - Farinelli, Il Castrato (OST), J.A. Hasse, Generoso risuegliati
33 Concentus Musicus Wien - Bach, Matthäus Passion BWV 244, Da ging hin der Zwölfen Einer
34 Daniel Barenboim - Mozart, Piano Concerto No 3 in D major, K40-3
35 Gérard Lesne - Vivaldi, Sonate Op 2 No 3 pour violon & bc, III. Adagio
36 Giuseppe Sinopoli - Elgar, Cello Concerto, Serenade for Strings, Enigma-Andante
37 Itzhak Perlman - Paganini, 24 Caprises, No 1 In E
38 Jascha Heifetz - Sarasate, Zigeunerweisen, Op 20 No 1
39 Jessye Norman - Angels we have heard on high (Trad.)
40 Kirov Orchestra & Chorus - Khatchaturian, Gayaneh, Säbeltanz
41 Leslie Howard - Liszt, Douze Grandes Études, S 137 No 1 in C
42 London Sinfonietta - Saint-Saëns, Le carnaval des animaux, Hémiones
43 London Symphony Orchestra - Ravel, Daphnis et Chloé, 10. Tres modere
44 Marie-Claire Alain - Bach, Wo Soll Ich Fliehen Hin BWV 646
45 Michael Nyman - The Piano OST, A Bed of Ferns
46 Orchestre Symphonique De Montreal - Elgar, Enigma Variations No 1
47 Philharmonia Slavonica - Bach, BWV 1067 Rondeau
48 The Cleveland Orcestra - Ravel Valses, nobles et sentimentales, 1. Modere
49 The Philadelphia Orchestra - Tchaikovsky, The Nutcracker, Op 71a (Ballet Suite) No 2
50 Zbigniew Preisner - Trois Couleurs Bleu (OST), First flute
  • Last Edit: 15 April, 2010, 05:00:13 PM by Alex B

  • Polar
  • [*][*][*][*]
Public Listening Test [2010]
Reply #271
And I actually didn't think you had any knowledge about AAC and any such stuff when this thing started so I thought it was going to flop.
It's hard not to label such an utterance as blunt and ignorant.  If you had done your homework, you'd know about Igor's numerous and elaborate double-blind AAC listening tests which he took the trouble about to document them and share them with everyone right here, in this very forum.  The search button is right there at the top.

  • Sylph
  • [*][*][*][*]
Public Listening Test [2010]
Reply #272
I'm not talking about ABX, it's other stuff.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #273
There is a simulation of ABC-HR and fake Sample01 packages.
Please report if there will be any error.

I have done the next steps:
1. Java abchr - java from here http://www.hydrogenaudio.org/forums/index....st&p=683924
2. Put FLAC 1.2.1 and FAAD (http://www.rarewares.org/aac-decoders.php)  decoders in the same folder.

Links:
ABC-HR    h*tp://www.mediafire.com/?gmdrnmmuwyj
Sample01  h*tp://www.mediafire.com/?inkymjmtdjj


P.S.
Low anchor is Itunes LC-AAC 64 kbps VBR
  • Last Edit: 18 April, 2010, 08:54:34 PM by IgorC

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #274
I tried the simulaton. Technically it works fine. Some comments:

- In the last MP3 test the bat files were in the BIN folder. The isolated location made easier to find them.
For reference, I uploaded Sebastian's MP3@128 "ABC-HR_bin" here: http://www.hydrogenaudio.org/forums/index....st&p=701994 (In case you don't have it archived.)

- You didn't include actual instructions. Here is Sebastian's readme.txt from the last test:

Code: [Select]
Public MP3 Listening Test @ 128 kbps
------------------------------------

Welcome and thank you very much for taking part in this listening test.
Your results will be very valuable.

Here are the instructions:

1. Decompress the "ABC-HR_bin.zip" package to its own folder.

2. Download one ore more sample packages (location of the packages is
   below).

3. Place the sample package ("SampleXX.zip") in the same folder as
   ABC/HR and uncompress it. Make sure you keep the directory structure
   intact or the configuration files will not work.
   For an idea how the folders should look like (in case you
   downloaded all sample packages), check "folder-setup.png".

4. WINDOWS USERS:
     Navigate to the "bin" directory and run "DecodeXX.bat" ("XX"
     being the number of the sample package you want to prepare for testing).
     If you would like to prepare all packages, run "DecodeAll.bat".
     Wait until the command prompt screen disappears.
   *NIX (Linux, OS X, BSD, Solaris, etc.) USERS:
     Linux users are asked to use Wine with "wine wcmd /c DecodeXX.bat" from
     the "bin" directory.
     OS X users are asked to handle decoding of samples themselves (sorry).

5. You need the Sun Java Runtime Environment version 1.5 or newer to run ABC/HR.
   If you don't have Java, download it from here:
   [url=http://www.java.com]http://www.java.com[/url].

   You need to use the ABC/HR for Java version provided in the
   "ABC-HR_bin.zip" package together with this readme. Older versions are not
   compatible with the configuration file format.
     WINDOWS USERS:
         Double-click "abchr.jar".
     *NIX USERS:
         Run "java -jar abchr.jar" from the shell.

   Once ABC/HR is open, click "Open ABC/HR Config..." and load the
   file "SampleXX.ecf" (again, "XX" being the number of the test you want
   to take - make sure you ran the batch file respective to that package
   before).

6. Take the test. If you need information on how to properly do that,
   check ff123's page: [url=http://ff123.net/64test/practice.html]http://ff123.net/64test/practice.html[/url].

7. After you finish the test, save the results, (7-)ZIP, RAR or ACE them
   together and mail the file to mail@listening-tests.info.
   The test is scheduled to end on November 3rd, 2008. No results
   will be accepted after that date, unless the test is extended. Possible
   extensions will be announced at the test page (http://www.listening-tests.info).
  
   You don't need to test all samples to participate. Even one single
   result is already very helpful. Of course, the more you test, the better
   for the final results' significance.

   All results and comments I receive will be published. If you want
   to be associated with your results, please enter your (nick)name in the
   "Show name in results file" field in ABC/HR (check the checkbox next to
   it to enable the field). Otherwise, your results will be anonymous.

---------------------------------------------

These are the sample packages:

http://.../~sebastian/Sample01.zip
http://.../~sebastian/Sample02.zip
http://.../~sebastian/Sample03.zip
... etc

- OR (alternate download links) -

http://.../Sample01.zip           
http://.../Sample02.zip           
http://.../Sample03.zip           
... etc

The average package size is 4 MB.

Thank you very much!

Best regards,
Sebastian Mares

- Regarding the low anchor, perhaps 64 kbps is too low for emese (the demo sample). In the last  test the low anchor choice was a bit unfortunate. Sebastian wanted to evaluate the progress from the first popular FhG MP3 encoder and picked an old FhG MP3 encoder from "really rarewares". However, that happened to be a buggy prerelease version (Roberto's description wasn't exactly accurate) and the encodings were really too bad to be anyhow useful.

I wonder if e.g. 80 kbps would be suitable for the test samples. The difference should be audible, but not like a night and day difference. Ideally the low anchor should give some hints about what kind of artifacts the contenders might produce. I.e. it should produce similar type of artifacts in a more pronounced form.

EDIT

Remember to include the licenses. You can copy them from Sebastian's test package.
  • Last Edit: 21 April, 2010, 07:13:21 PM by Alex B