Skip to main content

Topic: 128kbps Extension Test - OPEN (Read 43398 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • rjamorim
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Hello.

I'd like to announce the start of the 128kbps extension test, comparing the AAC winner (QuickTime) to Musepack, Vorbis, WMA Pro and Lame MP3.

Instructions are available at the announcement page:
http://audio.ciara.us/test/128extension/pr...esentation.html

If questions or issues arise, please post at this thread.

Thank-you.

Regards;

Roberto Amorim
  • Last Edit: 29 July, 2003, 01:02:34 PM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • Canar
  • [*][*][*][*][*]
  • Global Moderator
  • Your mom's favourite moderator
128kbps Extension Test - OPEN
Reply #1
Any .torrents going to be made available this time?

And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)

I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.
  • Last Edit: 23 July, 2003, 09:55:45 PM by Canar
1. Attack the argument, not the arguer.
2. Assume good faith.

  • rjamorim
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #2
Quote
Any .torrents going to be made available this time?

Probably, yes.

But Dibrom has been too busy lately, and he plans to create the torrents after HA moves to another server.

Quote
And should this test be advertised to the broader community? (ie. Slashdot, Kuro5hin, etc)


Yes, it would be very welcome.

Unfortunately, I'm not a member of Slashdot or Kuro5hin, so I don't know about article submissions. I would be very grateful if some insider can post about this test there, pointing to the presentation page.

Quote
I think it should be mentioned that you're comparing to Blade MP3 as well, even though it's meant to show just how bad a psychoacoustic codec can be. That way it gives people a sense of why the 128kbps MP3s they downloaded really suck.


It will only be mentioned at the results page

That way, people won't be confused, since it's there for perspective purposes, not really to be compared.

Regards;

Roberto.
  • Last Edit: 23 July, 2003, 10:10:21 PM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • dev0
  • [*][*][*][*][*]
  • Developer
128kbps Extension Test - OPEN
Reply #3
Just submitted it to slashdot.
"To understand me, you'll have to swallow a world." Or maybe your words.

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
128kbps Extension Test - OPEN
Reply #4
Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio

ff123

  • Artemis3
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #5
Made a little comment at Ars Technica.
She is waiting in the air

  • Cobra
  • [*][*][*][*]
  • Banned
128kbps Extension Test - OPEN
Reply #6
How it`s done? All files are VBR but calculated averange 128kbps? So you encoed each file in each format in VBR mode unless acheving e.g. 127-129 kbps averange? IMO it`s best choice to test >>>128kbps VBR<<< files.

EDIT:
"Ogg Vorbis 1.0 post-CVS -q 4.25" - it`s not `128kbps every time, you can`t say that it is TRUE 128kbps test!
  • Last Edit: 24 July, 2003, 04:04:41 AM by Cobra

128kbps Extension Test - OPEN
Reply #7

  • Cobra
  • [*][*][*][*]
  • Banned
128kbps Extension Test - OPEN
Reply #8
Quote
ogg q 4.25

122.70901287553649, 131.1936462778568, 123.71958369470946, 126.4998828125, 124.25014547160957, 135.24413014700758, 125.75588978120831, 130.33543256174835, 128.77903642773208, 121.00131765659843
127.599113763
- these samples (exept one) are not in 128kbps...

128kbps Extension Test - OPEN
Reply #9
Quote
Quote
ogg q 4.25

122.70901287553649, 131.1936462778568, 123.71958369470946, 126.4998828125, 124.25014547160957, 135.24413014700758, 125.75588978120831, 130.33543256174835, 128.77903642773208, 121.00131765659843
127.599113763
- these samples (exept one) are not in 128kbps...

nobody said that, but their average bitrates are closesed to 128 from the tested ogg settings (~127).

i think this method was chosen, because vbr encoders get used this way (through quality settings). usually nobody would encode a files several times with vbr to get the bitrate closest to 128kb 

regards; ilikedirt

  • Cobra
  • [*][*][*][*]
  • Banned
128kbps Extension Test - OPEN
Reply #10
Of course nobody will encode in such way! But we test _128kbps_  files.  After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?

I just want to say that normal users behaviour is completely not important in 128kbps test.

128kbps Extension Test - OPEN
Reply #11
of course i see your point here. maybe rjamorim can tell you why he decided to test this way...

regards; ilikedirt

  • Cobra
  • [*][*][*][*]
  • Banned
128kbps Extension Test - OPEN
Reply #12
I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?

  • Gabriel
  • [*][*][*][*][*]
  • Developer
128kbps Extension Test - OPEN
Reply #13
Quote
Of course nobody will encode in such way! But we test _128kbps_ files. After test we can say that in for 128kbps for samples X codec Y was better. What if WMA will have 130 kbps and Vorbis 110 kbps?


What is important is that the overall bitrate is about the same for every codec. If for a specific track codec A is using an average bitrate of 130 and codec B an average bitrate of 110, it is just the decision made by the codec. It is programmed this way, an no one manually forced codec A to use an higher bitrate.

Manually forcing each codec to reach exactly 128kbps for each sample would be unfair, as the codecs are not designed to exhibit such a behaviour. What is important is the overall bitrate.
  • Last Edit: 24 July, 2003, 04:53:26 AM by Gabriel

  • Mac
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #14
As Gabriel points out - this way you are also testing a codecs ability to judge how hard a sample is.  If Vorbis only gives the sample 110kbs and ends up sounding bad because of it - that is the fault of Vorbis's VBR handling, and so it should be punished

As far as I can tell, you can see which files are which because you load them into ABC-HR - which does all the blinding for you  Otherwise stick a biro in your eyes and you will be blind!  (joke)
< w o g o n e . c o m / l o l >

  • Volcano
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - OPEN
Reply #15
Quote
I downloaded 12 files with test samples. Two questions: 1. why it`s not a blind test? 2. Why only original, mp4 and wma files in packages?

Have you looked at the contents of the zip archives, or even read the readme?

The abc-hr_bin.zip package includes various CLI en-/decoders - and each sample package comes with a batch file which uses these CLI en-/decoders to convert the supplied FLAC file to a lossy format (and decode back to WAV again for ABC/HR to accept it). For those formats where this isn't possible because of the lack of CLI encoders (WMA, MP4), readily encoded files are supplied. This method saves loads of bandwidth.

  • Daybreak
  • [*][*][*]
128kbps Extension Test - OPEN
Reply #16
Okay I'm new to this thing....


Does using the ABX tool and comparing the original and sample x repeatedly before giving an actual rating skew the results?


Or are we supposed to simply listen from the rating table itself?

  • bond
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #17
just one thing to say:

thanks a lot for this great test, rjamorim!!!
I know, that I know nothing (Socrates)

128kbps Extension Test - OPEN
Reply #18
Two tests done... death2 and waiting.

/EDIT\
No, comparing two samples before deciding on the result doesn't skew the results.

ABX testing proves your results, especially if the difference is subtle.
\EDIT/
  • Last Edit: 24 July, 2003, 07:03:59 AM by AstralStorm
ruxvilti'a

  • Daybreak
  • [*][*][*]
128kbps Extension Test - OPEN
Reply #19
Another question on this testing thing ...

Rating 1-5 is against the original file right? Not against each other right? So assuming two codecs sound the same to you in comparision with the original file, they would rate about the same score? ( that's what I get from reading the Pratice With ABC/HR page )

Or is scoring done differently?

Also, are comments absolutely necessary?

128kbps Extension Test - OPEN
Reply #20
The tests are rating against original.

Comments aren't necessary, but useful for others.
  • Last Edit: 24 July, 2003, 07:48:09 AM by AstralStorm
ruxvilti'a

  • rjamorim
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #21
Quote
Just submitted it to slashdot.

Quote
Posted to newsgroups:

rec.audio.opinion
rec.audio.high-end
alt.music.mp3, rec.audio.misc, uk.rec.audio


Quote
Made a little comment at Ars Technica.


Thanks a lot!

And thanks to everybody that is taking care or answering the questions that arise.

Regards;

Roberto.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

128kbps Extension Test - OPEN
Reply #22
Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.
  • Last Edit: 24 July, 2003, 08:09:15 AM by AstralStorm
ruxvilti'a

  • rjamorim
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #23
Quote
Oh BTW, will the personal results of the tests be published on the net or just the summary?
I'm especially interested in comments.

This time they will. I changed the readme for that purpose.

Only when the test finishes though.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

128kbps Extension Test - OPEN
Reply #24
Submitted to Warp2Search.