Skip to main content

Topic: 128kbps Extension Test - OPEN (Read 43382 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • bond
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #50
Quote
Old issue with oggenc rises the head again:
Quote
C:\Temp\128kbps\Bin>oggenc -q 4.25 ..\Sample01\41_30sec.wav --output=..\Sample01
\41_30sec_ogg.ogg
Opening with wav module: WAV file reader
Encoding "..\Sample01\41_30sec.wav" to
         "..\Sample01\41_30sec_ogg.ogg"
at quality 4,00
Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.

wah, the same for me (and i already wondered about the results)


Quote
I've found this test quite dis-heartening.  Out of 5 samples I have tried, I could only pick out Blade one 1 of them.  I can't even ABX another codec in any of them    [span style='font-size:7pt;line-height:100%'](why isn't there a crying emoticon?)[/span]

you can have a look at the coments on the samples in the aac listening test (pretty much the same as now) here
there you can find hints on what you should listen in each sample

first time i tried it i couldnt also hear many differences, but now i listened to 5 samples and i could hear differences in every sample (voted 5 only twice till now)
I know, that I know nothing (Socrates)

  • bond
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #51
Quote
I uploaded modified oggenc.exe that will use proper quality level, it uses recent CVS libraries.

thanks but is this the same cvs version rjamorim used?

he used "Ogg Vorbis 1.0 post-CVS". what does "post-cvs" mean?


edit:
according to the filesize case's oggenc produces exactly the same output as the original oggenc with -q 4!

->
i would recommend changing the *.bat files by hand if "," is used as a comma in your country!
  • Last Edit: 24 July, 2003, 01:09:28 PM by bond
I know, that I know nothing (Socrates)

  • Case
  • [*][*][*][*][*]
  • Developer (Donating)
128kbps Extension Test - OPEN
Reply #52
Quote
thanks but is this the same cvs version rjamorim used?



edit:
according to the filesize case's oggenc produces exactly the same output as the original oggenc with -q 4!

->
i would recommend changing the *.bat files by hand if "," is used as a comma in your country!

I suppose so, except library version string isn't modified like john33 did.

Quote
he used "Ogg Vorbis 1.0 post-CVS". what does "post-cvs" mean?

I think it should be "post 1.0 CVS".

  • bond
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #53
some one here who has access to the presentation page to update it with this (not unimportant) info?
I know, that I know nothing (Socrates)

  • askoff
  • [*][*][*][*]
128kbps Extension Test - OPEN
Reply #54
Should this test to be called Medium quality Extension test? If all samples are'nt close enough at 128kbps. Yes i've read all comments, but still this is not the right way i think. If MP3, AAC and WMA are forced to be near as possible to 128kbps and ogg and mpc can be what ever they want to bee...

  • JohnV
  • [*][*][*][*][*]
  • Developer
128kbps Extension Test - OPEN
Reply #55
Quote
Should this test to be called Medium quality Extension test? If all samples are'nt close enough at 128kbps. Yes i've read all comments, but still this is not the right way i think. If MP3, AAC and WMA are forced to be near as possible to 128kbps and ogg and mpc can be what ever they want to bee...

And what would be the right way in your opinion? Forcing vbr samples separately to 128kbps? I hope you understand that in that case you'd have 12 different settings for one vbr codec, so what would the results tell then about the codec's quality at certain setting - nothing...
Simply put: VBR is used with average bitrate basis here, that's the whole idea and the only sain thing to do. I'm sorry if you don't understand that, but that is the most fair way to test vbr.
Lame MP3 is ABR because it performs better than Lame MP3 vbr near the 128kbps. Quicktime is CBR because there's no VBR/ABR setting for it.
  • Last Edit: 24 July, 2003, 02:07:13 PM by JohnV
Juha Laaksonheimo

  • Volcano
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - OPEN
Reply #56
Quote
Quote
Old issue with oggenc rises the head again:

[...]

Anyone with regional settings specifying other character than '.' for decimal separator will get too low quality Vorbis files.

Ouch.. this is definitely a problem, although not catastrophic. -q4 is officially 128kbps nominal anyway.

Bummer!  If only someone had spotted this before the articles were submitted to those large sites...

I guess I'm lucky that at least on one of the two samples I've submitted results for so far, the Vorbis encoded version was totally transparent to me even though it was only encoded at -q 4, so I won't have to re-test that one.

So far, I am very surprised by the test... on the samples I've tested so far, the differences were very hard to detect and ABX most of the time (the two MP3 competitors - OK, Blade isn't really a "competitor"  - aside). Nice to see how the encoders have improved over time.


Quote
Still, this comparison a lot more difficult than I think some people may have thought, given that this is "only" 128 kbit/s.

I can still vaguely remember your last 128kbps test more than 1 1/2 years ago. Yeah, that was indeed a *lot* easier.
  • Last Edit: 24 July, 2003, 03:10:20 PM by Volcano

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
128kbps Extension Test - OPEN
Reply #57
I suggest that Roberto verifies with each person who submits results that their ogg ratings weren't affected by the comma bug (only a potential problem if they use the regional comma setting *and* happened to rate ogg files worse than the reference).

Meanwhile, I guess I'll append a note to my Usenet postings.

ff123

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - OPEN
Reply #58
I'm a bit afraid to see that everybody could check immediatly after a test their results. Names are now explicit. It's easy to 'correct' some results, in order why not to give to our favorite format the best place... It's stupid I know, but there is a disease here called zealotry, and we can fear some cheating...

It's time for encrypted result log file...
  • Last Edit: 24 July, 2003, 03:34:02 PM by guruboolez

  • bond
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #59
i would say that this would be the best way:

upload as fast as possible a new abc-hr_bin.zip package instead (!) of the old one

the following changes should be done in the package:

1) exchange the old oggenc.exe with a version that isnt "," or "." sensitive like the one from case (although i am not sure if his version produces really 4.25 output in every case)
2) change the sampleXX.bat files in a way that the .ogg.wav files created with this updated package have a different name than the original ones (for example append a "_2" to the ogg filename so that rjamorim knows that there is the possibility that this file (and the rating) is flawed
-> than rjamorim can verify with the person who sent this package if the comma bug was present (so he doesnt have to check it with every person)
3) it will be than also possible to give the files "anonymous" names (like in the aac test) like guruboolez suggested


but the most important part will be that this happens as fast as possbile!!!
were is rjamorim or does anbody else has access to upload an updated package?
  • Last Edit: 24 July, 2003, 03:39:59 PM by bond
I know, that I know nothing (Socrates)

128kbps Extension Test - OPEN
Reply #60
Okay, if required to, I can redo three tests I've sent. Got the bug.

The batch file would need to be unreadable and have no output.
Which isn't possible, of course.

/EDIT\
Oh well, it is... with a tool called Bat2Exe.
Now, how to suppress the output of all encoders?

Of course, you'll need to use anonymous names for all flacs and output files.
\EDIT/


You might want to write a dedicated program for this.

Encryption would be nice, but it would require modified version of ABC/HR.
And lots of work!
  • Last Edit: 24 July, 2003, 04:14:59 PM by AstralStorm
ruxvilti'a

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
128kbps Extension Test - OPEN
Reply #61
Quote
Okay, if required to, I can redo three tests I've sent. Got the bug.

The batch file would need to be unreadable and have no output.
Which isn't possible, of course. /EDIT\ No, it is... know a tool called Bat2Exe? \EDIT/
But you might want to write a dedicated program for this.

Encryption would be nice, but it would require modified version of ABC/HR.
And lots of work!

Unfortunately, I don't really have time anymore to modify ABC/HR.  Although the source is open, in practice, only the original authors seem to dare to modify such programs.

A program to modify the batch files should not be hard to gin up using something like python.  Let's see what Roberto wants to do.

ff123

128kbps Extension Test - OPEN
Reply #62
Reread my post, please.
Modifying the batches won't help - how Roberto will know which file is which?

It has to be set earlier.
Additionally, encoding/decoding order should be 'randomized' (in the batch files, of course).
  • Last Edit: 24 July, 2003, 04:25:31 PM by AstralStorm
ruxvilti'a

  • spoon
  • [*][*][*][*][*]
  • Administrator
128kbps Extension Test - OPEN
Reply #63
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

  • JohnV
  • [*][*][*][*][*]
  • Developer
128kbps Extension Test - OPEN
Reply #64
Quote
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.
  • Last Edit: 24 July, 2003, 04:57:54 PM by JohnV
Juha Laaksonheimo

128kbps Extension Test - OPEN
Reply #65
Here is the sample batch - you need to make an executable from it!
/EDIT\
Okay, final, working version!
\EDIT/
Code: [Select]
@echo off
flac -d --silent -o ..\Sample05\orig.wav ..\Sample05\orig.flac >> NUL:

rem Don't forget to randomize names and order of the parts!

rem Lossless part
copy /Y ..\Sample05\orig.wav ..\Sample05\5.wav >> NUL:
rem End

rem WMA part
flac -d --silent -o ..\Sample05\2.wav ..\Sample05\wma.flac >> NUL:
rem End

rem MP4 part
rem FAAD does support neither silent encoding nor pipes (weird, it should work with pipes...)
faad -o ..\Sample05\3.wav ..\Sample05\mp4.mp4 >> NUL:
rem End

rem MPC part
rem Workaround for MPPENC showing its version
mppenc --quality 4 --xlevel --silent ..\Sample05\orig.wav ..\Sample05\orig.mpc >> NUL:
mppdec --silent ..\Sample05\orig.mpc ..\Sample05\1.wav >> NUL:
del ..\Sample05\orig.mpc >> NUL:
rem End

rem Vorbis part
oggenc -Q -q 4.25 ..\Sample05\orig.wav --output=..\Sample05\orig.ogg >> NUL:
rem OggDec doesn't support silent decoding... here's a workaround
del ..\Sample05\4.wav >> NUL:
oggdec -o ..\Sample05\orig.ogg >> ..\Sample05\4.wav
del ..\Sample05\orig.ogg >> NUL:
rem End

rem LAME part
lame --alt-preset 128 --scale 1 --silent ..\Sample05\orig.wav ..\Sample05\orig.mp3 >> NUL:
lame --decode --silent ..\Sample05\orig.mp3 ..\Sample05\7.wav >> NUL:
del ..\Sample05\orig.mp3 >> NUL:
rem End

rem Blade part
bladeenc -q -quiet ..\Sample05\orig.wav ..\Sample05\orig.mp3 >> NUL:
lame --decode --silent ..\Sample05\orig.mp3 ..\Sample05\6.wav >> NUL:
del ..\Sample05\orig.mp3 >> NUL:
rem End

rem Cleanup
rem You may ommit this if you want, but this increases the security slightly
del ..\Sample05\orig.flac >> NUL:
del ..\Sample05\wma.flac >> NUL:
del ..\Sample05\mp4.mp4 >> NUL:
del ..\Sample05\orig.wav >> NUL:
  • Last Edit: 24 July, 2003, 06:20:43 PM by AstralStorm
ruxvilti'a

  • spoon
  • [*][*][*][*][*]
  • Administrator
128kbps Extension Test - OPEN
Reply #66
Quote
Quote
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.

Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?

  • JohnV
  • [*][*][*][*][*]
  • Developer
128kbps Extension Test - OPEN
Reply #67
Quote
Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?

Pretty much yeah, because it has been explained many times now, and frankly this is an issue which should be pretty much clear that you are getting no interpretable results about vbr codec's quality with the method you are suggesting..

So for example regarding Vorbis (just example, not real):
sample 1: -q2
sample 2: -q6
sample 3: -q3
sample 4: -q2.5
sample 5: -q3.3
sample 6: -q1.8
sample 7: -q4
sample 8: -q4.2
sample 9: -q5
sample10: -q6.5
sample11: -q2.3
sample12: -q3.5

Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...

And posting here is not any "democratic right". It's a priviledge which is possible to be taken away.
  • Last Edit: 24 July, 2003, 05:39:40 PM by JohnV
Juha Laaksonheimo

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - OPEN
Reply #68
Quote
Quote
Quote
If I am reading this, some encoders can output upto 190Kbps? based on quality settings. In a fair world each sample would be compressed by hand until the final output fitted just under the filesize for a 128Kbps CBR file.

Jesus.. How many times this has to be explained in the same thread???
http://www.hydrogenaudio.org/forums/index....ndpost&p=117956
http://www.hydrogenaudio.org/forums/index....ndpost&p=117779
http://www.hydrogenaudio.org/forums/index....ndpost&p=117895

I'd understand if this was at slashdot.org..

Next same kind of message goes straight to off-topic.

Well Jesus...I have already read those and didn't agree with the messages, so being as this is a free and open message board I used my democratic right to post what I wanted. Or is it because you a Moderator have replied to it, then that is gospel and no one else can reply?


In a fair world, you had to cut each track of each album in small 5 seconds parts, encode them manually with proper setting in order to reach x kbps, then merge all small parts into a complete track again. By doing that you will obtain VBR file, with each difficult or easy part encoded at x kbps. In other words, CBR with bit reservoir...

If some people doesn't like the kbps amplitude phenomenon, they just had to avoid VBR, and chose CBR encodings. Quality is worse, but bitrate will be constant... You can't expect choosing VBR for its quality and fighting against its own logical : more bitrate when needed. Is it so hard to understand ?
  • Last Edit: 24 July, 2003, 05:31:54 PM by guruboolez

  • bond
  • [*][*][*][*][*]
128kbps Extension Test - OPEN
Reply #69
so perhaps we will see mpc as winner (or on a top position)  just because it's smart enough to "cheat" the comparison by using as much bitrate as possible to reach great quality (and everybody just wanted to add it for "academical reasons") 

"hey, i always had the opinion that mpc was the best codec @128kbps and the test also came to the same conlcusion" (although mpc hardly ever really used 128kbps)

 

in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...
I know, that I know nothing (Socrates)

  • JohnV
  • [*][*][*][*][*]
  • Developer
128kbps Extension Test - OPEN
Reply #70
Quote
so perhaps we will see mpc as winner (or on a top position)  just because it's smart enough to "cheat" the comparison by using as much bitrate as possible to reach great quality (and everybody just wanted to add it for "academical reasons") 

"hey, i always had the opinion that mpc was the best codec @128kbps and the test also came to the same conlcusion" (although mpc hardly ever really used 128kbps)

 

in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...

Read this:
http://www.hydrogenaudio.org/forums/index....showtopic=11134

The reason why mpc and vorbis give high bitrates is because some of the samples are short and hard to encode in order to hear some difference between the codec qualities. I don't understand why vbr codecs should be punished, just because the test samples are short and have been chosen to be such that listeners may have easier job to distinguish differences..
  • Last Edit: 24 July, 2003, 05:58:51 PM by JohnV
Juha Laaksonheimo

  • voltron
  • [*][*][*][*]
128kbps Extension Test - OPEN
Reply #71
The encoder doesn't cheat.. it's encoding algorithms are such that when more bits are needed, more bits will be used. It is pointless to discriminate against one encoder by cutting out "problematic" parts of a song. This 128kbs test is for the very purpose of seeing which encoder can produce the best sound while using a relatively (depending on who) low bitrate. If it takes --quality 4 or whatever is being used to attain ~128 (ABOUT 128kbs), so be it.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - OPEN
Reply #72
Quote
in a fair world the whole song would have been encoded at an average bitrate around 128 and then the 30sec problematic part would have been cut out...

Encoding a whole album in order to reach 128 kbps, and let the same setting for a microscopic sample, is something fair. This is what Roberto asked us to do, by comparing mpc, ogg and wma at different settings on our favorite discs. Musepack --radio is close to 128 kbps, and for oggenc it's -b 4,25.

If you're familiar with your encoder, it's possible to anticipate some of its behaviour, and to chose different setting according to the disc you had to encode.
If I had a piano disc to encode with mpc at ~130 kbps, I will set the encoder at --quality 4.5 without trying --quality 4. For an harpsichord disc, I won't go beyond --quality 3. For orchestral, --quality 4 have the most chances to be close to the targeted bitrate...

  • spoon
  • [*][*][*][*][*]
  • Administrator
128kbps Extension Test - OPEN
Reply #73
Still have my priviledge

Quote
Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...


The reason VBR exists is that a codec can be advanced enough to lower its bitrate and up its bit rate, but for this to be a fair test - especially with different sample types - for all we know on the harpsicord: ogg will go to an average of 200Kbps, whilst WMA goes to 128Kbps. Now you could say, tough luck WMA for not matching Ogg when it goes to 200Kbps, but you could also say that WMA is better programmed because it stays within its quality range and does not vary wildly. *** these codecs and numbers are totally made up ***

I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

128kbps Extension Test - OPEN
Reply #74
My batch file is finished!
Now I'm going to check if Bat2Exe works correctly with it.

/EDIT\
It does! Of course there are still ways to detect which file is which, of course...
But these aren't that easy.
\EDIT/
  • Last Edit: 24 July, 2003, 06:31:21 PM by AstralStorm
ruxvilti'a