Skip to main content

Topic: ITU-R BS.1387 Analysis Tool finally available, under GPL! (Read 31286 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #25
What is the offset for FhG. Fastenc (CoolEdit Pro filter)? With the test Ivan described I think it's about 96 (that's 192 for EAQUAL) when decode with LAME, but when I put an higher offset number in EAQUAL the ODG gets better. Wouldn't it be logical that the offset that gives best ODG is the right offset :confused:

  • Ardax
  • [*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #26
Here's some more ODG values for Speek's samples, except one, because it's smarter than me. 

(Speek, I can't decode sade_sweetest_taboo.pac.  LPAC says that it's not a valid LPAC file.  Did I pick the wrong decoder?)

These tests were done with lame 3.91 using --alt-preset 128 and oggenc rc3 -q 4.00.  This was to test files that were nominally targeting 128 kbps, since that's the file size I want, and I don't want to futz with command lines until they're exactly 128 kbps (although that's useful information too).  I just want to cram in the command line that gives me the best quality near 128 kbps.  Quick and easy.

lame 3.91 --alt-preset 128
fools.wav : 126.4 kbps : ODG -0.9395
ftb_samp.wav : 121.1 kbps : ODG -1.3005
iron.wav : 126.9 kbps : ODG -0.8146

oggenc rc3 -q 4.00
fools.wav : 121.9 kbps : ODG -0.8032
ftb_samp.wav : 131.0 kbps : ODG -0.8292
iron.wav : 123.6 kbps : ODG -0.8476

Interesting results, to look at.  Oggenc does a pretty good job at looking "consistent."  I think I'm going to go play some Civ III now. 

  • Xiphiod
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #27
Here's some real world numbers with ogg:

Drowning Pool - Bodies
GT2          192.0   -.36184*
5   160.2   -.3340
5.5   180.8   -.0975
6   201.2   .0290

Primus - Mrs. Blaileen
5   145.0   -.3014
5.5   161.8   -.0781
6   177.7   .0332

Propellerheads - Bang On!
5   152.0   -.3271
5.5   170.0   -.0932
6   187.0   .0182

*Garf Tuned RC2 @ 160

  • CiTay
  • [*][*][*][*][*]
  • Administrator
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #28
Hmm, why do you use the "sade__sweetest_taboo" sample anyways? It was a sample i sent to Roel about a year ago, because of some lowpass issues. This sample isn't really a "hard" sample for encoders... or was it that what you wanted, an everyday-sample?

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #29
I tested my dogies.wav encodes using the analysis tool, and here are the results:

ODG

xing:  -1.35
wma: -1.46
lame: -1.24
ogg:  -1.48
aac:  -1.15
mpc:  -0.73

The listeners rated the files as follows:

xing:  -2.61
wma:  -1.77
lame:  -1.75
ogg:  -1.63
aac:  -0.85
mpc:  -0.74

In other words, xing was much worse than wma/lame/ogg, which in turn was much worse than aac/mpc.  The analysis tool does not seem to rate the files the same way.

BTW, I went through a lot of trouble when I first made the WAV sample to time align all the files, and I just now double-checked to make sure that the samples were all aligned.  The ogg file was shifted by 1 sample (23 microseconds), so I shifted it into alignment, but the numeric results didn't change.

I wouldn't necessarily declare results from this program infallible yet

ff123

  • dosdan
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #30
To those using the w32 LAME 3.90.2 binary. I've noticed that
the decoded.wav generated by "lame --decode test.mp3 decoded.mp3" while not having a offset at the front may have added trailing silence so that the size of decoded.wav is slightly bigger than test.wav

This will cause incorrect ODGs particularly on smaller test files say 1MB.  I've been improving accuracy by removing the trailing silence in CoolEdit.  The extra size varies from 100 bytes to 3KB so far. So check those decoded wav file sizes.

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #31
Quote
Speek, I can't decode sade_sweetest_taboo.pac. LPAC says that it's not a valid LPAC file. Did I pick the wrong decoder?

I see, the file got truncated with uploading. I'm uploading it again now.

Quote
Hmm, why do you use the "sade__sweetest_taboo" sample anyways? It was a sample i sent to Roel about a year ago, because of some lowpass issues. This sample isn't really a "hard" sample for encoders... or was it that what you wanted, an everyday-sample?

So that's were it came from! I was just picking some of the longer testfiles I have. Most are from the LAME test samples webpage, but I saw that this one wasn't. In the readme of EAQUAL it says that a test sample should be at least 10 to 20 sec., but the longer the better. Many test samples are to short.

Quote
I wouldn't necessarily declare results from this program infallible yet


No, I wouldn't to. I'm just getting some data. I don't know were it leads to. One of the things I'm thinking about is: Ogg seems to get the best ODG scores. Is it really the best encoder, or does it something better than other encoders that EAQUAL pays much attention to? Maybe Ogg does something else worse than some other encoders, but EAQUAL doesn't pay much attention to that factor? Just open questions, I don't have a clue

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #32
It is using 1024 point FFT window for the analysis - so it could miss some pre-echo events now and then.

It is ranking psytel velvet.aac (encoded with 2.01) better than Dolby Professional AAC encoding, and at least to my ears Dolby sounds better (not too much, but I can hear the difference)

ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #33
Hi all.

Hope you enjoy EAQUAL.
Sorry about the confusion about the offset, it was a bug and now it is fixed (and available at sourceforge). Now the offset is really counted in samples. If you use the old version of EAQUAL, you have to multiply the number of samples with the number of channels.

Alexander
zplane.development
http://www.zplane.de

ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #34
Hi!

You have to use any -scale switches very carefully. The algorithm is stable for small level differences, but it will suck for big level differences.
Scaling can be very useful because many decoders tend to clip full scale audio output - in this case the test signal would be distorted. Here you should use scaling to avoid wrong ratings with clipped audio signals.

Alexander
zplane.development
http://www.zplane.de

  • Jan S.
  • [*][*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #35
Quote
of the things I'm thinking about is: Ogg seems to get the best ODG scores.


Hmmm... that's not what I see. From the results on your site it looks like mpc -xtreme is the best.


Jan.

ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #36
My last message now

If you must use EAQUAL for codec ratings (which is not really its intention), please keep in mind that you have to average over _many_ audio files to get reasonable results - and with good reasons you may doubt those ratings too. You have to average results over least 20, 50 or 100 signals I would guess.

Alexander
zplane.development
http://www.zplane.de

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #37
Quote
Originally posted by Jansemanden
 

Hmmm... that's not what I see. From the results on your site it looks like mpc -xtreme is the best.


Jan.


Look again  Most times Ogg Vorbis -q 6 gets a better score than MPC -xtreme. The times MPC gets a better score it has a much higher bitrate.

BTW. a positive ODG is very good according to Alexander Lerch (he is the author of EAQUAL).

BTW2. I'm not saying Ogg is better than MPC, I'm just saying it gets a better score from EAQUAL on the samples I tried.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #38
Hehe... competition is now having to tune psymodels of their encoders to match EAQUAL values  hehehe (just kidding)

Did anybody run EAQUAL on WMA? I know it sucks, but just for a quick check

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #39
Quote
Originally posted by Ivan Dimkovic
Hehe... competition is now having to tune psymodels of their encoders to match EAQUAL values  hehehe (just kidding)

Did anybody run EAQUAL on WMA? I know it sucks, but just for a quick check


What bitrate would you like?

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #40
96 and 128, but WMA is a VBR codec - it sometimes gives too large bitrate, greater than nominal...

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #41
I've justed added WMA8 results for 128 kb/s. It doesn't suck to hard according to EAQUAL, but the bitrates are a bit higher than the others.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #42
Quote
Originally posted by Speek
I've justed added WMA8 results for 128 kb/s. It doesn't suck to hard according to EAQUAL, but the bitrates are a bit higher than the others.


Well, try psytel aacenc with -noshort switch (disables short blocks) - you'll notice that it also performs "good" but the results are unacceptable from the pre-echo point of view

  • Skeeve242
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #43
If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?


bye, Skeeve

  • dosdan
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #44
Quote
Originally posted by Skeeve242
If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?


The offset in question is not caused by leading or trailing digital silence in the original (ripped) wave file.  The offset is caused when it converted to an MP3 and then reconverted back to a wav.  MP3 can have overlap of info between frames e.g. when you decode the first frame of a MP3 you can't be sure you have all the data for the very start of the music info. I believe there is feame overlap both in encoding and decoding but if you know what this delay is you can remove it.  If you use LAME to decode a LAME encoded file these constant values are known to the program and it can compensate when decoding i.e.

skipping initial 1105 samples (encoder+decoder delay)[/b]

Not sure about the trailing digital silence that I'm seeing from the decoded file. This was not on the original and has been added after decoding so EAC could never strip it. The amount of added digital silence varies so it is not a constant number of samples to be removed.

  • dosdan
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #45
Quote
Originally posted by Skeeve242
If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?


The offset in question is not caused by leading or trailing digital silence in the original (ripped) wave file.  The offset is caused when it converted to an MP3 and then reconverted back to a wav.  MP3 can have overlap of info between frames i.e when you decode the first frame of a MP3 you can't be sure you have all the data for the start of the data stream. I believe there is overlap both in encoding and decoding but if you know what this delay is you can remove it.  If you use LAME to decode a LAME encoded file these constant values are known to the program and it can compensate when decoding i.e.

skipping initial 1105 samples (encoder+decoder delay)[/b]

Not sure about the trailing digital silence that I'm seeing from the decoded file. This was not on the original and has been added after decoding so EAC could never strip it. The amount of added digital silence varies so it is not a constant number of samples to be removed.

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #46
I've added some more results to my EAQUAL page:
http://home.wanadoo.nl/~w.speek/eaqual.htm

  • cd-rw.org
  • [*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #47
Just out of curiosity, could someone also give --r3mix a shot.

ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #48
Quote
Originally posted by ff123
I tested my dogies.wav encodes using the analysis tool, and here are the results:

In other words, xing was much worse than wma/lame/ogg, which in turn was much worse than aac/mpc.  The analysis tool does not seem to rate the files the same way.

BTW, I went through a lot of trouble when I first made the WAV sample to time align all the files, and I just now double-checked to make sure that the samples were all aligned.  The ogg file was shifted by 1 sample (23 microseconds), so I shifted it into alignment, but the numeric results didn't change.

I wouldn't necessarily declare results from this program infallible yet

ff123


You have to keep in mind, that it is difficult already to compare results from different listening tests, even if they were realized ITU-R  BS.1116-compliant.
Perhaps you should think of EAQUAL as _one_ listener whose results are not depending on his mood and his concentration, who never gets tired and whose results are repeatable, though only his opinion.
BTW, what are the confidence intervals for your listeners?

The difference of one sample has indeed very little influence on the result. If you want to interpret this in the frequency domain, the big difference will only be there for frequencies near Nyquist. These frequencies will not influence the result much.

And it's true, no model can be infallible

Alexander
zplane.development
http://www.zplane.de

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #49
The results of the dogies listening test can be found at:

http://ff123.net/dogies/dogies_plots.html

With an experiment-wise confidence of 95%, MPC and AAC were deemed better than the others, and LAME/WMA8/OGG were deemed better than XING.  N = 15.

The Fisher's protected LSD is known to not be "protected" if the number of samples exceeds 3, but I verified via resampling analysis that the results are indeed as given above with 95% confidence.

In other words, the listeners ranked the various codecs on this sample with clear and statistically significant preference.

ff123