Skip to main content

Topic: ITU-R BS.1387 Analysis Tool finally available, under GPL! (Read 31268 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Alexander Lerch from HHI/ZPlane released perceptual quality measurement tool under GPL!  Get it at:

http://sourceforge.net/projects/eaqual/


Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:

Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9  @128 Kbits/s: -0.7479

Remeber - this tool can't replace listening tests, but it could help in codec quality testing and development!

  • amp
  • [*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #1
Wow, great news!
OGG RC3 and this, nice way to start the year !!!

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #2
Ivan,

Neat.  What does psytel 1.2 score at 128 kbit/s?  You could compare it against what your listening test said.  For that matter, I could compare notes using dogies.wav and wayitis.wav to see if it matches up reasonably well.

ff123

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #3
I haven't tested it yet with PsyTEL 1.2, but others that wish to use this tool I must give some more information

- Offset value is essentially important, wrong offset leads to very wrong results

Some of the offsets:

Old FAAD + PsyTEL AACEnc (all versions): 2048 (1024 samples)

LAME --decode + LAME MP3: 64 (32 samples)

Lame --decode + FhG MP3Enc: 1172 (576 samples)

Old FAAD + FhG/Dolby AAC (hehe, if you have them  4392 (1024 + 1172 samples)

Etc...

I couldn't get eaqual to work with new FAAD - probably I wasn't too careful, offset might be different...

Anyway - a good way to find offset if you don't know encoder charateristics is:

- Encode castanets sample in very high quality VBR
- Open it in CoolEdit
- Setup the time slider to "samples"
- Identify first peak, and set the pointer right in the middle of the peak in both samples
- Subtract the sample values

If you were careful enough, the delay will be +/- 20-30 samples, and find first "expected" number (512, 576, 1024, 1172 and multiples) - multiply it by 2 and that is the sample offset.

Now, try it with the EAQUAL tool, and the resulting ODG should be in the range of -1.0 - 0.0 for high quality VBR presets.

  • Sachankara
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #4
Could anyone compile it for me...? I can´t seem to get it to work correctly with VC++ 6.0...

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #5
I get this with death2.wav:

c:ProgsEncodersEaqual>eaqual -fref  d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100

EAQUAL - Evaluation of Audio Quality
Version:        0.1alpha
Author:        Alexander Lerch, zplane.development
Copyright:      Heinrich- Hertz-Institut (http://www.hhi.de)
                zplane.development (http://www.zplane.de)
_______________________________________________________
Reference File:        d:testgeluidendeath2.wav
Test File:              d:death2.wav
Sample Rate:            44100
Number of Channels:    2

Press Escape to cancel...

Frame:          806
Time elapsed:  41.39

Resulting ODG:  -3.9122
Resulting DIX:  -4.1096

BandwidthRef    21521.9414
BandwidthTest  16249.1914
NMR            41.9918
WinModDiff1    163.0067
ADB            3.1234
EHS            1.4913
AvgModDiff1    200.9328
AvgModDiff2    573.1494
NoiseLoud      53.7488
MFPD            1.0000
RDF            0.9988

Is there something wrong in my command line?

Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #6
Quote
Originally posted by Sachankara
Could anyone compile it for me...? I can´t seem to get it to work correctly with VC++ 6.0...

I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip

  • Sachankara
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #7
Quote
Originally posted by Speek

I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip
Thanks a lot...

Hmm... As the read-me file says, I thought the possible ODG values were from 0 to -4, so how do you explain a result like this? :confused:

Frame:          8339
Time elapsed:  340.78

Resulting ODG:  0.0655
Resulting DIX:  3.2649

BandwidthRef    21536.1094
BandwidthTest  20665.5117
NMR            -20.0063
WinModDiff1    4.7231
ADB            -0.3251
EHS            0.1606
AvgModDiff1    4.7216
AvgModDiff2    6.3346
NoiseLoud      0.0898
MFPD            1.0000
RDF            0.0000

(Encoded "Ace of base - Wave wet sand" with Lame 3.90 stable using "--abr 256 -q 0 -m j", and I also decoded it with Lame...)

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #8
Quote
Originally posted by Speek
I get this with death2.wav:

c:ProgsEncodersEaqual>eaqual -fref  d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100

EAQUAL - Evaluation of Audio Quality
Version:        0.1alpha
Author:         Alexander Lerch, zplane.development
Copyright:      Heinrich- Hertz-Institut (http://www.hhi.de)
                zplane.development (http://www.zplane.de)
_______________________________________________________
Reference File:         d:testgeluidendeath2.wav
Test File:              d:death2.wav
Sample Rate:            44100
Number of Channels:     2

Press Escape to cancel...

Frame:          806
Time elapsed:   41.39

Resulting ODG:  -3.9122
Resulting DIX:  -4.1096

BandwidthRef    21521.9414
BandwidthTest   16249.1914
NMR             41.9918
WinModDiff1     163.0067
ADB             3.1234
EHS             1.4913
AvgModDiff1     200.9328
AvgModDiff2     573.1494
NoiseLoud       53.7488
MFPD            1.0000
RDF             0.9988

Is there something wrong in my command line?

Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.



Yes - I can't seem to find proper offset for latest FAAD! Please use older FAAD (I use build from September 2001) or ISO AACDec - their offests are 2048.

I will see with Menno to find proper value for latest faad...

  • Skeeve242
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #9
Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.

Is there no easier way to find out the offset?


bye, Skeeve

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #10
I think that MPC has no delay, because it is using subband filtering without one frame look-ahead for block switching, but I will have to recheck..

I will do my best to find offsets for most popular codecs, and publish those values..

  • dosdan
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #11
Quote
Originally posted by Ivan Dimkovic


Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:

Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9  @128 Kbits/s: -0.7479


Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:


c:lametest>lame3902.exe --decode castanets_test.mp3 castanets_test.wav
input:  castanets_test.mp3  (44.1 kHz, 2 channels, MPEG-1 Layer III)
output: castanets_test.wav  (16 bit, Microsoft WAVE)
skipping initial 1105 samples (encoder+decoder delay)
[/b]

I think this offset stripping has been there for a while. I got the following ODGs with LAME 3.90.2 @128kbits/s on CASTANETS.WAV:


Lame3902 --abr 128  -h test.wav (136.4 kbps; qval=2) = -.5301
Lame3902 --abr 128 test.wav (135.6 kbps; qval=5) = -.5492
Lame3902 -q0 test.wav (qval=0) = -.7132
Lame3902 -q1 test.wav (qval=1) = -.7429
Lame3902 -nspsytune test.wav  (qval=5) = -.7732
Lame3902 -h -nspsytune test.wav (qval=2) = -.7712
Lame3902 -h test.wav (qval=2) = -.7302
Lame3902 test.wav (qval=5) = -.7649
Lame3902 -f test.wav (qval=7) = -.7822


Some possible conclusions:
1. -q1 is inferior to -q2 (I don't think its use is recommended anymore).
2. If -nspsytune improves the perceived quality them this program penalises it.
3. ABR measures much better than CBR

  • Sachankara
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #12
Quote
Originally posted by dosdan


Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:
Actually, the offset is still there... It doesn´t remove all of it...

Look:

Offset 64:

Frame: 8339
Time elapsed: 340.78

Resulting ODG: 0.0655
Resulting DIX: 3.2649

BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000
RDF 0.0000

Offset 0:

Frame:          8339
Time elapsed:  311.29

Resulting ODG:  0.0856
Resulting DIX:  3.4091

BandwidthRef    21536.2383
BandwidthTest  20665.4160
NMR            -21.2534
WinModDiff1    2.6592
ADB            -0.4966
EHS            0.1676
AvgModDiff1    2.5517
AvgModDiff2    3.5112
NoiseLoud      0.0434
MFPD            0.9642
RDF            0.0000

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #13
Quote
Originally posted by Skeeve242
Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.

I also got a positive ODG with Ogg Vorbis RC3 -q 6 on ftb_samp.wav (ODG = 0.0228). I'm pretty sure that MPC and Ogg both have zero offset. I did the test with CoolEdit that Ivan described. These occasional positive ODG make me doubt the results of Eaqual. Would be nice if somebody had an explanation.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #14
I'll talk with Alexander about that issue..

I've noticed that with PsyTEL VBR modes, too

Anyway, Menno fixed FAAD so now it decodes with zero offset, too! I am not sure when will he upload it to CVS, but expect that to happen soon..

-- Ivan

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #15
Here is what alexander told me regarding positive ODG:

Quote
Hi,

It's true that a positive ODG should not occur
(theoretically). 
However, the ITU recommendation does allow positive ODGs,
because this could also happen in listening tests, where the
file under test is rated better than the reference file.
Furthermore you may not forget, that there is a neural network
builtin which was trained on real listening test ratings. The
confidence intervall in those tests may ly between 0 and 0.5,
and thus EAQUAL may be wrong within those intervalls.

  • Ardax
  • [*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #16
Just for giggles, and because I love Ogg Vorbis , here's some more ODG values for castanets.wav:

OggEnc RC3 @ -q 3.00: -0.7350
OggEnc RC3 @ -q 3.50: -0.6381
OggEnc RC3 @ -q 4.00: -0.5918
OggEnc RC3 @ -q 4.50: -0.5131
OggEnc RC3 @ -q 5.00: -0.3051

Ogg should have a zero offset, IIRC.  Decoding was done with WinAmp 2.78 w/ Ogg Vorbis Input Plugin 1.17 beta, and Nullsoft Disk Writer Plugin 2.0b.

ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #17
Now we can see that ogg is the best

  • lassal66
  • [*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #18
What kind of ODG values do you get using common settings/presets (standard to high quality) with LAME, MPC, AAC and OGG?

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #19

ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #20
hehe only ogg sounds better then the original 
the only positive value in speeks tests

  • Sachankara
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #21
Quote
Originally posted by Benjamin Lebsanft
hehe only ogg sounds better then the original  
the only positive value in speeks tests
Try Lame with "--abr 256 -q 0 -m j" and you´ll see some more posivite values...
Quote
Originally posted by Ivan Dimkovic
Here is what alexander told me regarding positive ODG:
[some text]
So in essence, harmonic distortion should make the ODG values positive, or am I way wrong here? (I´m kinda´ new to this you know... )

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #22
Well, like I said - this tool could be used for codec debugging, and not for final marking of which codec sound better

For example - this codec ranks FAAC castanets @ 128 with ODG of  -0.8 , and I am sure that even for ISO listener FAAC is still below "excellent" range at 128 kbps, etc..

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #23
Quote
I posted some results here: 
http://home.wanadoo.nl/~w.speek/eaqual.htm


I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original).  However, it may not have a big impact on the results.

ff123

  • Speek
  • [*][*][*][*]
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Reply #24
Quote
Originally posted by ff123


I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original).  However, it may not have a big impact on the results.

ff123


Here are the results with your 128 kbps command-line, but without --scale 0.93:
ftb_samp: -1.2156
sade__sweetest_taboo: -0.9003
iron: -0.8857
fools: -1.0177

ftb_samp and sade are slightly better with --scale 0.93. Iron and Fools are about the same. I didn't try the --alt-presets with --scale 1. Maybe later.