Alexander Lerch from HHI/ZPlane released perceptual quality measurement tool under GPL! Get it at:
http://sourceforge.net/projects/eaqual/ (http://sourceforge.net/projects/eaqual/)
Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:
Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9 @128 Kbits/s: -0.7479
Remeber - this tool can't replace listening tests, but it could help in codec quality testing and development!
Wow, great news!
OGG RC3 and this, nice way to start the year !!!
Ivan,
Neat. What does psytel 1.2 score at 128 kbit/s? You could compare it against what your listening test said. For that matter, I could compare notes using dogies.wav and wayitis.wav to see if it matches up reasonably well.
ff123
I haven't tested it yet with PsyTEL 1.2, but others that wish to use this tool I must give some more information
- Offset value is essentially important, wrong offset leads to very wrong results
Some of the offsets:
Old FAAD + PsyTEL AACEnc (all versions): 2048 (1024 samples)
LAME --decode + LAME MP3: 64 (32 samples)
Lame --decode + FhG MP3Enc: 1172 (576 samples)
Old FAAD + FhG/Dolby AAC (hehe, if you have them 4392 (1024 + 1172 samples)
Etc...
I couldn't get eaqual to work with new FAAD - probably I wasn't too careful, offset might be different...
Anyway - a good way to find offset if you don't know encoder charateristics is:
- Encode castanets sample in very high quality VBR
- Open it in CoolEdit
- Setup the time slider to "samples"
- Identify first peak, and set the pointer right in the middle of the peak in both samples
- Subtract the sample values
If you were careful enough, the delay will be +/- 20-30 samples, and find first "expected" number (512, 576, 1024, 1172 and multiples) - multiply it by 2 and that is the sample offset.
Now, try it with the EAQUAL tool, and the resulting ODG should be in the range of -1.0 - 0.0 for high quality VBR presets.
Could anyone compile it for me...? I can´t seem to get it to work correctly with VC++ 6.0...
I get this with death2.wav:
c:ProgsEncodersEaqual>eaqual -fref d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100
EAQUAL - Evaluation of Audio Quality
Version: 0.1alpha
Author: Alexander Lerch, zplane.development
Copyright: Heinrich- Hertz-Institut (http://www.hhi.de (http://www.hhi.de))
zplane.development (http://www.zplane.de (http://www.zplane.de))
_______________________________________________________
Reference File: d:testgeluidendeath2.wav
Test File: d:death2.wav
Sample Rate: 44100
Number of Channels: 2
Press Escape to cancel...
Frame: 806
Time elapsed: 41.39
Resulting ODG: -3.9122
Resulting DIX: -4.1096
BandwidthRef 21521.9414
BandwidthTest 16249.1914
NMR 41.9918
WinModDiff1 163.0067
ADB 3.1234
EHS 1.4913
AvgModDiff1 200.9328
AvgModDiff2 573.1494
NoiseLoud 53.7488
MFPD 1.0000
RDF 0.9988
Is there something wrong in my command line?
Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.
Originally posted by Sachankara
Could anyone compile it for me...? I can´t seem to get it to work correctly with VC++ 6.0...
I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip (http://home.wanadoo.nl/~w.speek/download/Eaqual.zip)
Originally posted by Speek
I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip (http://home.wanadoo.nl/~w.speek/download/Eaqual.zip)
Thanks a lot...
Hmm... As the read-me file says, I thought the possible ODG values were from 0 to -4, so how do you explain a result like this? :confused:
Frame: 8339
Time elapsed: 340.78
Resulting ODG: 0.0655
Resulting DIX: 3.2649
BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000
RDF 0.0000
(Encoded "Ace of base - Wave wet sand" with Lame 3.90 stable using "--abr 256 -q 0 -m j", and I also decoded it with Lame...)
Originally posted by Speek
I get this with death2.wav:
c:ProgsEncodersEaqual>eaqual -fref d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100
EAQUAL - Evaluation of Audio Quality
Version: 0.1alpha
Author: Alexander Lerch, zplane.development
Copyright: Heinrich- Hertz-Institut (http://www.hhi.de (http://www.hhi.de))
zplane.development (http://www.zplane.de (http://www.zplane.de))
_______________________________________________________
Reference File: d:testgeluidendeath2.wav
Test File: d:death2.wav
Sample Rate: 44100
Number of Channels: 2
Press Escape to cancel...
Frame: 806
Time elapsed: 41.39
Resulting ODG: -3.9122
Resulting DIX: -4.1096
BandwidthRef 21521.9414
BandwidthTest 16249.1914
NMR 41.9918
WinModDiff1 163.0067
ADB 3.1234
EHS 1.4913
AvgModDiff1 200.9328
AvgModDiff2 573.1494
NoiseLoud 53.7488
MFPD 1.0000
RDF 0.9988
Is there something wrong in my command line?
Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.
Yes - I can't seem to find proper offset for latest FAAD! Please use older FAAD (I use build from September 2001) or ISO AACDec - their offests are 2048.
I will see with Menno to find proper value for latest faad...
Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.
Is there no easier way to find out the offset?
bye, Skeeve
I think that MPC has no delay, because it is using subband filtering without one frame look-ahead for block switching, but I will have to recheck..
I will do my best to find offsets for most popular codecs, and publish those values..
Originally posted by Ivan Dimkovic
Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:
Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9 @128 Kbits/s: -0.7479
Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:
c:lametest>lame3902.exe --decode castanets_test.mp3 castanets_test.wav
input: castanets_test.mp3 (44.1 kHz, 2 channels, MPEG-1 Layer III)
output: castanets_test.wav (16 bit, Microsoft WAVE)
skipping initial 1105 samples (encoder+decoder delay)
[/b]
I think this offset stripping has been there for a while. I got the following ODGs with LAME 3.90.2 @128kbits/s on CASTANETS.WAV:
Lame3902 --abr 128 -h test.wav (136.4 kbps; qval=2) = -.5301
Lame3902 --abr 128 test.wav (135.6 kbps; qval=5) = -.5492
Lame3902 -q0 test.wav (qval=0) = -.7132
Lame3902 -q1 test.wav (qval=1) = -.7429
Lame3902 -nspsytune test.wav (qval=5) = -.7732
Lame3902 -h -nspsytune test.wav (qval=2) = -.7712
Lame3902 -h test.wav (qval=2) = -.7302
Lame3902 test.wav (qval=5) = -.7649
Lame3902 -f test.wav (qval=7) = -.7822
Some possible conclusions:
1. -q1 is inferior to -q2 (I don't think its use is recommended anymore).
2. If -nspsytune improves the perceived quality them this program penalises it.
3. ABR measures much better than CBR
Originally posted by dosdan
Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:
Actually, the offset is still there... It doesn´t remove all of it...
Look:
Offset 64:Frame: 8339
Time elapsed: 340.78
Resulting ODG:
0.0655Resulting DIX: 3.2649
BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000RDF 0.0000
Offset 0:Frame: 8339
Time elapsed: 311.29
Resulting ODG:
0.0856Resulting DIX: 3.4091
BandwidthRef 21536.2383
BandwidthTest 20665.4160
NMR -21.2534
WinModDiff1 2.6592
ADB -0.4966
EHS 0.1676
AvgModDiff1 2.5517
AvgModDiff2 3.5112
NoiseLoud 0.0434
MFPD 0.9642RDF 0.0000
Originally posted by Skeeve242
Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.
I also got a positive ODG with Ogg Vorbis RC3 -q 6 on ftb_samp.wav (ODG = 0.0228). I'm pretty sure that MPC and Ogg both have zero offset. I did the test with CoolEdit that Ivan described. These occasional positive ODG make me doubt the results of Eaqual. Would be nice if somebody had an explanation.
I'll talk with Alexander about that issue..
I've noticed that with PsyTEL VBR modes, too
Anyway, Menno fixed FAAD so now it decodes with zero offset, too! I am not sure when will he upload it to CVS, but expect that to happen soon..
-- Ivan
Here is what alexander told me regarding positive ODG:
Hi,
It's true that a positive ODG should not occur
(theoretically).
However, the ITU recommendation does allow positive ODGs,
because this could also happen in listening tests, where the
file under test is rated better than the reference file.
Furthermore you may not forget, that there is a neural network
builtin which was trained on real listening test ratings. The
confidence intervall in those tests may ly between 0 and 0.5,
and thus EAQUAL may be wrong within those intervalls.
Just for giggles, and because I love Ogg Vorbis , here's some more ODG values for castanets.wav:
OggEnc RC3 @ -q 3.00: -0.7350
OggEnc RC3 @ -q 3.50: -0.6381
OggEnc RC3 @ -q 4.00: -0.5918
OggEnc RC3 @ -q 4.50: -0.5131
OggEnc RC3 @ -q 5.00: -0.3051
Ogg should have a zero offset, IIRC. Decoding was done with WinAmp 2.78 w/ Ogg Vorbis Input Plugin 1.17 beta, and Nullsoft Disk Writer Plugin 2.0b.
Now we can see that ogg is the best
What kind of ODG values do you get using common settings/presets (standard to high quality) with LAME, MPC, AAC and OGG?
I posted some results here:
http://home.wanadoo.nl/~w.speek/eaqual.htm (http://home.wanadoo.nl/~w.speek/eaqual.htm)
hehe only ogg sounds better then the original
the only positive value in speeks tests
Originally posted by Benjamin Lebsanft
hehe only ogg sounds better then the original
the only positive value in speeks tests
Try Lame with "--abr 256 -q 0 -m j" and you´ll see some more posivite values...
Originally posted by Ivan Dimkovic
Here is what alexander told me regarding positive ODG:
[some text]
So in essence, harmonic distortion should make the ODG values positive, or am I way wrong here? (I´m kinda´ new to this you know... )
Well, like I said - this tool could be used for codec debugging, and not for final marking of which codec sound better
For example - this codec ranks FAAC castanets @ 128 with ODG of -0.8 , and I am sure that even for ISO listener FAAC is still below "excellent" range at 128 kbps, etc..
I posted some results here:
http://home.wanadoo.nl/~w.speek/eaqual.htm (http://home.wanadoo.nl/~w.speek/eaqual.htm)
I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original). However, it may not have a big impact on the results.
ff123
Originally posted by ff123
I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original). However, it may not have a big impact on the results.
ff123
Here are the results with your 128 kbps command-line, but without --scale 0.93:
ftb_samp: -1.2156
sade__sweetest_taboo: -0.9003
iron: -0.8857
fools: -1.0177
ftb_samp and sade are slightly better with --scale 0.93. Iron and Fools are about the same. I didn't try the --alt-presets with --scale 1. Maybe later.
What is the offset for FhG. Fastenc (CoolEdit Pro filter)? With the test Ivan described I think it's about 96 (that's 192 for EAQUAL) when decode with LAME, but when I put an higher offset number in EAQUAL the ODG gets better. Wouldn't it be logical that the offset that gives best ODG is the right offset :confused:
Here's some more ODG values for Speek's samples, except one, because it's smarter than me.
(Speek, I can't decode sade_sweetest_taboo.pac. LPAC says that it's not a valid LPAC file. Did I pick the wrong decoder?)
These tests were done with lame 3.91 using --alt-preset 128 and oggenc rc3 -q 4.00. This was to test files that were nominally targeting 128 kbps, since that's the file size I want, and I don't want to futz with command lines until they're exactly 128 kbps (although that's useful information too). I just want to cram in the command line that gives me the best quality near 128 kbps. Quick and easy.
lame 3.91 --alt-preset 128
fools.wav : 126.4 kbps : ODG -0.9395
ftb_samp.wav : 121.1 kbps : ODG -1.3005
iron.wav : 126.9 kbps : ODG -0.8146
oggenc rc3 -q 4.00
fools.wav : 121.9 kbps : ODG -0.8032
ftb_samp.wav : 131.0 kbps : ODG -0.8292
iron.wav : 123.6 kbps : ODG -0.8476
Interesting results, to look at. Oggenc does a pretty good job at looking "consistent." I think I'm going to go play some Civ III now.
Here's some real world numbers with ogg:
Drowning Pool - Bodies
GT2 192.0 -.36184*
5 160.2 -.3340
5.5 180.8 -.0975
6 201.2 .0290
Primus - Mrs. Blaileen
5 145.0 -.3014
5.5 161.8 -.0781
6 177.7 .0332
Propellerheads - Bang On!
5 152.0 -.3271
5.5 170.0 -.0932
6 187.0 .0182
*Garf Tuned RC2 @ 160
Hmm, why do you use the "sade__sweetest_taboo" sample anyways? It was a sample i sent to Roel about a year ago, because of some lowpass issues. This sample isn't really a "hard" sample for encoders... or was it that what you wanted, an everyday-sample?
I tested my dogies.wav encodes using the analysis tool, and here are the results:
ODG
xing: -1.35
wma: -1.46
lame: -1.24
ogg: -1.48
aac: -1.15
mpc: -0.73
The listeners rated the files as follows:
xing: -2.61
wma: -1.77
lame: -1.75
ogg: -1.63
aac: -0.85
mpc: -0.74
In other words, xing was much worse than wma/lame/ogg, which in turn was much worse than aac/mpc. The analysis tool does not seem to rate the files the same way.
BTW, I went through a lot of trouble when I first made the WAV sample to time align all the files, and I just now double-checked to make sure that the samples were all aligned. The ogg file was shifted by 1 sample (23 microseconds), so I shifted it into alignment, but the numeric results didn't change.
I wouldn't necessarily declare results from this program infallible yet
ff123
To those using the w32 LAME 3.90.2 binary. I've noticed that
the decoded.wav generated by "lame --decode test.mp3 decoded.mp3" while not having a offset at the front may have added trailing silence so that the size of decoded.wav is slightly bigger than test.wav
This will cause incorrect ODGs particularly on smaller test files say 1MB. I've been improving accuracy by removing the trailing silence in CoolEdit. The extra size varies from 100 bytes to 3KB so far. So check those decoded wav file sizes.
Speek, I can't decode sade_sweetest_taboo.pac. LPAC says that it's not a valid LPAC file. Did I pick the wrong decoder?
I see, the file got truncated with uploading. I'm uploading it again now.
Hmm, why do you use the "sade__sweetest_taboo" sample anyways? It was a sample i sent to Roel about a year ago, because of some lowpass issues. This sample isn't really a "hard" sample for encoders... or was it that what you wanted, an everyday-sample?
So that's were it came from! I was just picking some of the longer testfiles I have. Most are from the LAME test samples webpage, but I saw that this one wasn't. In the readme of EAQUAL it says that a test sample should be at least 10 to 20 sec., but the longer the better. Many test samples are to short.
I wouldn't necessarily declare results from this program infallible yet
No, I wouldn't to. I'm just getting some data. I don't know were it leads to. One of the things I'm thinking about is: Ogg seems to get the best ODG scores. Is it really the best encoder, or does it something better than other encoders that EAQUAL pays much attention to? Maybe Ogg does something else worse than some other encoders, but EAQUAL doesn't pay much attention to that factor? Just open questions, I don't have a clue
It is using 1024 point FFT window for the analysis - so it could miss some pre-echo events now and then.
It is ranking psytel velvet.aac (encoded with 2.01) better than Dolby Professional AAC encoding, and at least to my ears Dolby sounds better (not too much, but I can hear the difference)
Hi all.
Hope you enjoy EAQUAL.
Sorry about the confusion about the offset, it was a bug and now it is fixed (and available at sourceforge). Now the offset is really counted in samples. If you use the old version of EAQUAL, you have to multiply the number of samples with the number of channels.
Alexander
Hi!
You have to use any -scale switches very carefully. The algorithm is stable for small level differences, but it will suck for big level differences.
Scaling can be very useful because many decoders tend to clip full scale audio output - in this case the test signal would be distorted. Here you should use scaling to avoid wrong ratings with clipped audio signals.
Alexander
of the things I'm thinking about is: Ogg seems to get the best ODG scores.
Hmmm... that's not what I see. From the results on your site it looks like mpc -xtreme is the best.
Jan.
My last message now
If you must use EAQUAL for codec ratings (which is not really its intention), please keep in mind that you have to average over _many_ audio files to get reasonable results - and with good reasons you may doubt those ratings too. You have to average results over least 20, 50 or 100 signals I would guess.
Alexander
Originally posted by Jansemanden
Hmmm... that's not what I see. From the results on your site it looks like mpc -xtreme is the best.
Jan.
Look again Most times Ogg Vorbis -q 6 gets a better score than MPC -xtreme. The times MPC gets a better score it has a much higher bitrate.
BTW. a positive ODG is very good according to Alexander Lerch (he is the author of EAQUAL).
BTW2. I'm not saying Ogg is better than MPC, I'm just saying it gets a better score from EAQUAL on the samples I tried.
Hehe... competition is now having to tune psymodels of their encoders to match EAQUAL values hehehe (just kidding)
Did anybody run EAQUAL on WMA? I know it sucks, but just for a quick check
Originally posted by Ivan Dimkovic
Hehe... competition is now having to tune psymodels of their encoders to match EAQUAL values hehehe (just kidding)
Did anybody run EAQUAL on WMA? I know it sucks, but just for a quick check
What bitrate would you like?
96 and 128, but WMA is a VBR codec - it sometimes gives too large bitrate, greater than nominal...
I've justed added WMA8 results for 128 kb/s. It doesn't suck to hard according to EAQUAL, but the bitrates are a bit higher than the others.
Originally posted by Speek
I've justed added WMA8 results for 128 kb/s. It doesn't suck to hard according to EAQUAL, but the bitrates are a bit higher than the others.
Well, try psytel aacenc with -noshort switch (disables short blocks) - you'll notice that it also performs "good" but the results are unacceptable from the pre-echo point of view
If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?
bye, Skeeve
Originally posted by Skeeve242
If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?
The offset in question is not caused by leading or trailing digital silence in the original (ripped) wave file. The offset is caused when it converted to an MP3 and then reconverted back to a wav. MP3 can have overlap of info between frames e.g. when you decode the first frame of a MP3 you can't be sure you have all the data for the very start of the music info. I believe there is feame overlap both in encoding and decoding but if you know what this delay is you can remove it. If you use LAME to decode a LAME encoded file these constant values are known to the program and it can compensate when decoding i.e.
skipping initial 1105 samples (encoder+decoder delay)[/b]
Not sure about the trailing digital silence that I'm seeing from the decoded file. This was not on the original and has been added after decoding so EAC could never strip it. The amount of added digital silence varies so it is not a constant number of samples to be removed.
Originally posted by Skeeve242
If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?
The offset in question is not caused by leading or trailing digital silence in the original (ripped) wave file. The offset is caused when it converted to an MP3 and then reconverted back to a wav. MP3 can have overlap of info between frames i.e when you decode the first frame of a MP3 you can't be sure you have all the data for the start of the data stream. I believe there is overlap both in encoding and decoding but if you know what this delay is you can remove it. If you use LAME to decode a LAME encoded file these constant values are known to the program and it can compensate when decoding i.e.
skipping initial 1105 samples (encoder+decoder delay)[/b]
Not sure about the trailing digital silence that I'm seeing from the decoded file. This was not on the original and has been added after decoding so EAC could never strip it. The amount of added digital silence varies so it is not a constant number of samples to be removed.
I've added some more results to my EAQUAL page:
http://home.wanadoo.nl/~w.speek/eaqual.htm (http://home.wanadoo.nl/~w.speek/eaqual.htm)
Just out of curiosity, could someone also give --r3mix a shot.
Originally posted by ff123
I tested my dogies.wav encodes using the analysis tool, and here are the results:
In other words, xing was much worse than wma/lame/ogg, which in turn was much worse than aac/mpc. The analysis tool does not seem to rate the files the same way.
BTW, I went through a lot of trouble when I first made the WAV sample to time align all the files, and I just now double-checked to make sure that the samples were all aligned. The ogg file was shifted by 1 sample (23 microseconds), so I shifted it into alignment, but the numeric results didn't change.
I wouldn't necessarily declare results from this program infallible yet
ff123
You have to keep in mind, that it is difficult already to compare results from different listening tests, even if they were realized ITU-R BS.1116-compliant.
Perhaps you should think of EAQUAL as _one_ listener whose results are not depending on his mood and his concentration, who never gets tired and whose results are repeatable, though only his opinion.
BTW, what are the confidence intervals for your listeners?
The difference of one sample has indeed very little influence on the result. If you want to interpret this in the frequency domain, the big difference will only be there for frequencies near Nyquist. These frequencies will not influence the result much.
And it's true, no model can be infallible
Alexander
The results of the dogies listening test can be found at:
http://ff123.net/dogies/dogies_plots.html (http://ff123.net/dogies/dogies_plots.html)
With an experiment-wise confidence of 95%, MPC and AAC were deemed better than the others, and LAME/WMA8/OGG were deemed better than XING. N = 15.
The Fisher's protected LSD is known to not be "protected" if the number of samples exceeds 3, but I verified via resampling analysis that the results are indeed as given above with 95% confidence.
In other words, the listeners ranked the various codecs on this sample with clear and statistically significant preference.
ff123
I'm not sure I buy the argument that averaging results over many different samples will improve the performance of the EAQUAL algorithm.
For example, suppose that EAQUAL in general fails to properly penalize pre-echo artifacts (which seems to be a plausible hypothesis). Then, averaging over many different samples won't necessarily compensate for this failing. Yes the results will be more reliable, but they won't be more valid.
ff123
Alexander,
I wonder why a test sample for EAQUAL should be at least 10 to 20 sec.?
Originally posted by ff123
For example, suppose that EAQUAL in general fails to properly penalize pre-echo artifacts (which seems to be a plausible hypothesis). Then, averaging over many different samples won't necessarily compensate for this failing. Yes the results will be more reliable, but they won't be more valid.
ff123
Hi,
Of course you are correct. EAQUAL in its current version is not very sensitive regarding pre-echo artifacts.
Let me formulate it in this way: the algorithm's results will be more valid within the limitations of the algorithm. This was more or less proved, see Treurniet et al., Evaluation of the ITU-R Objective Audio Measurement Tool, Journal of the AES No.48, 2000
Alexander
Originally posted by Speek
Alexander,
I wonder why a test sample for EAQUAL should be at least 10 to 20 sec.?
For the analysis, a few seconds at beginning and end do not influence the result. This should be because the listeners are often less concentrated or may think artefacts like clicks etc are due to their playing equipment which is starting or stopping. Don't know if this is true or not. Perhaps there is another explanation too, but standards don't explain anything.
Alexander
For the analysis, a few seconds at beginning and end do not influence the result. This should be because the listeners are often less concentrated or may think artefacts like clicks etc are due to their playing equipment which is starting or stopping. Don't know if this is true or not. Perhaps there is another explanation too, but standards don't explain anything.
There was another objective tool based on the work of Frank Baumgarte, which EarGuy implemented and played with in the r3mix forums. It seemed to consistently show bad results at the beginning of an mp3 file. I don't know of any listening tests which confirm this effect, or what could explain it, but in the files I prepare for listening tests, I sidestep this problem (if it is a real effect) by duplicating the entire sample, and then by cutting it in half (choosing the second half, of course) for the final file.
[tangent]Interestingly, Fraunhofer's mp3enc codec will produce different artifacting depending on what signal went before, so it can be tricky to capture bad sections for this codec by excerpting sections of a song, and the method I use above will yield different results for mp3enc than just using the straight sample.[/tangent]
I also found early on that playing files which don't begin or end with silence can cause clicks initially upon playback. I avoid this problem by adding some silence before and after the sample.
ff123
Just out of curiosity, could someone also give --r3mix a shot.
Just added --r3mix to my test results.
Strange things happening on deerhunter.wav (sample from ff123). Lame --r3mix with only 113 kb/s is better than --alt-preset (fast) standard (154 and 163 kb/s). Also Psytel aacenc -normal (153 kb/s) is better than -extreme (183 kb/s). :confused:
Link: http://home.wanadoo.nl/~w.speek/eaqual.htm (http://home.wanadoo.nl/~w.speek/eaqual.htm)
ITU-R BS.1387 also has "advanced" ear model, which is implemented in Opticom OPERA software,
here is what AES paper #4931 (regarding to PEAQ) says about "advanced" model:
3.4.3 Advanced Version
Compared to the "basic" version, this model performs the time to frequency warping using a filter bank, thus grouping the signal into 40 auditory bands with a temporal resolution of approximately 0.66 ms. This allows for a very accurate modelling of backward masking effects.
Andree Buschmann told me in some mail that he wasn't satisfied with PEAQ tool used in debugging of the AAC codec in Bosch - tool missed some important artifacts in fatboy.wav - but that's ok from my point of view - BS.1387 has its application, it is good to some degree, but it can't replace audiophile's ear
Originally posted by Alexander Lerch
For the analysis, a few seconds at beginning and end do not influence the result.
Alexander
Oops, here I was not correct. It is some time ago I implemented this and kept it wrong in mind, perhaps I read something about this. It is only .5 seconds at beginning and end that don't influence the result. So, the minimum filesize would reduce a little bit.
Alexander
Im in the middle of some testing, Id just like to confirm that 1024 is the correct offset for Psytel AAC, I`ll post the results later.
Thanks,
Alex
Originally posted by Alex
Im in the middle of some testing, Id just like to confirm that 1024 is the correct offset for Psytel AAC, I`ll post the results later.
It depends on what decoder you use. With the newest FAAD decoder dated 2002-01-05 the offset is 0.
so with 2002-01-05 its 0 and pre-2002-01-05 its 1024?
I just want to confirm this, Im getting some supprising results for Psytel, the results for other codecs are what I would expect.
I will post my results shortly but Id like to make sure they will be accurate.
So if I wanted to test a LAME encoded and decoded file I shouldn't set any offset?
Jan.
yes so long as you use lame.exe --decode then there is no offset, Im not sure about any other decoders.
so with 2002-01-05 its 0 and pre-2002-01-05 its 1024?
With FAAD 2002-01-05 the offset is 0
With FAAD 2001-06-06 the offset is 1024
With other FAAD I don't know. But you can easily do the test with castanets and CoolEdit (or any other wave editor) that Ivan described at the beginning of this thread.
So if I wanted to test a LAME encoded and decoded file I shouldn't set any offset?
The LAME decoder corrects the offset. So if you decode a LAME MP3 file with LAME the offset is 0. If you use another decoder with no offset correction the offset will probably be 1105.
I downloaded the 2002-01-05 FAAD and the results were ok, although Psytel AAC still performed worse than Lame in the test that I did.
Originally posted by ff123
There was another objective tool based on the work of Frank Baumgarte, which EarGuy implemented
Yeah, I hope Todd (Earguy) can come up with improved Baumgarte's digital ear model.
Quoting his email:
"I discovered a major mistake that I made. I can't read German, so I was using online translators to translate Frank's disertation to English. In his disertation, he gives two different sets of model parameters to use. I didn't understand what the difference was between the two until now (online translators have improved in the last two years). And of course I was using the wrong set of parameters! So the previous results are in error. I will definitely have something to show early next year so watch the forums for my posts."I hope that correctly working Baumgarte's digital ear will be better than PEAQ ITU-R BS.1387 basic model.
Originally posted by JohnV
Yeah, I hope Todd (Earguy) can come up with improved Baumgarte's digital ear model.
Quoting his email: "I discovered a major mistake that I made. I can't read German, so I was using online translators to translate Frank's disertation to English. In his disertation, he gives two different sets of model parameters to use. I didn't understand what the difference was between the two until now (online translators have improved in the last two years). And of course I was using the wrong set of parameters! So the previous results are in error. I will definitely have something to show early next year so watch the forums for my posts."
I hope that correctly working Baumgarte's digital ear will be better than PEAQ ITU-R BS.1387 basic model.
Speaking about EAQUAL (PEAQ) psychoacoustic model, I had finally taken a look in the recommendation (psychoacoustic model description) and I must say that it is based on more advanced model than ISO Psychoacoustic Model II - I also know that one state-of-the-art compressor (I won't tell which !) is using many of properties from this model.
Non-linear Spreading function used in this model is described in Baumgarte's "Application of non-linear psychoacoustic model in ISO Layer III codec" (used to avoid tonality estimation problems). Also it uses temporal masking.
Its only weakness is 2048 sample length of the analysis window.
Its only weakness is 2048 sample length of the analysis window.
Ivan, could you explain this a little bit more?
Here's how imagine things:
A = reference file
B = test file
x = window where the attack/transient is (or begins)
I suppose EAQUAL compares each window of B to the corresponding window of A. So when there's pre-echo in B either Bx or Bx-1 will not be the same as Ax or Ax-1. So EAQUAL will detect the pre-echo. But I guess things don't work as I imagine?
Basically, the tool sees all audio in 20ms chunks in the frequency domain. This means that it will not clearly see any distortion smaller than 20ms in the temporal domain, because the error from the preecho is 'spread out' over a 20ms timeframe.
The preecho is audible because it is a relatively big distortion over a very small time interval. The tool will not see the difference between it and a very small distortion over a larger interval, which would be inaudible.
2048 samples is about _twice_ the size of an MP3 _large_ block btw. This would mean it's really completely blind to any temporal artifact.
(Disclaimer: Not sure this is all 100% correct, and I hope those that know will correct the mistakes)
--
GCP
Originally posted by Speek
Ivan, could you explain this a little bit more?
Here's how imagine things:
A = reference file
B = test file
x = window where the attack/transient is (or begins)
I suppose EAQUAL compares each window of B to the corresponding window of A. So when there's pre-echo in B either Bx or Bx-1 will not be the same as Ax or Ax-1. So EAQUAL will detect the pre-echo. But I guess things don't work as I imagine?
Imagine this situation:
Divide the 2048 in 8 blocks of 256 time-domain samples (128 coefficients)
Now, inject the noise in each of the 8 blocks, say 10 dB - in all cases tool with 2048 sample window will detect it as the same increase of noise, but it can't guess in what segment problem occured.
2048 sample (1024 coefficient) window is standard MPEG AAC "LONG" window size. Because it leads to serious pre-echo problems, smaller window of 256 samples is employed if the signal conditions require it.
Originally posted by Ivan Dimkovic
I also know that one state-of-the-art compressor (I won't tell which !)
Fraunhofer aac demo v2.2?
bye, dB
Could someone test the GOGO NoCoda with competitive bitrate VBR settings?
hello all
have anybody have the free source code for ITU_R B.1387 or link to it( http://sourceforge.net/projects/eaqual/ (http://sourceforge.net/projects/eaqual/) this link does not contains any code.)
If i have to purchase the eaqual source code what is the price and source for it
u can contact me at [a href='mailto:ashok_i_m@rediffmail.com'][/a]
Originally posted by ashok
hello all
have anybody have the free source code for ITU_R B.1387 or link to it( http://sourceforge.net/projects/eaqual/ (http://sourceforge.net/projects/eaqual/) this link does not contains any code.)
If i have to purchase the eaqual source code what is the price and source for it
u can contact me at ()[/a][/a][/a]
thanks speek
the code here is building and running fine
thanks alot:)
or mp3-tech.org
which version of EAQUAL is the latest available for download and where?
http://rarewares.org/others.html (http://rarewares.org/others.html)
http://rarewares.org/others.html (http://rarewares.org/others.html)
thx.