This is NOT a general 3.96b ABX results thread. This is just for people to post (any) samples which regress from 3.90.3. This means that I only want cases where 3.96 is WORSE than 3.90.3.
Please try to keep this on-topic. The idea is that if this thread stays empty, we should push for 3.96 to become official. And if it doesn't stay empty, samples will help the LAME devs with future versions.
Any and all settings are fine... just make sure you post how you encoded it, and link to samples.
Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC) (http://members.cox.net/moitah/doesnair.flac). Listen to the part where the snair hits 4 times, about 1.5 seconds in. I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.
I tried a 28-second sample from Tool's track (-) Ions. I chose this to test first, as it gets an average bitrate of 384 kbps using Vorbis GT3b1 at q5.
Original sample is in FLAC format.
I encoded mp3 versions using only the commandline --alt-preset 128 in RazorLame. I used LAME 3.90.3 modified and 3.96b1.
FLAC and mp3 versions have ReplayGain information applied.
I found that the 3.96b1 version has some kind of distortion during the lower volume parts of this clip - between the peak volume levels of the buzzing. I was able to use this to ABX 3.96 from 3.90.3 8/8.
(-) Ions (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=191563)
Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.
Also, I would encourage people to also post improvements from the old encoder, since otherwise we might be skewing our preception. Usually, improvements also introduce other minor flaws that nonetheless end up making the thing a good trade-off.
Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.
I agree. I didn't know that my uploaded sample would end up on the portal. It was rather embarrassing.
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.90.3.wav
9 out of 10, pval = 0.011
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
9 out of 10, pval = 0.011
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
13 out of 16, pval = 0.011
There is a HF hiss present (0:01-0:02).
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
12 out of 17, pval = 0.072
Guitar sounds watery.
Moitah's The Dance of Eternity sample uploaded at HA,
Play - Is It Love (4 sec, FLAC) (http://members.cox.net/moitah/isitloveintro.flac) (try this link first, to save HA bandwidth)
--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001). The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
12 out of 17, pval = 0.072
Guitar sounds watery.
As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.
And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.
Play - Is It Love (4 sec, FLAC) (http://members.cox.net/moitah/isitloveintro.flac) (try this link first, to save HA bandwidth)
--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001). The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.
I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).
And please compare your results with 3.90.3. To be fair chose the same session procedure. and try to ABX 3.90.3 against 3.96.
EDIT: I think everyone should act according to tigre's proposals in the LAME poll thread:
# Posting results in the thread requires:
* Upload or link to sample
* ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
* Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones
# Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.
Since we have a regression thread, maybe we should also have a progression thread.
As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.
And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.
It was the only test I did with 3.96, I also did 1 ABX test with 3.90.3 got 5/5 thus my post of the sample.
Anyway your not a moderator, you didn't start the thread or even post a sample so I don't really see what your interest is, you just must like telling everyone what to do .
(HA Wiki)
A difference is concluded to be heard when 13 correct identifications out of 16 trials is achieved
This is what we are expected to score for a valid problem sample, people, not what amano wants us to do.
Westgroveg, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.
EDIT: Corrected Westgroveg's name
Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96
My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?
& my nck is westgroveg.
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96
My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?
Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.
Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96
My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?
& my nck is westgroveg.
As Freakngoat just posted, the idea is that we ONLY post cases where 3.96 is WORSE than 3.90.3. You just posted cases where they both mess up - which isn't the point of this thread.
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96
My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?
Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.
Oh, sorry I misunderstood, I thought the thread asked, samples which are problematic for both 3.90.3 & 3.96, sorry Jebus.
np
I added a clarification sentence to the first post
I just had a go at PVNC's (-) Ions sample (I used the contained mp3's for ABX)
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav
16 out of 16, pval < 0.001
C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.90.3 (-) Ions.wav
10 out of 10, pval < 0.001
Both have a huge distortion (0:02-->0:04), 3.90.3 sounds slightly better.
Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC) (http://members.cox.net/moitah/doesnair.flac). Listen to the part where the snair hits 4 times, about 1.5 seconds in. I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.
I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?
I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?
I'm not sure what it's called, pre-echo maybe. If you start with 128 it should be easier.
I need to do more testing with my second sample, it might be happening in 3.90.3 as well.
I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).
You should
always add all ABX result together for the same sample. It is statistically valid.
What you should
never do is do more than one set of ABX tests, but only report
some of them - you always need to report
all the ABX testing you did with a sample. Selective ABX results are not valid.
Also, watching the result, and stopping when it's "statistically valid" isn't right either, but I haven't seen an explanation of what the correct statistics are in this case. The advice has always been to decide how many trials you will take
before you start. But if you watch your result, then the "standard" statistics (which are found in most ABX programs) are wrong.
Cheers,
David.
My second sample isn't a regression, I ABXed 16/20 and 12/12 (total 28/32, pval < 0.001) for 3.90.3 --alt-preset standard (I think I was able to ABX 3.90.3 better because I knew what to listen for, not because it sounded worse than 3.96).
westgroveg, I was just suggesting a way to make this thread as useful as possible. I am not your mother, therefore I am just suggesting, not commanding...
David, tnx for clarification. I was assuming, that you had to set a specific number of trials before the test. And the result would have to be taken from this session.
Not trying again a session and then a next one, until you find a result that is pleasing.
OK did some more testing on the (-) Ions sample,
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\(-) Ions-APS-3.96b1.wav
15 out of 16, pval < 0.001
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\(-) Ions-APS-3.90.3.wav
14 out of 16, pval = 0.002
ABX Results:
C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\(-) Ions-APS-3.90.3.wav vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\(-) Ions-APS-3.96b1.wav
9 out of 10, pval = 0.011
I'm disappointed to say 3.90.3 definitely outperformed 3.96.
Edit: If it was unclear I tested --alt-preset standard.
If I'm reading this right, you were able to ABX 3.96b1 15/16 times, and 3.90.3 14/16 times. That seems pretty darn close to me. Maybe I'm misinterpreting the data.
If I'm reading this right, you were able to ABX 3.96b1 15/16 times, and 3.90.3 14/16 times. That seems pretty darn close to me. Maybe I'm misinterpreting the data.
Then I ABXed 3.90.3 against 3.96 & as I wrote 3.96 sounded worse.
# Posting results in the thread requires:
* Upload or link to sample
* ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
* Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones
# Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.
Wouldn't we rather need ABC/HR results (with ratings) in order to know which codec sounds better ?
I mean, if you can ABX 3.90.3 vs. 3.96, it only proves you can hear a difference between the two, but you can't tell which one sounds better (because you don't know which one you're listening to).
If you rate the codecs with ABC/HR, it will clearly indicate which one sounds better.
Please correct me if I'm wrong.
Please correct me if I'm wrong.
You are wrong. With file ABX you chose which one is A and which one is B, so you would definitely be able to make a judgement as to which sounds better. ABC/HR is designed to eliminate personal bias, with ABX one could easily skew the results (in fact you could flat out lie and say that the one that sounded worse actually sounded better). That, I believe, is one of the reasons for requiring that at least one person verifies results in this thread.
I've tested 3.96beta against 3.90.3 on my hustlejet sample.
Hustle Jet LAME test (http://cyberquebec.ca/harashin/Hustle_Jet_LAME_test.htm)
@harashin
Did you replaygain the samples in a way?
Husltlejet suffers from clipping, even more with 3.96b
@westgroveg
Ions is a sample showing 3.90.3 superior due to how it works.
3.90.3 Chooses 320kbit for short blocks in general i think. Ions is full of it.
3.96b doesn´t use always 320kbit for short blocks. This may cause the slight degration here.
Edit: If this is the case, it would be hard for 396b outperforming 3.90.3
Wombat
@harashin
Did you replaygain the samples in a way?
Husltlejet suffers from clipping, even more with 3.96b
No, I didn't. So I tried again on that sample.
Hustle_Jet_LAME_RG_test (http://cyberquebec.ca/harashin/Hustle_Jet_LAME_RG_test.htm)
Tnx harashin,
these samples are hard to compare. 3.96b is about 50 kbps smaller.
@amano
I wouldn't hear the difference, if the average bitrate of both files were similar.
I did another test on PVNC's (-) Ions sample.
(-) Ions LAME test (http://cyberquebec.ca/harashin/Ions_LAME_test.htm)
Tnx harashin,
these samples are hard to compare. 3.96b is about 50 kbps smaller.
That doesn't matter - we're comparing --aps to --aps, if 3.96 is using too low a bitrate then that is a legitimate regression for --aps.
Possible regression in badvilbel (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=16&t=1059) (download from ff123.net (http://www.ff123.net/samples.html)):
Original vs. 3.96b1: 35 out of 50, pval = 0.003
Original vs. 3.90.3: 20 out of 52, pval = 0.965
3.90.3 vs. 3.96b1: 20 out of 46, pval = 0.849
Tested using --alt-preset standard, 4.4 -> 5.8 seconds into the clip, with ABC/HR (so I couldn't tell which version I was ABXing).
When I saw that I couldn't ABX the difference between the two compressed versions, I went back and did 8 more of each vs. original. 3/8 for the 3.90.3, 7/8 for 3.96b1. These are included in the total ABX scores.
Definite regression in trumpets1 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=16&t=3594):
Original vs. 3.96b1: 8 out of 8, pval = 0.004
Original vs. 3.90.3: 9 out of 16, pval = 0.402
3.90.3 vs. 3.96b1: 8 out of 8, pval = 0.004
The 3.96b1 encode is really warbly, worse than any artifact I've heard with Lame 3.90 --aps, worse even than the erhu sample (http://www.hydrogenaudio.org/forums/index.php?s=&act=ST&f=16&t=4687) (which is equally bad with both Lame versions: 13/16 for both, couldn't ABX the difference).
In case anyone wants the erhu sample, the link in the erhu thread is broken, so I've attached my copy of the sample.
Tell me to shut up if I'm talking nonsense, but...
I've missed the post where someone said they'd adapted the (--alt)-preset standard tuning to work with 3.96.
Surely, if the tunings which worked for the old encoder are simply copied over to the new (updated, i.e. changed!) encoder, then they can't be expected to work as well - simply because they were specifically targeted at problems in the older encoder.
Or is there a general expectation that the problems which --alt-preset standard addressed in 3.90.3 are still present in 3.96, and therefore it should still work well?
Or has --alt-preset standard been updated in this version?
Otherwise, it seems to me that the only way to get anywhere is to show that some vanilla command line is better in 3.96 than in 3.90.3 at the same (approx.) bitrate, and then to make a new preset/hack/whatever to fix any remaining problems.
Sorry if I've jumped in too late and missed all this already. I'm sure Gabriel will put me right.
Cheers,
David.
Surely, if the tunings which worked for the old encoder are simply copied over to the new (updated, i.e. changed!) encoder, then they can't be expected to work as well - simply because they were specifically targeted at problems in the older encoder.
Of course the presets have been updated for the new code.
@2Bdecided
The preset is totally different in behaviour. Gabriel seems to try to port the preset to the newer optimizations.
The vbr mode changed, short block behaviour is totally different and maybe other things.
Still the preset must be more than only a command line. Maybe Gabriel can explain?
Till here it seems that only my birds and sophia2 sample became better with 3.96b. They seem to profit from better masking or something like that. All other samples seem to became worse
Edit: Ups! Same time posted as Gabriel!
Wombat
All other samples seem to became worse
awe32_20sec is much better, and at a slightly lower bitrate.
That's why it would be helpful to have two sticky threads: samples where it's worse, and samples where it's better. That way, it would be easier to form a fair opinion.
Cheers,
David.
I thought fatboy was a regression example - but if you try the full intro (I've uploaded it in a separate thread) it seems that both 3.90.3 and 3.96 both have similar problems, but in different places.
Cheers,
David.
Another definite regression, this time in death2:
Original vs. 3.96b1: 8 out of 8, pval = 0.004
Original vs. 3.90.3: 0 out of 2, pval = 1.000 (didn't bother)
3.90.3 vs. 3.96b1: 18 out of 24, pval = 0.011
3.96b1 has lots of smearing on the transients. I tested somewhere in the middle of the sample, not right at the beginning. I will try adding -b128 to the 3.96b1 commandline to see if that improves things. I also tried castanets while I was at it: 5/5 ABX for both vs. original, but 15/30 trying to ABX against each other.
I'm in the process of ABXing the Birds sample, too. I need to rest my ears and run more trials, though, before I report my results. So far, I cannot ABX Sophia2, Liebestod, or awe32_20sec any which way, but this may be due to extreme listener fatigue.
Sophia2
Introduces added noise from 1.2-2sec and 4-6sec
Birds
Distortion is added mostly with the "e" when she sings "become"
I just copied this from the thread i introduced these samples. If you heard the problems once, it is easy. If you want to spot the problem easier for learning try to disable -Z again or use pure 3.90.
I call this distortion sandpaper noise.
The Liebestod sample was ok since aps 3.90.3 used -Z by default, before it was horrible and the most annoying artifacts i heard with aps.
btw.
Cool thing, you try so much!
It seems 3.95.1 has similar regressions such is with 3.96.
I hope more people start ABXing samples, even the same ones I did. I'm just one particular set of ears, testing specific time intervals of a few sampless. It's possible that I would have heard improvements instead of regressions, had I picked other one-second snippets of each sample to analyze! There's not enough information in this thread to absolve nor condemn 3.96b1.
I just copied this from the thread i introduced these samples. If you heard the problems once, it is easy. If you want to spot the problem easier for learning try to disable -Z again or use pure 3.90.
I call this distortion sandpaper noise.
btw.
Cool thing, you try so much!
Well, I just got these new headphones, so I'm having a bit of fun comparing my ABX scores to the ones I got with my old headphones in previous ABX tests. Two years ago, I wrote down for the Birds sample "successful ABX (barely)", and that's how well I'm doing now, hovering between 5% and 1% pval. Interestingly enough, I got the same score for Garf's underwater love sample this time (15/20) as I did when he first made it available!
Gabriel, have you updated the bitrate presets for 3.96b1, such as --preset 128? I want to know if I should test lower bitrates as well.
Gabriel, have you updated the bitrate presets for 3.96b1, such as --preset 128? I want to know if I should test lower bitrates as well.
Yes, they are using less short blocks than 3.95.1.
On mid/low bitrates it should improve samples like dafunk, fsol or waiting.
I've done a quick listening test. I selected the twelve samples according to my preferences and trying to represent all various genres. The results are quite clear, 3.90.2 (or 3.90.3) is still the best according to my tastes. Here is the summary table (raw abc/hr results available (http://xoomer.virgilio.it/fofobella/396b1/396b1.zip)):
(http://xoomer.virgilio.it/fofobella/396b1/results_table.png)
Since when i noticed and reported (http://www.hydrogenaudio.org/forums/index.php?showtopic=7790&st=0&#) a problem with 3.94alpha i can hear a sort of chirping or unstable noise in some passages, the major problem was fixed by Gabriel but i think that something else is still problematic. 3.90.x versions does not suffers of this problem.
Some samples are really worsened (very easy to perceive) with the new version, i hope something could be done before releasing 3.96 stable:
- fall.wav (preecho and chirping)
- applaud.wav (droputs)
- campestre.wav (chirping)
- rebel.wav (ringing)
Three of these samples are taken from my collection only because i was suspecting problems with newest lame versions.
Fall.wav preecho is really annoying, maybe is better to use again more short blocks, even for 128 kbps.
The positive news is that 3.96b1 -q0 is almost as good as --ap 128, maybe a bit better.
This sample is from Cooking with Miles Davis called "My Funny Valentine". 3.90.3's APS sounds much better than 3.96b1's APS on this sample.
This thread is closed now because there's an official test thread now (http://www.hydrogenaudio.org/forums/index.php?showtopic=19813&). I'm finished with editing in usable results from here. If you want some of your results from here included that aren't yet, please post in the other thread and provide the 'minimum requirements' (or links to where we can find them):
4. Your test results have to include the following:- ABX results for
3.90.3 vs. Original
3.96b1 vs. Original
3.96b1 vs. 3.90.3
- ABC/HR results are appreciated especially at lower bitrates, but shouldn't be considered a requirement.
- (Short) descriptions of the artifacts/differences